2024-09-16T04:54:31.990 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T04:54:32.011 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T04:54:32.089 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906097 branch: squid description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906097' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request sha1: 8921c8ef947807b717db041d85f0761b7914d5fc ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 700 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 5616 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 15927/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 targets: smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNE/0il3+C8uzd5E8kQ/LerQQAwQhbXmAw7gc4pFg2UaFFfODWF4PiaiInerRO/H6G9S6HOuXtl8cL8olrhm2gU= smithi149.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPSsAMjpZuRE7q2ekaDwBf90GoBlnSKGa6anDr0+/5foCmP6kr4QubL5rgiMiV8eCrnDEsjJbY9sSPD6taahvPw= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - rados/test.sh unit_test_scan teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T04:54:32.089 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T04:54:32.090 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T04:54:32.090 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T04:54:32.091 INFO:teuthology.task.internal:Checking packages... 2024-09-16T04:54:32.112 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-16T04:54:32.112 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T04:54:32.112 INFO:teuthology.packaging:ref: None 2024-09-16T04:54:32.112 INFO:teuthology.packaging:tag: None 2024-09-16T04:54:32.112 INFO:teuthology.packaging:branch: squid 2024-09-16T04:54:32.112 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:54:32.113 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-09-16T04:54:32.341 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-326-g8921c8ef-1jammy 2024-09-16T04:54:32.343 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T04:54:32.350 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T04:54:32.350 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T04:54:32.392 INFO:teuthology.task.internal:Saving configuration 2024-09-16T04:54:32.406 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T04:54:32.427 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T04:54:32.453 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906097', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 04:48:52.795499', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBqyoMGItHxPc04FnmdfdvjCSzZPQRGCHcBLEdhIqgOu'} 2024-09-16T04:54:32.478 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906097', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 04:48:52.797450', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFMPzgrfQDCNR85uQZWHnn960MYzx5JYWqPBipMSgWRS'} 2024-09-16T04:54:32.478 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T04:54:32.484 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-16T04:54:32.484 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-16T04:54:32.484 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T04:54:32.579 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc4c1fa6e60>, signals=[15]) 2024-09-16T04:54:32.579 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T04:54:32.587 INFO:teuthology.task.internal:Opening connections... 2024-09-16T04:54:32.587 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T04:54:32.588 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:54:32.670 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi149.front.sepia.ceph.com 2024-09-16T04:54:32.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:54:32.747 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T04:54:32.758 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-09-16T04:54:32.764 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-09-16T04:54:32.764 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-09-16T04:54:32.811 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T04:54:32.811 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="Ubuntu" 2024-09-16T04:54:32.811 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="22.04" 2024-09-16T04:54:32.811 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T04:54:32.811 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_CODENAME=jammy 2024-09-16T04:54:32.811 INFO:teuthology.orchestra.run.smithi114.stdout:ID=ubuntu 2024-09-16T04:54:32.812 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE=debian 2024-09-16T04:54:32.812 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T04:54:32.812 INFO:teuthology.orchestra.run.smithi114.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T04:54:32.812 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T04:54:32.812 INFO:teuthology.orchestra.run.smithi114.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T04:54:32.812 INFO:teuthology.orchestra.run.smithi114.stdout:UBUNTU_CODENAME=jammy 2024-09-16T04:54:32.813 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-09-16T04:54:32.831 DEBUG:teuthology.orchestra.run.smithi149:> uname -m 2024-09-16T04:54:32.836 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64 2024-09-16T04:54:32.837 DEBUG:teuthology.orchestra.run.smithi149:> cat /etc/os-release 2024-09-16T04:54:32.882 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T04:54:32.882 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="Ubuntu" 2024-09-16T04:54:32.882 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="22.04" 2024-09-16T04:54:32.883 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T04:54:32.883 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_CODENAME=jammy 2024-09-16T04:54:32.883 INFO:teuthology.orchestra.run.smithi149.stdout:ID=ubuntu 2024-09-16T04:54:32.883 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE=debian 2024-09-16T04:54:32.883 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T04:54:32.883 INFO:teuthology.orchestra.run.smithi149.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T04:54:32.883 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T04:54:32.883 INFO:teuthology.orchestra.run.smithi149.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T04:54:32.883 INFO:teuthology.orchestra.run.smithi149.stdout:UBUNTU_CODENAME=jammy 2024-09-16T04:54:32.883 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2024-09-16T04:54:32.906 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T04:54:32.920 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T04:54:32.954 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T04:54:32.955 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-09-16T04:54:32.957 DEBUG:teuthology.orchestra.run.smithi149:> test '!' -e /home/ubuntu/cephtest 2024-09-16T04:54:32.962 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T04:54:32.987 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T04:54:32.988 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-09-16T04:54:33.004 DEBUG:teuthology.orchestra.run.smithi149:> test -z $(ls -A /var/lib/ceph) 2024-09-16T04:54:33.020 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T04:54:33.078 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T04:54:33.099 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T04:54:33.099 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T04:54:33.100 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T04:54:33.101 DEBUG:teuthology.orchestra.run.smithi149:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T04:54:33.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:54:33.105 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-09-16T04:54:33.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:54:33.106 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2024-09-16T04:54:33.151 INFO:teuthology.orchestra.run.smithi114.stdout:5.15.0-56-generic 2024-09-16T04:54:33.151 INFO:teuthology.task.kernel:Running kernel on smithi114: 5.15.0-56-generic 2024-09-16T04:54:33.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo apt-get clean 2024-09-16T04:54:33.158 INFO:teuthology.orchestra.run.smithi149.stdout:5.15.0-56-generic 2024-09-16T04:54:33.158 INFO:teuthology.task.kernel:Running kernel on smithi149: 5.15.0-56-generic 2024-09-16T04:54:33.159 DEBUG:teuthology.orchestra.run.smithi149:> sudo apt-get clean 2024-09-16T04:54:33.278 DEBUG:teuthology.orchestra.run.smithi114:> sudo apt-get update 2024-09-16T04:54:33.293 DEBUG:teuthology.orchestra.run.smithi149:> sudo apt-get update 2024-09-16T04:54:33.549 INFO:teuthology.orchestra.run.smithi114.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:54:33.550 INFO:teuthology.orchestra.run.smithi114.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T04:54:33.574 INFO:teuthology.orchestra.run.smithi149.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:54:33.574 INFO:teuthology.orchestra.run.smithi149.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T04:54:33.617 INFO:teuthology.orchestra.run.smithi114.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T04:54:33.624 INFO:teuthology.orchestra.run.smithi149.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T04:54:33.971 INFO:teuthology.orchestra.run.smithi114.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T04:54:33.978 INFO:teuthology.orchestra.run.smithi149.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T04:54:33.996 INFO:teuthology.orchestra.run.smithi114.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T04:54:33.999 INFO:teuthology.orchestra.run.smithi149.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T04:54:34.109 INFO:teuthology.orchestra.run.smithi114.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T04:54:34.115 INFO:teuthology.orchestra.run.smithi149.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T04:54:34.289 INFO:teuthology.orchestra.run.smithi114.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T04:54:34.295 INFO:teuthology.orchestra.run.smithi149.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T04:54:34.302 INFO:teuthology.orchestra.run.smithi114.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T04:54:34.309 INFO:teuthology.orchestra.run.smithi149.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T04:54:34.461 INFO:teuthology.orchestra.run.smithi114.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T04:54:34.462 INFO:teuthology.orchestra.run.smithi114.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T04:54:34.469 INFO:teuthology.orchestra.run.smithi149.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T04:54:34.470 INFO:teuthology.orchestra.run.smithi149.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T04:54:34.475 INFO:teuthology.orchestra.run.smithi114.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T04:54:34.475 INFO:teuthology.orchestra.run.smithi114.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T04:54:34.475 INFO:teuthology.orchestra.run.smithi114.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T04:54:34.476 INFO:teuthology.orchestra.run.smithi114.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T04:54:34.482 INFO:teuthology.orchestra.run.smithi149.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T04:54:34.483 INFO:teuthology.orchestra.run.smithi149.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T04:54:34.485 INFO:teuthology.orchestra.run.smithi149.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T04:54:34.485 INFO:teuthology.orchestra.run.smithi149.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T04:54:34.486 INFO:teuthology.orchestra.run.smithi149.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T04:54:34.542 INFO:teuthology.orchestra.run.smithi149.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T04:54:34.546 INFO:teuthology.orchestra.run.smithi114.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T04:54:34.547 INFO:teuthology.orchestra.run.smithi114.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T04:54:34.569 INFO:teuthology.orchestra.run.smithi149.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T04:54:34.573 INFO:teuthology.orchestra.run.smithi114.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T04:54:34.573 INFO:teuthology.orchestra.run.smithi114.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T04:54:34.574 INFO:teuthology.orchestra.run.smithi114.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T04:54:34.574 INFO:teuthology.orchestra.run.smithi114.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T04:54:34.577 INFO:teuthology.orchestra.run.smithi149.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T04:54:34.577 INFO:teuthology.orchestra.run.smithi149.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T04:54:34.582 INFO:teuthology.orchestra.run.smithi114.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T04:54:34.582 INFO:teuthology.orchestra.run.smithi114.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T04:54:34.586 INFO:teuthology.orchestra.run.smithi149.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T04:54:34.593 INFO:teuthology.orchestra.run.smithi114.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T04:54:34.593 INFO:teuthology.orchestra.run.smithi114.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T04:54:34.594 INFO:teuthology.orchestra.run.smithi149.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T04:54:34.600 INFO:teuthology.orchestra.run.smithi149.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T04:54:34.601 INFO:teuthology.orchestra.run.smithi149.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T04:54:34.613 INFO:teuthology.orchestra.run.smithi114.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T04:54:34.619 INFO:teuthology.orchestra.run.smithi114.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T04:54:34.620 INFO:teuthology.orchestra.run.smithi114.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T04:54:34.625 INFO:teuthology.orchestra.run.smithi114.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T04:54:34.625 INFO:teuthology.orchestra.run.smithi114.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T04:54:34.625 INFO:teuthology.orchestra.run.smithi114.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T04:54:34.628 INFO:teuthology.orchestra.run.smithi149.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T04:54:34.628 INFO:teuthology.orchestra.run.smithi114.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T04:54:34.628 INFO:teuthology.orchestra.run.smithi114.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T04:54:34.628 INFO:teuthology.orchestra.run.smithi149.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T04:54:34.634 INFO:teuthology.orchestra.run.smithi149.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T04:54:34.637 INFO:teuthology.orchestra.run.smithi149.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T04:54:34.637 INFO:teuthology.orchestra.run.smithi149.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T04:54:34.637 INFO:teuthology.orchestra.run.smithi149.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T04:54:34.649 INFO:teuthology.orchestra.run.smithi149.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T04:54:34.684 INFO:teuthology.orchestra.run.smithi149.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T04:54:34.684 INFO:teuthology.orchestra.run.smithi149.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T04:54:34.684 INFO:teuthology.orchestra.run.smithi149.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T04:54:34.685 INFO:teuthology.orchestra.run.smithi149.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T04:54:34.695 INFO:teuthology.orchestra.run.smithi114.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T04:54:34.695 INFO:teuthology.orchestra.run.smithi114.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T04:54:34.696 INFO:teuthology.orchestra.run.smithi114.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T04:54:34.697 INFO:teuthology.orchestra.run.smithi114.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T04:54:34.698 INFO:teuthology.orchestra.run.smithi114.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T04:54:34.706 INFO:teuthology.orchestra.run.smithi114.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T04:54:34.707 INFO:teuthology.orchestra.run.smithi114.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T04:54:34.707 INFO:teuthology.orchestra.run.smithi114.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T04:54:34.707 INFO:teuthology.orchestra.run.smithi114.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T04:54:34.708 INFO:teuthology.orchestra.run.smithi114.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T04:54:34.715 INFO:teuthology.orchestra.run.smithi149.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T04:54:34.716 INFO:teuthology.orchestra.run.smithi149.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T04:54:34.717 INFO:teuthology.orchestra.run.smithi149.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T04:54:34.718 INFO:teuthology.orchestra.run.smithi149.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T04:54:34.720 INFO:teuthology.orchestra.run.smithi149.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T04:54:34.788 INFO:teuthology.orchestra.run.smithi114.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T04:54:34.788 INFO:teuthology.orchestra.run.smithi114.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T04:54:34.798 INFO:teuthology.orchestra.run.smithi149.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T04:54:34.798 INFO:teuthology.orchestra.run.smithi149.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T04:54:34.799 INFO:teuthology.orchestra.run.smithi149.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T04:54:34.800 INFO:teuthology.orchestra.run.smithi149.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T04:54:34.800 INFO:teuthology.orchestra.run.smithi149.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T04:54:40.674 INFO:teuthology.orchestra.run.smithi149.stdout:Fetched 16.0 MB in 3s (5,872 kB/s) 2024-09-16T04:54:40.676 INFO:teuthology.orchestra.run.smithi114.stdout:Fetched 16.0 MB in 3s (5,865 kB/s) 2024-09-16T04:54:42.108 INFO:teuthology.orchestra.run.smithi114.stdout:Reading package lists... 2024-09-16T04:54:42.111 INFO:teuthology.orchestra.run.smithi149.stdout:Reading package lists... 2024-09-16T04:54:42.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T04:54:42.129 DEBUG:teuthology.orchestra.run.smithi149:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T04:54:42.191 INFO:teuthology.orchestra.run.smithi149.stdout:Reading package lists... 2024-09-16T04:54:42.193 INFO:teuthology.orchestra.run.smithi114.stdout:Reading package lists... 2024-09-16T04:54:42.379 INFO:teuthology.orchestra.run.smithi149.stdout:Building dependency tree... 2024-09-16T04:54:42.379 INFO:teuthology.orchestra.run.smithi149.stdout:Reading state information... 2024-09-16T04:54:42.402 INFO:teuthology.orchestra.run.smithi114.stdout:Building dependency tree... 2024-09-16T04:54:42.403 INFO:teuthology.orchestra.run.smithi114.stdout:Reading state information... 2024-09-16T04:54:42.552 INFO:teuthology.orchestra.run.smithi149.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:54:42.552 INFO:teuthology.orchestra.run.smithi149.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:54:42.552 INFO:teuthology.orchestra.run.smithi149.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:54:42.553 INFO:teuthology.orchestra.run.smithi149.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:54:42.553 INFO:teuthology.orchestra.run.smithi149.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T04:54:42.553 INFO:teuthology.orchestra.run.smithi149.stdout: python2.7-minimal 2024-09-16T04:54:42.553 INFO:teuthology.orchestra.run.smithi149.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:54:42.554 INFO:teuthology.orchestra.run.smithi149.stdout:The following additional packages will be installed: 2024-09-16T04:54:42.554 INFO:teuthology.orchestra.run.smithi149.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T04:54:42.554 INFO:teuthology.orchestra.run.smithi149.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T04:54:42.554 INFO:teuthology.orchestra.run.smithi149.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T04:54:42.556 INFO:teuthology.orchestra.run.smithi149.stdout:Suggested packages: 2024-09-16T04:54:42.556 INFO:teuthology.orchestra.run.smithi149.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T04:54:42.556 INFO:teuthology.orchestra.run.smithi149.stdout:Recommended packages: 2024-09-16T04:54:42.556 INFO:teuthology.orchestra.run.smithi149.stdout: thermald 2024-09-16T04:54:42.591 INFO:teuthology.orchestra.run.smithi114.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:54:42.591 INFO:teuthology.orchestra.run.smithi114.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:54:42.591 INFO:teuthology.orchestra.run.smithi114.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:54:42.592 INFO:teuthology.orchestra.run.smithi114.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:54:42.592 INFO:teuthology.orchestra.run.smithi114.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T04:54:42.592 INFO:teuthology.orchestra.run.smithi114.stdout: python2.7-minimal 2024-09-16T04:54:42.592 INFO:teuthology.orchestra.run.smithi114.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:54:42.593 INFO:teuthology.orchestra.run.smithi114.stdout:The following additional packages will be installed: 2024-09-16T04:54:42.593 INFO:teuthology.orchestra.run.smithi114.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T04:54:42.593 INFO:teuthology.orchestra.run.smithi114.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T04:54:42.594 INFO:teuthology.orchestra.run.smithi114.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T04:54:42.595 INFO:teuthology.orchestra.run.smithi114.stdout:Suggested packages: 2024-09-16T04:54:42.595 INFO:teuthology.orchestra.run.smithi114.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T04:54:42.595 INFO:teuthology.orchestra.run.smithi114.stdout:Recommended packages: 2024-09-16T04:54:42.595 INFO:teuthology.orchestra.run.smithi114.stdout: thermald 2024-09-16T04:54:42.608 INFO:teuthology.orchestra.run.smithi149.stdout:The following NEW packages will be installed: 2024-09-16T04:54:42.608 INFO:teuthology.orchestra.run.smithi149.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T04:54:42.608 INFO:teuthology.orchestra.run.smithi149.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T04:54:42.609 INFO:teuthology.orchestra.run.smithi149.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T04:54:42.610 INFO:teuthology.orchestra.run.smithi149.stdout:The following packages will be upgraded: 2024-09-16T04:54:42.611 INFO:teuthology.orchestra.run.smithi149.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T04:54:42.645 INFO:teuthology.orchestra.run.smithi114.stdout:The following NEW packages will be installed: 2024-09-16T04:54:42.645 INFO:teuthology.orchestra.run.smithi114.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T04:54:42.645 INFO:teuthology.orchestra.run.smithi114.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T04:54:42.645 INFO:teuthology.orchestra.run.smithi114.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T04:54:42.646 INFO:teuthology.orchestra.run.smithi114.stdout:The following packages will be upgraded: 2024-09-16T04:54:42.647 INFO:teuthology.orchestra.run.smithi114.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T04:54:42.700 INFO:teuthology.orchestra.run.smithi114.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T04:54:42.700 INFO:teuthology.orchestra.run.smithi114.stdout:Need to get 113 MB of archives. 2024-09-16T04:54:42.700 INFO:teuthology.orchestra.run.smithi114.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T04:54:42.700 INFO:teuthology.orchestra.run.smithi114.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T04:54:42.790 INFO:teuthology.orchestra.run.smithi149.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T04:54:42.790 INFO:teuthology.orchestra.run.smithi149.stdout:Need to get 113 MB of archives. 2024-09-16T04:54:42.790 INFO:teuthology.orchestra.run.smithi149.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T04:54:42.790 INFO:teuthology.orchestra.run.smithi149.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T04:54:43.125 INFO:teuthology.orchestra.run.smithi114.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T04:54:43.214 INFO:teuthology.orchestra.run.smithi114.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T04:54:44.029 INFO:teuthology.orchestra.run.smithi114.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T04:54:44.029 INFO:teuthology.orchestra.run.smithi114.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T04:54:44.030 INFO:teuthology.orchestra.run.smithi114.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T04:54:44.156 INFO:teuthology.orchestra.run.smithi114.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T04:54:44.177 INFO:teuthology.orchestra.run.smithi114.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T04:54:44.194 INFO:teuthology.orchestra.run.smithi149.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T04:54:44.551 INFO:teuthology.orchestra.run.smithi149.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T04:54:44.679 INFO:teuthology.orchestra.run.smithi114.stdout:Fetched 113 MB in 2s (70.9 MB/s) 2024-09-16T04:54:44.838 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T04:54:46.765 INFO:teuthology.orchestra.run.smithi149.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T04:54:46.765 INFO:teuthology.orchestra.run.smithi149.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T04:54:46.765 INFO:teuthology.orchestra.run.smithi149.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T04:54:46.912 INFO:teuthology.orchestra.run.smithi114.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T04:54:46.916 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:54:47.076 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:54:47.126 INFO:teuthology.orchestra.run.smithi149.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T04:54:47.208 INFO:teuthology.orchestra.run.smithi149.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T04:54:47.603 INFO:teuthology.orchestra.run.smithi149.stdout:Fetched 113 MB in 5s (24.7 MB/s) 2024-09-16T04:54:47.807 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T04:54:49.669 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T04:54:49.687 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:54:49.782 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:54:49.783 INFO:teuthology.orchestra.run.smithi149.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T04:54:49.788 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:54:49.880 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:54:50.264 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T04:54:50.285 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:54:50.322 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:54:52.298 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T04:54:52.319 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:54:52.410 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:54:52.885 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T04:54:52.905 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:54:52.935 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:54:56.638 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:54:56.765 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:54:57.081 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:54:57.185 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:54:57.480 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T04:54:57.502 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T04:54:57.546 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T04:54:59.495 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:54:59.621 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:54:59.913 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:55:00.034 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:55:00.362 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T04:55:00.384 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T04:55:00.428 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T04:55:02.779 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T04:55:02.816 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:55:02.854 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:55:06.015 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:55:06.140 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:55:06.473 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T04:55:06.577 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T04:55:06.595 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:55:06.614 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T04:55:06.652 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:55:06.721 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T04:55:06.916 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:55:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:55:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T04:55:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:55:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T04:55:08.394 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:55:10.194 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:55:10.284 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T04:55:10.398 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T04:55:10.764 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T04:55:10.886 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:55:11.012 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T04:55:11.140 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:55:11.604 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T04:55:11.705 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T04:55:11.831 INFO:teuthology.orchestra.run.smithi114.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:55:11.886 INFO:teuthology.orchestra.run.smithi114.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T04:55:11.886 INFO:teuthology.orchestra.run.smithi114.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:55:12.487 INFO:teuthology.orchestra.run.smithi149.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:55:12.487 INFO:teuthology.orchestra.run.smithi149.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T04:55:12.487 INFO:teuthology.orchestra.run.smithi149.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:55:12.487 INFO:teuthology.orchestra.run.smithi149.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T04:55:12.635 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:55:14.378 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:55:16.138 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T04:55:16.247 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T04:55:16.357 INFO:teuthology.orchestra.run.smithi149.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T04:55:16.421 INFO:teuthology.orchestra.run.smithi149.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T04:55:16.421 INFO:teuthology.orchestra.run.smithi149.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:55:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T04:55:26.031 INFO:teuthology.orchestra.run.smithi114.stdout:Sourcing file `/etc/default/grub' 2024-09-16T04:55:26.042 INFO:teuthology.orchestra.run.smithi114.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T04:55:26.059 INFO:teuthology.orchestra.run.smithi114.stdout:Generating grub configuration file ... 2024-09-16T04:55:26.299 INFO:teuthology.orchestra.run.smithi114.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:55:26.325 INFO:teuthology.orchestra.run.smithi114.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:55:26.575 INFO:teuthology.orchestra.run.smithi114.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:55:26.581 INFO:teuthology.orchestra.run.smithi114.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T04:55:26.650 INFO:teuthology.orchestra.run.smithi114.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T04:55:26.655 INFO:teuthology.orchestra.run.smithi114.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T04:55:26.811 INFO:teuthology.orchestra.run.smithi114.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T04:55:26.811 INFO:teuthology.orchestra.run.smithi114.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T04:55:26.811 INFO:teuthology.orchestra.run.smithi114.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T04:55:26.849 INFO:teuthology.orchestra.run.smithi114.stdout:done 2024-09-16T04:55:27.299 DEBUG:teuthology.orchestra.run.smithi114:> dpkg -s linux-image-generic 2024-09-16T04:55:27.331 INFO:teuthology.orchestra.run.smithi114.stdout:Package: linux-image-generic 2024-09-16T04:55:27.331 INFO:teuthology.orchestra.run.smithi114.stdout:Status: install ok installed 2024-09-16T04:55:27.331 INFO:teuthology.orchestra.run.smithi114.stdout:Priority: optional 2024-09-16T04:55:27.331 INFO:teuthology.orchestra.run.smithi114.stdout:Section: kernel 2024-09-16T04:55:27.332 INFO:teuthology.orchestra.run.smithi114.stdout:Installed-Size: 21 2024-09-16T04:55:27.332 INFO:teuthology.orchestra.run.smithi114.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T04:55:27.332 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: amd64 2024-09-16T04:55:27.332 INFO:teuthology.orchestra.run.smithi114.stdout:Source: linux-meta 2024-09-16T04:55:27.332 INFO:teuthology.orchestra.run.smithi114.stdout:Version: 5.15.0.121.121 2024-09-16T04:55:27.332 INFO:teuthology.orchestra.run.smithi114.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T04:55:27.332 INFO:teuthology.orchestra.run.smithi114.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T04:55:27.332 INFO:teuthology.orchestra.run.smithi114.stdout:Recommends: thermald 2024-09-16T04:55:27.332 INFO:teuthology.orchestra.run.smithi114.stdout:Description: Generic Linux kernel image 2024-09-16T04:55:27.332 INFO:teuthology.orchestra.run.smithi114.stdout: This package will always depend on the latest generic kernel image 2024-09-16T04:55:27.332 INFO:teuthology.orchestra.run.smithi114.stdout: available. 2024-09-16T04:55:27.333 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T04:55:27.333 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T04:55:27.333 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T04:55:27.333 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T04:55:27.333 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T04:55:27.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo apt-get clean 2024-09-16T04:55:27.529 DEBUG:teuthology.orchestra.run.smithi114:> sudo apt-get update 2024-09-16T04:55:27.660 INFO:teuthology.orchestra.run.smithi114.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T04:55:27.781 INFO:teuthology.orchestra.run.smithi114.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:55:27.865 INFO:teuthology.orchestra.run.smithi114.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T04:55:27.943 INFO:teuthology.orchestra.run.smithi114.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T04:55:29.128 INFO:teuthology.orchestra.run.smithi114.stdout:Reading package lists... 2024-09-16T04:55:29.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T04:55:29.200 INFO:teuthology.orchestra.run.smithi114.stdout:Reading package lists... 2024-09-16T04:55:29.385 INFO:teuthology.orchestra.run.smithi114.stdout:Building dependency tree... 2024-09-16T04:55:29.386 INFO:teuthology.orchestra.run.smithi114.stdout:Reading state information... 2024-09-16T04:55:29.553 INFO:teuthology.orchestra.run.smithi114.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T04:55:29.553 INFO:teuthology.orchestra.run.smithi114.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:55:29.553 INFO:teuthology.orchestra.run.smithi114.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:55:29.553 INFO:teuthology.orchestra.run.smithi114.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:55:29.554 INFO:teuthology.orchestra.run.smithi114.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:55:29.554 INFO:teuthology.orchestra.run.smithi114.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T04:55:29.554 INFO:teuthology.orchestra.run.smithi114.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T04:55:29.554 INFO:teuthology.orchestra.run.smithi114.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:55:29.592 INFO:teuthology.orchestra.run.smithi114.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T04:55:29.594 DEBUG:teuthology.orchestra.run.smithi114:> dpkg -s linux-image-generic 2024-09-16T04:55:29.657 INFO:teuthology.orchestra.run.smithi114.stdout:Package: linux-image-generic 2024-09-16T04:55:29.657 INFO:teuthology.orchestra.run.smithi114.stdout:Status: install ok installed 2024-09-16T04:55:29.657 INFO:teuthology.orchestra.run.smithi114.stdout:Priority: optional 2024-09-16T04:55:29.658 INFO:teuthology.orchestra.run.smithi114.stdout:Section: kernel 2024-09-16T04:55:29.658 INFO:teuthology.orchestra.run.smithi114.stdout:Installed-Size: 21 2024-09-16T04:55:29.658 INFO:teuthology.orchestra.run.smithi114.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T04:55:29.658 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: amd64 2024-09-16T04:55:29.658 INFO:teuthology.orchestra.run.smithi114.stdout:Source: linux-meta 2024-09-16T04:55:29.658 INFO:teuthology.orchestra.run.smithi114.stdout:Version: 5.15.0.121.121 2024-09-16T04:55:29.658 INFO:teuthology.orchestra.run.smithi114.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T04:55:29.658 INFO:teuthology.orchestra.run.smithi114.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T04:55:29.658 INFO:teuthology.orchestra.run.smithi114.stdout:Recommends: thermald 2024-09-16T04:55:29.658 INFO:teuthology.orchestra.run.smithi114.stdout:Description: Generic Linux kernel image 2024-09-16T04:55:29.658 INFO:teuthology.orchestra.run.smithi114.stdout: This package will always depend on the latest generic kernel image 2024-09-16T04:55:29.658 INFO:teuthology.orchestra.run.smithi114.stdout: available. 2024-09-16T04:55:29.659 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-09-16T04:55:29.706 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.13uFlAGOdo 2024-09-16T04:55:29.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub/grub.cfg /tmp/tmp.13uFlAGOdo 2024-09-16T04:55:29.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.13uFlAGOdo 2024-09-16T04:55:29.892 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.13uFlAGOdo is 10KB 2024-09-16T04:55:29.905 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.13uFlAGOdo 2024-09-16T04:55:29.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T04:55:29.966 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:55:29.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T04:55:29.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T04:55:30.032 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T04:55:30.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo update-grub 2024-09-16T04:55:30.717 INFO:teuthology.orchestra.run.smithi149.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T04:55:30.717 INFO:teuthology.orchestra.run.smithi149.stdout:Sourcing file `/etc/default/grub' 2024-09-16T04:55:30.729 INFO:teuthology.orchestra.run.smithi149.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T04:55:30.753 INFO:teuthology.orchestra.run.smithi149.stdout:Generating grub configuration file ... 2024-09-16T04:55:31.037 INFO:teuthology.orchestra.run.smithi149.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:55:31.055 INFO:teuthology.orchestra.run.smithi149.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:55:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:Sourcing file `/etc/default/grub' 2024-09-16T04:55:31.240 INFO:teuthology.orchestra.run.smithi114.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T04:55:31.241 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-09-16T04:55:31.363 INFO:teuthology.orchestra.run.smithi149.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:55:31.369 INFO:teuthology.orchestra.run.smithi149.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T04:55:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:55:31.426 INFO:teuthology.orchestra.run.smithi114.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:55:31.449 INFO:teuthology.orchestra.run.smithi149.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T04:55:31.455 INFO:teuthology.orchestra.run.smithi149.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T04:55:31.616 INFO:teuthology.orchestra.run.smithi149.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T04:55:31.616 INFO:teuthology.orchestra.run.smithi149.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T04:55:31.616 INFO:teuthology.orchestra.run.smithi149.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T04:55:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:55:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T04:55:31.666 INFO:teuthology.orchestra.run.smithi149.stdout:done 2024-09-16T04:55:31.707 INFO:teuthology.orchestra.run.smithi114.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T04:55:31.712 INFO:teuthology.orchestra.run.smithi114.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T04:55:31.851 INFO:teuthology.orchestra.run.smithi114.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T04:55:31.852 INFO:teuthology.orchestra.run.smithi114.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T04:55:31.852 INFO:teuthology.orchestra.run.smithi114.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T04:55:31.862 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-09-16T04:55:31.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-09-16T04:55:32.187 DEBUG:teuthology.orchestra.run.smithi149:> dpkg -s linux-image-generic 2024-09-16T04:55:32.205 INFO:teuthology.orchestra.run.smithi149.stdout:Package: linux-image-generic 2024-09-16T04:55:32.205 INFO:teuthology.orchestra.run.smithi149.stdout:Status: install ok installed 2024-09-16T04:55:32.205 INFO:teuthology.orchestra.run.smithi149.stdout:Priority: optional 2024-09-16T04:55:32.205 INFO:teuthology.orchestra.run.smithi149.stdout:Section: kernel 2024-09-16T04:55:32.205 INFO:teuthology.orchestra.run.smithi149.stdout:Installed-Size: 21 2024-09-16T04:55:32.205 INFO:teuthology.orchestra.run.smithi149.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T04:55:32.206 INFO:teuthology.orchestra.run.smithi149.stdout:Architecture: amd64 2024-09-16T04:55:32.206 INFO:teuthology.orchestra.run.smithi149.stdout:Source: linux-meta 2024-09-16T04:55:32.206 INFO:teuthology.orchestra.run.smithi149.stdout:Version: 5.15.0.121.121 2024-09-16T04:55:32.206 INFO:teuthology.orchestra.run.smithi149.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T04:55:32.206 INFO:teuthology.orchestra.run.smithi149.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T04:55:32.206 INFO:teuthology.orchestra.run.smithi149.stdout:Recommends: thermald 2024-09-16T04:55:32.206 INFO:teuthology.orchestra.run.smithi149.stdout:Description: Generic Linux kernel image 2024-09-16T04:55:32.206 INFO:teuthology.orchestra.run.smithi149.stdout: This package will always depend on the latest generic kernel image 2024-09-16T04:55:32.206 INFO:teuthology.orchestra.run.smithi149.stdout: available. 2024-09-16T04:55:32.206 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T04:55:32.206 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T04:55:32.206 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T04:55:32.207 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T04:55:32.207 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi149.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T04:55:32.207 DEBUG:teuthology.orchestra.run.smithi149:> sudo apt-get clean 2024-09-16T04:55:32.370 DEBUG:teuthology.orchestra.run.smithi149:> sudo apt-get update 2024-09-16T04:55:32.494 INFO:teuthology.orchestra.run.smithi149.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T04:55:32.615 INFO:teuthology.orchestra.run.smithi149.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T04:55:32.708 INFO:teuthology.orchestra.run.smithi149.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T04:55:32.779 INFO:teuthology.orchestra.run.smithi149.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T04:55:33.963 INFO:teuthology.orchestra.run.smithi149.stdout:Reading package lists... 2024-09-16T04:55:33.982 DEBUG:teuthology.orchestra.run.smithi149:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T04:55:34.046 INFO:teuthology.orchestra.run.smithi149.stdout:Reading package lists... 2024-09-16T04:55:34.258 INFO:teuthology.orchestra.run.smithi149.stdout:Building dependency tree... 2024-09-16T04:55:34.259 INFO:teuthology.orchestra.run.smithi149.stdout:Reading state information... 2024-09-16T04:55:34.448 INFO:teuthology.orchestra.run.smithi149.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T04:55:34.448 INFO:teuthology.orchestra.run.smithi149.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T04:55:34.449 INFO:teuthology.orchestra.run.smithi149.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T04:55:34.449 INFO:teuthology.orchestra.run.smithi149.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T04:55:34.449 INFO:teuthology.orchestra.run.smithi149.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T04:55:34.449 INFO:teuthology.orchestra.run.smithi149.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T04:55:34.449 INFO:teuthology.orchestra.run.smithi149.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T04:55:34.449 INFO:teuthology.orchestra.run.smithi149.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T04:55:34.495 INFO:teuthology.orchestra.run.smithi149.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T04:55:34.496 DEBUG:teuthology.orchestra.run.smithi149:> dpkg -s linux-image-generic 2024-09-16T04:55:34.511 INFO:teuthology.orchestra.run.smithi149.stdout:Package: linux-image-generic 2024-09-16T04:55:34.511 INFO:teuthology.orchestra.run.smithi149.stdout:Status: install ok installed 2024-09-16T04:55:34.511 INFO:teuthology.orchestra.run.smithi149.stdout:Priority: optional 2024-09-16T04:55:34.511 INFO:teuthology.orchestra.run.smithi149.stdout:Section: kernel 2024-09-16T04:55:34.511 INFO:teuthology.orchestra.run.smithi149.stdout:Installed-Size: 21 2024-09-16T04:55:34.512 INFO:teuthology.orchestra.run.smithi149.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T04:55:34.512 INFO:teuthology.orchestra.run.smithi149.stdout:Architecture: amd64 2024-09-16T04:55:34.512 INFO:teuthology.orchestra.run.smithi149.stdout:Source: linux-meta 2024-09-16T04:55:34.512 INFO:teuthology.orchestra.run.smithi149.stdout:Version: 5.15.0.121.121 2024-09-16T04:55:34.512 INFO:teuthology.orchestra.run.smithi149.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T04:55:34.512 INFO:teuthology.orchestra.run.smithi149.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T04:55:34.512 INFO:teuthology.orchestra.run.smithi149.stdout:Recommends: thermald 2024-09-16T04:55:34.512 INFO:teuthology.orchestra.run.smithi149.stdout:Description: Generic Linux kernel image 2024-09-16T04:55:34.512 INFO:teuthology.orchestra.run.smithi149.stdout: This package will always depend on the latest generic kernel image 2024-09-16T04:55:34.512 INFO:teuthology.orchestra.run.smithi149.stdout: available. 2024-09-16T04:55:34.512 DEBUG:teuthology.orchestra.run.smithi149:> mktemp 2024-09-16T04:55:34.557 INFO:teuthology.orchestra.run.smithi149.stdout:/tmp/tmp.1GEUytoLBj 2024-09-16T04:55:34.558 DEBUG:teuthology.orchestra.run.smithi149:> sudo cp /boot/grub/grub.cfg /tmp/tmp.1GEUytoLBj 2024-09-16T04:55:34.613 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0666 /tmp/tmp.1GEUytoLBj 2024-09-16T04:55:34.725 DEBUG:teuthology.orchestra.remote:smithi149:/tmp/tmp.1GEUytoLBj is 10KB 2024-09-16T04:55:34.738 DEBUG:teuthology.orchestra.run.smithi149:> rm -fr /tmp/tmp.1GEUytoLBj 2024-09-16T04:55:34.743 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T04:55:34.798 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T04:55:34.798 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T04:55:34.798 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T04:55:34.865 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T04:55:34.865 DEBUG:teuthology.orchestra.run.smithi149:> sudo update-grub 2024-09-16T04:55:36.128 INFO:teuthology.orchestra.run.smithi149.stderr:Sourcing file `/etc/default/grub' 2024-09-16T04:55:36.130 INFO:teuthology.orchestra.run.smithi149.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T04:55:36.132 INFO:teuthology.orchestra.run.smithi149.stderr:Generating grub configuration file ... 2024-09-16T04:55:36.354 INFO:teuthology.orchestra.run.smithi149.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T04:55:36.360 INFO:teuthology.orchestra.run.smithi149.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T04:55:36.607 INFO:teuthology.orchestra.run.smithi149.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T04:55:36.612 INFO:teuthology.orchestra.run.smithi149.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T04:55:36.652 INFO:teuthology.orchestra.run.smithi149.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T04:55:36.657 INFO:teuthology.orchestra.run.smithi149.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T04:55:36.784 INFO:teuthology.orchestra.run.smithi149.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T04:55:36.784 INFO:teuthology.orchestra.run.smithi149.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T04:55:36.784 INFO:teuthology.orchestra.run.smithi149.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T04:55:36.797 INFO:teuthology.orchestra.run.smithi149.stderr:done 2024-09-16T04:55:36.808 DEBUG:teuthology.orchestra.run.smithi149:> sudo shutdown -r now 2024-09-16T04:56:01.905 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T04:56:01.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T04:56:01.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:56:06.817 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T04:56:06.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-09-16T04:56:06.817 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:56:20.285 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-09-16T04:56:25.149 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.149 2024-09-16T04:56:29.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T04:56:29.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:56:32.349 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-09-16T04:56:34.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-09-16T04:56:34.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:56:37.217 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.149 2024-09-16T04:56:44.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T04:56:44.358 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:56:49.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-09-16T04:56:49.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:57:02.781 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-09-16T04:57:07.645 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.149 2024-09-16T04:57:17.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T04:57:17.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:57:18.486 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-09-16T04:57:19.444 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T04:57:19.445 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-16T04:57:19.445 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-09-16T04:57:19.497 INFO:teuthology.orchestra.run.smithi114.stdout:5.15.0-121-generic 2024-09-16T04:57:19.498 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T04:57:19.498 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T04:57:19.498 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T04:57:20.499 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T04:57:20.499 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T04:57:20.608 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2024-09-16T04:57:20.627 DEBUG:teuthology.parallel:result is None 2024-09-16T04:57:22.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-09-16T04:57:22.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:57:23.115 DEBUG:teuthology.orchestra.run.smithi149:> true 2024-09-16T04:57:24.328 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-09-16T04:57:24.328 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-121-generic"... 2024-09-16T04:57:24.328 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2024-09-16T04:57:24.373 INFO:teuthology.orchestra.run.smithi149.stdout:5.15.0-121-generic 2024-09-16T04:57:24.374 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T04:57:24.374 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T04:57:24.374 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T04:57:25.375 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T04:57:25.375 DEBUG:teuthology.orchestra.run.smithi149:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T04:57:25.769 INFO:teuthology.orchestra.run.smithi149.stdout:ttyS1 2024-09-16T04:57:25.800 DEBUG:teuthology.parallel:result is None 2024-09-16T04:57:25.800 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T04:57:25.806 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T04:57:25.807 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T04:57:25.809 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T04:57:25.815 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T04:57:25.897 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T04:57:25.933 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T04:57:25.934 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T04:57:25.936 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T04:57:25.959 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T04:57:25.965 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T04:57:25.966 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T04:57:25.999 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T04:57:26.015 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:57:26.020 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:57:26.023 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:57:26.029 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:57:26.030 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T04:57:26.036 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T04:57:26.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T04:57:26.076 DEBUG:teuthology.orchestra.run.smithi149:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T04:57:26.090 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T04:57:26.105 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T04:57:26.106 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T04:57:26.135 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T04:57:26.140 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:57:26.186 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:57:26.233 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T04:57:26.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T04:57:26.304 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:57:26.309 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:57:26.357 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T04:57:26.357 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T04:57:26.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-09-16T04:57:26.433 DEBUG:teuthology.orchestra.run.smithi149:> sudo service rsyslog restart 2024-09-16T04:57:26.567 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T04:57:26.574 INFO:teuthology.task.internal:Starting timer... 2024-09-16T04:57:26.575 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T04:57:26.620 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T04:57:26.653 DEBUG:teuthology.task.selinux:Excluding smithi114: OS 'ubuntu' does not support SELinux 2024-09-16T04:57:26.653 DEBUG:teuthology.task.selinux:Excluding smithi149: OS 'ubuntu' does not support SELinux 2024-09-16T04:57:26.653 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T04:57:26.653 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T04:57:26.653 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T04:57:26.653 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T04:57:26.685 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T04:57:26.700 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T04:57:26.701 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi114.front.sepia.ceph.com,smithi149.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T05:05:01.788 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi114.front.sepia.ceph.com'), Remote(name='ubuntu@smithi149.front.sepia.ceph.com')] 2024-09-16T05:05:01.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T05:05:01.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:05:01.873 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-09-16T05:05:01.949 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T05:05:01.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-09-16T05:05:01.950 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:05:02.032 DEBUG:teuthology.orchestra.run.smithi149:> true 2024-09-16T05:05:02.104 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-09-16T05:05:02.105 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T05:05:02.115 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T05:05:02.116 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T05:05:02.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T05:05:02.118 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T05:05:02.118 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T05:05:02.144 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T05:05:02.144 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: Command line: ntpd -gq 2024-09-16T05:05:02.144 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: ---------------------------------------------------- 2024-09-16T05:05:02.144 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T05:05:02.144 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T05:05:02.145 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: corporation. Support and training for ntp-4 are 2024-09-16T05:05:02.145 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: available at https://www.nwtime.org/support 2024-09-16T05:05:02.145 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: ---------------------------------------------------- 2024-09-16T05:05:02.145 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: proto: precision = 0.060 usec (-24) 2024-09-16T05:05:02.145 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: basedate set to 2022-02-04 2024-09-16T05:05:02.145 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: gps base set to 2022-02-06 (week 2196) 2024-09-16T05:05:02.146 INFO:teuthology.orchestra.run.smithi114.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T05:05:02.146 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T05:05:02.146 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T05:05:02.146 INFO:teuthology.orchestra.run.smithi114.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T05:05:02.147 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T05:05:02.147 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T05:05:02.147 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T05:05:02.147 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: Listen normally on 3 enp3s0f1 172.21.15.114:123 2024-09-16T05:05:02.147 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: Listen normally on 4 lo [::1]:123 2024-09-16T05:05:02.148 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cb63%5]:123 2024-09-16T05:05:02.148 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:02 ntpd[18486]: Listening on routing socket on fd #22 for interface updates 2024-09-16T05:05:02.174 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T05:05:02.174 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: Command line: ntpd -gq 2024-09-16T05:05:02.174 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: ---------------------------------------------------- 2024-09-16T05:05:02.174 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T05:05:02.174 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T05:05:02.174 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: corporation. Support and training for ntp-4 are 2024-09-16T05:05:02.174 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: available at https://www.nwtime.org/support 2024-09-16T05:05:02.174 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: ---------------------------------------------------- 2024-09-16T05:05:02.175 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: proto: precision = 0.057 usec (-24) 2024-09-16T05:05:02.175 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: basedate set to 2022-02-04 2024-09-16T05:05:02.175 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: gps base set to 2022-02-06 (week 2196) 2024-09-16T05:05:02.175 INFO:teuthology.orchestra.run.smithi149.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T05:05:02.176 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T05:05:02.176 INFO:teuthology.orchestra.run.smithi149.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T05:05:02.177 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T05:05:02.177 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T05:05:02.177 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T05:05:02.177 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T05:05:02.177 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: Listen normally on 3 enp3s0f1 172.21.15.149:123 2024-09-16T05:05:02.177 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: Listen normally on 4 lo [::1]:123 2024-09-16T05:05:02.177 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7d57%5]:123 2024-09-16T05:05:02.177 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:02 ntpd[18469]: Listening on routing socket on fd #22 for interface updates 2024-09-16T05:05:03.145 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:03.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:03.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:03.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:03.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:03.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:03.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:03 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:03.174 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:03 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:03.174 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:03 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:03.174 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:03 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:03.174 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:03 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:03.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:03 ntpd[18469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:03.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:03 ntpd[18469]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:04.174 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:04 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:04.174 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:04 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:04.174 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:04 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:04.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:04 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:04.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:04 ntpd[18469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:04.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:04 ntpd[18469]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:05.145 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:05.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:05.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:05.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:05.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:05.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:05 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:05.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:05 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:05.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:05 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:05.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:05 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:05.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:05 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:05.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:05 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:05.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:05 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:05.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:05 ntpd[18469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:05.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:05 ntpd[18469]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:05.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:05 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:05.176 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:05 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:05.176 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:05 ntpd[18469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:05.176 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:05 ntpd[18469]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:06.144 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:06 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:06.144 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:06 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:06.174 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:06 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:06.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:06 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:06.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:06 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:06.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:06 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:06.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:06 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:06.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:06 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:06.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:06 ntpd[18469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:06.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:06 ntpd[18469]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:07.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:07.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:07.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:07.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:07.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:07.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:07.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:07.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:07.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:07.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:07.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:07.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:07.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:07.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:07.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:07.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:07.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:07.147 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:07 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:07.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:07 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:07.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:07 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:07.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:07 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:07.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:07 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:07.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:07 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:07.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:07 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:07.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:07 ntpd[18469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:07.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:07 ntpd[18469]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:07.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:07 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:07.176 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:07 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:07.176 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:07 ntpd[18469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:07.176 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:07 ntpd[18469]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:08.174 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:08 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:08.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:08 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:08.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:08 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:08.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:08 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:08.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:08 ntpd[18469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:08.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:08 ntpd[18469]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:09 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:09 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:09 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:09 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:09 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:09 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:09 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:09 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:09.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:09 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:09.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:09 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:09.146 INFO:teuthology.orchestra.run.smithi114.stdout:16 Sep 05:05:09 ntpd[18486]: ntpd: time slew +0.001102 s 2024-09-16T05:05:09.146 INFO:teuthology.orchestra.run.smithi114.stdout:ntpd: time slew +0.001102s 2024-09-16T05:05:09.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:09 ntpd[18486]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T05:05:09.146 INFO:teuthology.orchestra.run.smithi114.stderr:16 Sep 05:05:09 ntpd[18486]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T05:05:09.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:09 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:09.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:09 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:09.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:09 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:09.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:09 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:09.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:09 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:09.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:09 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:09.175 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:09 ntpd[18469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:09.176 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:09 ntpd[18469]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:09.176 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:09 ntpd[18469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:05:09.176 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:09 ntpd[18469]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:05:09.176 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:09 ntpd[18469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:05:09.176 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:09 ntpd[18469]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:05:09.176 INFO:teuthology.orchestra.run.smithi149.stdout:16 Sep 05:05:09 ntpd[18469]: ntpd: time slew -0.002627 s 2024-09-16T05:05:09.176 INFO:teuthology.orchestra.run.smithi149.stdout:ntpd: time slew -0.002627s 2024-09-16T05:05:09.177 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:09 ntpd[18469]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T05:05:09.177 INFO:teuthology.orchestra.run.smithi149.stderr:16 Sep 05:05:09 ntpd[18469]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T05:05:09.202 INFO:teuthology.orchestra.run.smithi114.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T05:05:09.202 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================== 2024-09-16T05:05:09.202 INFO:teuthology.orchestra.run.smithi114.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:05:09.202 INFO:teuthology.orchestra.run.smithi114.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:05:09.202 INFO:teuthology.orchestra.run.smithi114.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:05:09.202 INFO:teuthology.orchestra.run.smithi114.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:05:09.227 INFO:teuthology.orchestra.run.smithi149.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T05:05:09.227 INFO:teuthology.orchestra.run.smithi149.stdout:============================================================================== 2024-09-16T05:05:09.227 INFO:teuthology.orchestra.run.smithi149.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:05:09.228 INFO:teuthology.orchestra.run.smithi149.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:05:09.228 INFO:teuthology.orchestra.run.smithi149.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:05:09.228 INFO:teuthology.orchestra.run.smithi149.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:05:09.228 INFO:teuthology.run_tasks:Running task install... 2024-09-16T05:05:09.237 DEBUG:teuthology.task.install:project ceph 2024-09-16T05:05:09.237 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T05:05:09.237 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T05:05:09.238 INFO:teuthology.task.install:Using flavor: default 2024-09-16T05:05:09.243 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T05:05:09.243 INFO:teuthology.task.install:extra packages: [] 2024-09-16T05:05:09.244 DEBUG:teuthology.orchestra.run.smithi114:> sudo apt-key list | grep Ceph 2024-09-16T05:05:09.244 DEBUG:teuthology.orchestra.run.smithi149:> sudo apt-key list | grep Ceph 2024-09-16T05:05:09.297 INFO:teuthology.orchestra.run.smithi114.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T05:05:09.324 INFO:teuthology.orchestra.run.smithi114.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T05:05:09.324 INFO:teuthology.orchestra.run.smithi114.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T05:05:09.325 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T05:05:09.325 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:05:09.330 INFO:teuthology.orchestra.run.smithi149.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T05:05:09.360 INFO:teuthology.orchestra.run.smithi149.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T05:05:09.360 INFO:teuthology.orchestra.run.smithi149.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T05:05:09.361 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T05:05:09.361 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:05:09.472 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default/ 2024-09-16T05:05:09.472 INFO:teuthology.task.install.deb:Package version is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:05:09.510 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default/ 2024-09-16T05:05:09.511 INFO:teuthology.task.install.deb:Package version is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:05:09.587 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:05:09.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T05:05:09.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo apt-get update 2024-09-16T05:05:09.625 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:05:09.625 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T05:05:09.638 DEBUG:teuthology.orchestra.run.smithi149:> sudo apt-get update 2024-09-16T05:05:09.768 INFO:teuthology.orchestra.run.smithi114.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T05:05:09.769 INFO:teuthology.orchestra.run.smithi114.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T05:05:09.780 INFO:teuthology.orchestra.run.smithi114.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T05:05:09.797 INFO:teuthology.orchestra.run.smithi114.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T05:05:09.823 INFO:teuthology.orchestra.run.smithi114.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T05:05:09.844 INFO:teuthology.orchestra.run.smithi114.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T05:05:09.861 INFO:teuthology.orchestra.run.smithi149.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T05:05:09.866 INFO:teuthology.orchestra.run.smithi114.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T05:05:09.883 INFO:teuthology.orchestra.run.smithi149.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T05:05:09.886 INFO:teuthology.orchestra.run.smithi114.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-09-16T05:05:09.906 INFO:teuthology.orchestra.run.smithi149.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T05:05:09.913 INFO:teuthology.orchestra.run.smithi114.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-16T05:05:09.938 INFO:teuthology.orchestra.run.smithi149.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T05:05:09.938 INFO:teuthology.orchestra.run.smithi149.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T05:05:09.938 INFO:teuthology.orchestra.run.smithi149.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-16T05:05:09.967 INFO:teuthology.orchestra.run.smithi149.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-09-16T05:05:10.008 INFO:teuthology.orchestra.run.smithi149.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T05:05:10.090 INFO:teuthology.orchestra.run.smithi149.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T05:05:10.109 INFO:teuthology.orchestra.run.smithi114.stdout:Fetched 29.0 kB in 0s (87.5 kB/s) 2024-09-16T05:05:10.391 INFO:teuthology.orchestra.run.smithi149.stdout:Fetched 158 kB in 1s (283 kB/s) 2024-09-16T05:05:11.090 INFO:teuthology.orchestra.run.smithi114.stdout:Reading package lists... 2024-09-16T05:05:11.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-326-g8921c8ef-1jammy cephadm=19.1.1-326-g8921c8ef-1jammy ceph-mds=19.1.1-326-g8921c8ef-1jammy ceph-mgr=19.1.1-326-g8921c8ef-1jammy ceph-common=19.1.1-326-g8921c8ef-1jammy ceph-fuse=19.1.1-326-g8921c8ef-1jammy ceph-test=19.1.1-326-g8921c8ef-1jammy ceph-volume=19.1.1-326-g8921c8ef-1jammy radosgw=19.1.1-326-g8921c8ef-1jammy python3-rados=19.1.1-326-g8921c8ef-1jammy python3-rgw=19.1.1-326-g8921c8ef-1jammy python3-cephfs=19.1.1-326-g8921c8ef-1jammy python3-rbd=19.1.1-326-g8921c8ef-1jammy libcephfs2=19.1.1-326-g8921c8ef-1jammy libcephfs-dev=19.1.1-326-g8921c8ef-1jammy librados2=19.1.1-326-g8921c8ef-1jammy librbd1=19.1.1-326-g8921c8ef-1jammy rbd-fuse=19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:05:11.169 INFO:teuthology.orchestra.run.smithi114.stdout:Reading package lists... 2024-09-16T05:05:11.378 INFO:teuthology.orchestra.run.smithi114.stdout:Building dependency tree... 2024-09-16T05:05:11.378 INFO:teuthology.orchestra.run.smithi114.stdout:Reading state information... 2024-09-16T05:05:11.446 INFO:teuthology.orchestra.run.smithi149.stdout:Reading package lists... 2024-09-16T05:05:11.463 DEBUG:teuthology.orchestra.run.smithi149:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-326-g8921c8ef-1jammy cephadm=19.1.1-326-g8921c8ef-1jammy ceph-mds=19.1.1-326-g8921c8ef-1jammy ceph-mgr=19.1.1-326-g8921c8ef-1jammy ceph-common=19.1.1-326-g8921c8ef-1jammy ceph-fuse=19.1.1-326-g8921c8ef-1jammy ceph-test=19.1.1-326-g8921c8ef-1jammy ceph-volume=19.1.1-326-g8921c8ef-1jammy radosgw=19.1.1-326-g8921c8ef-1jammy python3-rados=19.1.1-326-g8921c8ef-1jammy python3-rgw=19.1.1-326-g8921c8ef-1jammy python3-cephfs=19.1.1-326-g8921c8ef-1jammy python3-rbd=19.1.1-326-g8921c8ef-1jammy libcephfs2=19.1.1-326-g8921c8ef-1jammy libcephfs-dev=19.1.1-326-g8921c8ef-1jammy librados2=19.1.1-326-g8921c8ef-1jammy librbd1=19.1.1-326-g8921c8ef-1jammy rbd-fuse=19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:05:11.516 INFO:teuthology.orchestra.run.smithi149.stdout:Reading package lists... 2024-09-16T05:05:11.572 INFO:teuthology.orchestra.run.smithi114.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T05:05:11.572 INFO:teuthology.orchestra.run.smithi114.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T05:05:11.572 INFO:teuthology.orchestra.run.smithi114.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T05:05:11.572 INFO:teuthology.orchestra.run.smithi114.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T05:05:11.572 INFO:teuthology.orchestra.run.smithi114.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T05:05:11.573 INFO:teuthology.orchestra.run.smithi114.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T05:05:11.573 INFO:teuthology.orchestra.run.smithi114.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T05:05:11.573 INFO:teuthology.orchestra.run.smithi114.stdout:The following additional packages will be installed: 2024-09-16T05:05:11.573 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T05:05:11.574 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T05:05:11.574 INFO:teuthology.orchestra.run.smithi114.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-16T05:05:11.574 INFO:teuthology.orchestra.run.smithi114.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T05:05:11.574 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-16T05:05:11.574 INFO:teuthology.orchestra.run.smithi114.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-16T05:05:11.574 INFO:teuthology.orchestra.run.smithi114.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T05:05:11.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T05:05:11.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T05:05:11.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T05:05:11.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T05:05:11.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T05:05:11.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T05:05:11.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T05:05:11.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T05:05:11.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T05:05:11.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T05:05:11.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T05:05:11.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T05:05:11.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T05:05:11.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T05:05:11.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T05:05:11.576 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T05:05:11.576 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet zip 2024-09-16T05:05:11.576 INFO:teuthology.orchestra.run.smithi114.stdout:Suggested packages: 2024-09-16T05:05:11.576 INFO:teuthology.orchestra.run.smithi114.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T05:05:11.576 INFO:teuthology.orchestra.run.smithi114.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T05:05:11.576 INFO:teuthology.orchestra.run.smithi114.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T05:05:11.576 INFO:teuthology.orchestra.run.smithi114.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T05:05:11.576 INFO:teuthology.orchestra.run.smithi114.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T05:05:11.576 INFO:teuthology.orchestra.run.smithi114.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T05:05:11.576 INFO:teuthology.orchestra.run.smithi114.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T05:05:11.577 INFO:teuthology.orchestra.run.smithi114.stdout:Recommended packages: 2024-09-16T05:05:11.577 INFO:teuthology.orchestra.run.smithi114.stdout: btrfs-tools 2024-09-16T05:05:11.681 INFO:teuthology.orchestra.run.smithi114.stdout:The following NEW packages will be installed: 2024-09-16T05:05:11.681 INFO:teuthology.orchestra.run.smithi114.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T05:05:11.681 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T05:05:11.681 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T05:05:11.681 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T05:05:11.681 INFO:teuthology.orchestra.run.smithi114.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T05:05:11.681 INFO:teuthology.orchestra.run.smithi114.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T05:05:11.683 INFO:teuthology.orchestra.run.smithi114.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T05:05:11.683 INFO:teuthology.orchestra.run.smithi114.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T05:05:11.683 INFO:teuthology.orchestra.run.smithi114.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T05:05:11.683 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T05:05:11.683 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T05:05:11.683 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T05:05:11.683 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T05:05:11.683 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T05:05:11.683 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T05:05:11.683 INFO:teuthology.orchestra.run.smithi114.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T05:05:11.684 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T05:05:11.684 INFO:teuthology.orchestra.run.smithi114.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T05:05:11.684 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T05:05:11.684 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T05:05:11.684 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T05:05:11.684 INFO:teuthology.orchestra.run.smithi114.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T05:05:11.684 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T05:05:11.684 INFO:teuthology.orchestra.run.smithi114.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T05:05:11.684 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T05:05:11.684 INFO:teuthology.orchestra.run.smithi114.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T05:05:11.704 INFO:teuthology.orchestra.run.smithi149.stdout:Building dependency tree... 2024-09-16T05:05:11.705 INFO:teuthology.orchestra.run.smithi149.stdout:Reading state information... 2024-09-16T05:05:11.828 INFO:teuthology.orchestra.run.smithi114.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T05:05:11.828 INFO:teuthology.orchestra.run.smithi114.stdout:Need to get 181 MB of archives. 2024-09-16T05:05:11.828 INFO:teuthology.orchestra.run.smithi114.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-16T05:05:11.828 INFO:teuthology.orchestra.run.smithi114.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-326-g8921c8ef-1jammy [3,590 kB] 2024-09-16T05:05:11.872 INFO:teuthology.orchestra.run.smithi114.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T05:05:11.894 INFO:teuthology.orchestra.run.smithi149.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T05:05:11.894 INFO:teuthology.orchestra.run.smithi149.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T05:05:11.894 INFO:teuthology.orchestra.run.smithi149.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T05:05:11.894 INFO:teuthology.orchestra.run.smithi149.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T05:05:11.895 INFO:teuthology.orchestra.run.smithi149.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T05:05:11.895 INFO:teuthology.orchestra.run.smithi149.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T05:05:11.895 INFO:teuthology.orchestra.run.smithi149.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T05:05:11.896 INFO:teuthology.orchestra.run.smithi149.stdout:The following additional packages will be installed: 2024-09-16T05:05:11.896 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T05:05:11.896 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T05:05:11.896 INFO:teuthology.orchestra.run.smithi149.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-16T05:05:11.896 INFO:teuthology.orchestra.run.smithi149.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T05:05:11.897 INFO:teuthology.orchestra.run.smithi149.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-16T05:05:11.897 INFO:teuthology.orchestra.run.smithi149.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-16T05:05:11.897 INFO:teuthology.orchestra.run.smithi149.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T05:05:11.897 INFO:teuthology.orchestra.run.smithi149.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T05:05:11.897 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T05:05:11.897 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T05:05:11.897 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T05:05:11.898 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T05:05:11.898 INFO:teuthology.orchestra.run.smithi149.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T05:05:11.898 INFO:teuthology.orchestra.run.smithi149.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T05:05:11.898 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T05:05:11.898 INFO:teuthology.orchestra.run.smithi149.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T05:05:11.898 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T05:05:11.898 INFO:teuthology.orchestra.run.smithi149.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T05:05:11.898 INFO:teuthology.orchestra.run.smithi149.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T05:05:11.898 INFO:teuthology.orchestra.run.smithi149.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T05:05:11.898 INFO:teuthology.orchestra.run.smithi149.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T05:05:11.898 INFO:teuthology.orchestra.run.smithi149.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T05:05:11.898 INFO:teuthology.orchestra.run.smithi149.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T05:05:11.898 INFO:teuthology.orchestra.run.smithi149.stdout: xmlstarlet zip 2024-09-16T05:05:11.899 INFO:teuthology.orchestra.run.smithi149.stdout:Suggested packages: 2024-09-16T05:05:11.899 INFO:teuthology.orchestra.run.smithi149.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T05:05:11.899 INFO:teuthology.orchestra.run.smithi149.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T05:05:11.899 INFO:teuthology.orchestra.run.smithi149.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T05:05:11.899 INFO:teuthology.orchestra.run.smithi149.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T05:05:11.899 INFO:teuthology.orchestra.run.smithi149.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T05:05:11.899 INFO:teuthology.orchestra.run.smithi149.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T05:05:11.899 INFO:teuthology.orchestra.run.smithi149.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T05:05:11.899 INFO:teuthology.orchestra.run.smithi149.stdout:Recommended packages: 2024-09-16T05:05:11.899 INFO:teuthology.orchestra.run.smithi149.stdout: btrfs-tools 2024-09-16T05:05:12.003 INFO:teuthology.orchestra.run.smithi149.stdout:The following NEW packages will be installed: 2024-09-16T05:05:12.004 INFO:teuthology.orchestra.run.smithi149.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T05:05:12.004 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T05:05:12.004 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T05:05:12.005 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T05:05:12.005 INFO:teuthology.orchestra.run.smithi149.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T05:05:12.005 INFO:teuthology.orchestra.run.smithi149.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T05:05:12.005 INFO:teuthology.orchestra.run.smithi149.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T05:05:12.005 INFO:teuthology.orchestra.run.smithi149.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T05:05:12.005 INFO:teuthology.orchestra.run.smithi149.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T05:05:12.005 INFO:teuthology.orchestra.run.smithi149.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T05:05:12.006 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T05:05:12.006 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T05:05:12.006 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T05:05:12.006 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T05:05:12.006 INFO:teuthology.orchestra.run.smithi149.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T05:05:12.006 INFO:teuthology.orchestra.run.smithi149.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T05:05:12.006 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T05:05:12.006 INFO:teuthology.orchestra.run.smithi149.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T05:05:12.006 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T05:05:12.006 INFO:teuthology.orchestra.run.smithi149.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T05:05:12.006 INFO:teuthology.orchestra.run.smithi149.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T05:05:12.006 INFO:teuthology.orchestra.run.smithi149.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T05:05:12.006 INFO:teuthology.orchestra.run.smithi149.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T05:05:12.007 INFO:teuthology.orchestra.run.smithi149.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T05:05:12.007 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T05:05:12.007 INFO:teuthology.orchestra.run.smithi149.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T05:05:12.011 INFO:teuthology.orchestra.run.smithi114.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-326-g8921c8ef-1jammy [3,241 kB] 2024-09-16T05:05:12.056 INFO:teuthology.orchestra.run.smithi114.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-326-g8921c8ef-1jammy [719 kB] 2024-09-16T05:05:12.067 INFO:teuthology.orchestra.run.smithi114.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-326-g8921c8ef-1jammy [356 kB] 2024-09-16T05:05:12.073 INFO:teuthology.orchestra.run.smithi114.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-326-g8921c8ef-1jammy [32.8 kB] 2024-09-16T05:05:12.073 INFO:teuthology.orchestra.run.smithi114.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-326-g8921c8ef-1jammy [180 kB] 2024-09-16T05:05:12.077 INFO:teuthology.orchestra.run.smithi114.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-326-g8921c8ef-1jammy [67.8 kB] 2024-09-16T05:05:12.077 INFO:teuthology.orchestra.run.smithi114.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-326-g8921c8ef-1jammy [333 kB] 2024-09-16T05:05:12.083 INFO:teuthology.orchestra.run.smithi114.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-326-g8921c8ef-1jammy [6,875 kB] 2024-09-16T05:05:12.196 INFO:teuthology.orchestra.run.smithi149.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T05:05:12.196 INFO:teuthology.orchestra.run.smithi149.stdout:Need to get 181 MB of archives. 2024-09-16T05:05:12.196 INFO:teuthology.orchestra.run.smithi149.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-16T05:05:12.196 INFO:teuthology.orchestra.run.smithi149.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T05:05:12.197 INFO:teuthology.orchestra.run.smithi114.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-326-g8921c8ef-1jammy [112 kB] 2024-09-16T05:05:12.199 INFO:teuthology.orchestra.run.smithi114.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-326-g8921c8ef-1jammy [472 kB] 2024-09-16T05:05:12.205 INFO:teuthology.orchestra.run.smithi114.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-326-g8921c8ef-1jammy [25.9 MB] 2024-09-16T05:05:12.214 INFO:teuthology.orchestra.run.smithi149.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-326-g8921c8ef-1jammy [3,590 kB] 2024-09-16T05:05:12.293 INFO:teuthology.orchestra.run.smithi114.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T05:05:12.305 INFO:teuthology.orchestra.run.smithi114.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T05:05:12.386 INFO:teuthology.orchestra.run.smithi114.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T05:05:12.615 INFO:teuthology.orchestra.run.smithi114.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T05:05:12.615 INFO:teuthology.orchestra.run.smithi149.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T05:05:12.625 INFO:teuthology.orchestra.run.smithi114.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T05:05:12.628 INFO:teuthology.orchestra.run.smithi149.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T05:05:12.707 INFO:teuthology.orchestra.run.smithi114.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T05:05:12.709 INFO:teuthology.orchestra.run.smithi149.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T05:05:12.766 INFO:teuthology.orchestra.run.smithi114.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T05:05:12.767 INFO:teuthology.orchestra.run.smithi114.stdout:Get:21 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-326-g8921c8ef-1jammy [5,145 kB] 2024-09-16T05:05:12.767 INFO:teuthology.orchestra.run.smithi114.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T05:05:12.768 INFO:teuthology.orchestra.run.smithi114.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T05:05:12.783 INFO:teuthology.orchestra.run.smithi114.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T05:05:12.788 INFO:teuthology.orchestra.run.smithi114.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T05:05:12.792 INFO:teuthology.orchestra.run.smithi114.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T05:05:12.794 INFO:teuthology.orchestra.run.smithi114.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T05:05:12.794 INFO:teuthology.orchestra.run.smithi114.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T05:05:12.833 INFO:teuthology.orchestra.run.smithi114.stdout:Get:29 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-326-g8921c8ef-1jammy [245 kB] 2024-09-16T05:05:12.835 INFO:teuthology.orchestra.run.smithi114.stdout:Get:30 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-326-g8921c8ef-1jammy [124 kB] 2024-09-16T05:05:12.836 INFO:teuthology.orchestra.run.smithi114.stdout:Get:31 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-326-g8921c8ef-1jammy [1,555 kB] 2024-09-16T05:05:12.850 INFO:teuthology.orchestra.run.smithi114.stdout:Get:32 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-326-g8921c8ef-1jammy [6,143 kB] 2024-09-16T05:05:12.860 INFO:teuthology.orchestra.run.smithi114.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T05:05:12.865 INFO:teuthology.orchestra.run.smithi114.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T05:05:12.868 INFO:teuthology.orchestra.run.smithi114.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T05:05:12.870 INFO:teuthology.orchestra.run.smithi114.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T05:05:12.938 INFO:teuthology.orchestra.run.smithi149.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T05:05:12.942 INFO:teuthology.orchestra.run.smithi114.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T05:05:12.944 INFO:teuthology.orchestra.run.smithi114.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T05:05:12.944 INFO:teuthology.orchestra.run.smithi114.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T05:05:12.945 INFO:teuthology.orchestra.run.smithi114.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T05:05:12.946 INFO:teuthology.orchestra.run.smithi114.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T05:05:12.947 INFO:teuthology.orchestra.run.smithi114.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T05:05:12.948 INFO:teuthology.orchestra.run.smithi114.stdout:Get:43 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-326-g8921c8ef-1jammy [22.8 MB] 2024-09-16T05:05:12.949 INFO:teuthology.orchestra.run.smithi149.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T05:05:12.979 INFO:teuthology.orchestra.run.smithi149.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T05:05:12.987 INFO:teuthology.orchestra.run.smithi149.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T05:05:12.988 INFO:teuthology.orchestra.run.smithi149.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T05:05:12.989 INFO:teuthology.orchestra.run.smithi149.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T05:05:13.008 INFO:teuthology.orchestra.run.smithi149.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T05:05:13.012 INFO:teuthology.orchestra.run.smithi149.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T05:05:13.016 INFO:teuthology.orchestra.run.smithi149.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T05:05:13.020 INFO:teuthology.orchestra.run.smithi149.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T05:05:13.023 INFO:teuthology.orchestra.run.smithi114.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T05:05:13.024 INFO:teuthology.orchestra.run.smithi114.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T05:05:13.030 INFO:teuthology.orchestra.run.smithi114.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T05:05:13.030 INFO:teuthology.orchestra.run.smithi114.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T05:05:13.101 INFO:teuthology.orchestra.run.smithi149.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T05:05:13.105 INFO:teuthology.orchestra.run.smithi114.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T05:05:13.105 INFO:teuthology.orchestra.run.smithi149.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T05:05:13.105 INFO:teuthology.orchestra.run.smithi114.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T05:05:13.107 INFO:teuthology.orchestra.run.smithi114.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T05:05:13.108 INFO:teuthology.orchestra.run.smithi114.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T05:05:13.108 INFO:teuthology.orchestra.run.smithi149.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T05:05:13.109 INFO:teuthology.orchestra.run.smithi114.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T05:05:13.111 INFO:teuthology.orchestra.run.smithi114.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T05:05:13.112 INFO:teuthology.orchestra.run.smithi149.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T05:05:13.113 INFO:teuthology.orchestra.run.smithi149.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T05:05:13.113 INFO:teuthology.orchestra.run.smithi149.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T05:05:13.116 INFO:teuthology.orchestra.run.smithi149.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T05:05:13.116 INFO:teuthology.orchestra.run.smithi149.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T05:05:13.117 INFO:teuthology.orchestra.run.smithi149.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T05:05:13.184 INFO:teuthology.orchestra.run.smithi149.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T05:05:13.186 INFO:teuthology.orchestra.run.smithi114.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T05:05:13.187 INFO:teuthology.orchestra.run.smithi114.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T05:05:13.198 INFO:teuthology.orchestra.run.smithi114.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T05:05:13.198 INFO:teuthology.orchestra.run.smithi114.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T05:05:13.199 INFO:teuthology.orchestra.run.smithi149.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T05:05:13.200 INFO:teuthology.orchestra.run.smithi149.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T05:05:13.200 INFO:teuthology.orchestra.run.smithi149.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T05:05:13.206 INFO:teuthology.orchestra.run.smithi149.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T05:05:13.207 INFO:teuthology.orchestra.run.smithi149.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T05:05:13.269 INFO:teuthology.orchestra.run.smithi149.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T05:05:13.270 INFO:teuthology.orchestra.run.smithi114.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T05:05:13.270 INFO:teuthology.orchestra.run.smithi114.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T05:05:13.271 INFO:teuthology.orchestra.run.smithi149.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T05:05:13.271 INFO:teuthology.orchestra.run.smithi149.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T05:05:13.271 INFO:teuthology.orchestra.run.smithi149.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T05:05:13.272 INFO:teuthology.orchestra.run.smithi114.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T05:05:13.276 INFO:teuthology.orchestra.run.smithi114.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T05:05:13.277 INFO:teuthology.orchestra.run.smithi114.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T05:05:13.280 INFO:teuthology.orchestra.run.smithi114.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T05:05:13.345 INFO:teuthology.orchestra.run.smithi149.stdout:Get:35 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-326-g8921c8ef-1jammy [3,241 kB] 2024-09-16T05:05:13.346 INFO:teuthology.orchestra.run.smithi149.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T05:05:13.349 INFO:teuthology.orchestra.run.smithi149.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T05:05:13.349 INFO:teuthology.orchestra.run.smithi114.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T05:05:13.350 INFO:teuthology.orchestra.run.smithi149.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T05:05:13.351 INFO:teuthology.orchestra.run.smithi149.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T05:05:13.362 INFO:teuthology.orchestra.run.smithi149.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T05:05:13.426 INFO:teuthology.orchestra.run.smithi114.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T05:05:13.427 INFO:teuthology.orchestra.run.smithi114.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T05:05:13.427 INFO:teuthology.orchestra.run.smithi114.stdout:Get:67 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-326-g8921c8ef-1jammy [14.1 kB] 2024-09-16T05:05:13.428 INFO:teuthology.orchestra.run.smithi114.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-326-g8921c8ef-1jammy [877 kB] 2024-09-16T05:05:13.428 INFO:teuthology.orchestra.run.smithi149.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T05:05:13.429 INFO:teuthology.orchestra.run.smithi114.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T05:05:13.429 INFO:teuthology.orchestra.run.smithi149.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T05:05:13.430 INFO:teuthology.orchestra.run.smithi149.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T05:05:13.433 INFO:teuthology.orchestra.run.smithi149.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T05:05:13.433 INFO:teuthology.orchestra.run.smithi114.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T05:05:13.436 INFO:teuthology.orchestra.run.smithi149.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T05:05:13.438 INFO:teuthology.orchestra.run.smithi114.stdout:Get:71 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-326-g8921c8ef-1jammy [2,462 kB] 2024-09-16T05:05:13.465 INFO:teuthology.orchestra.run.smithi114.stdout:Get:72 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-326-g8921c8ef-1jammy [794 kB] 2024-09-16T05:05:13.474 INFO:teuthology.orchestra.run.smithi114.stdout:Get:73 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-326-g8921c8ef-1jammy [152 kB] 2024-09-16T05:05:13.476 INFO:teuthology.orchestra.run.smithi114.stdout:Get:74 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-326-g8921c8ef-1jammy [2,275 kB] 2024-09-16T05:05:13.497 INFO:teuthology.orchestra.run.smithi114.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-326-g8921c8ef-1jammy [8,625 kB] 2024-09-16T05:05:13.510 INFO:teuthology.orchestra.run.smithi149.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T05:05:13.512 INFO:teuthology.orchestra.run.smithi149.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T05:05:13.514 INFO:teuthology.orchestra.run.smithi149.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T05:05:13.518 INFO:teuthology.orchestra.run.smithi149.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T05:05:13.519 INFO:teuthology.orchestra.run.smithi149.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T05:05:13.563 INFO:teuthology.orchestra.run.smithi114.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T05:05:13.565 INFO:teuthology.orchestra.run.smithi114.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T05:05:13.567 INFO:teuthology.orchestra.run.smithi114.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T05:05:13.568 INFO:teuthology.orchestra.run.smithi114.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T05:05:13.570 INFO:teuthology.orchestra.run.smithi114.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T05:05:13.592 INFO:teuthology.orchestra.run.smithi149.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T05:05:13.596 INFO:teuthology.orchestra.run.smithi149.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T05:05:13.634 INFO:teuthology.orchestra.run.smithi114.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T05:05:13.640 INFO:teuthology.orchestra.run.smithi114.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T05:05:13.640 INFO:teuthology.orchestra.run.smithi114.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T05:05:13.655 INFO:teuthology.orchestra.run.smithi114.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-326-g8921c8ef-1jammy [14.2 kB] 2024-09-16T05:05:13.655 INFO:teuthology.orchestra.run.smithi114.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-326-g8921c8ef-1jammy [51.4 MB] 2024-09-16T05:05:13.701 INFO:teuthology.orchestra.run.smithi114.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T05:05:13.702 INFO:teuthology.orchestra.run.smithi114.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T05:05:13.703 INFO:teuthology.orchestra.run.smithi114.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T05:05:13.704 INFO:teuthology.orchestra.run.smithi114.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T05:05:13.705 INFO:teuthology.orchestra.run.smithi114.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T05:05:13.706 INFO:teuthology.orchestra.run.smithi114.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T05:05:13.746 INFO:teuthology.orchestra.run.smithi149.stdout:Get:53 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-326-g8921c8ef-1jammy [719 kB] 2024-09-16T05:05:13.758 INFO:teuthology.orchestra.run.smithi149.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T05:05:13.759 INFO:teuthology.orchestra.run.smithi114.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T05:05:13.760 INFO:teuthology.orchestra.run.smithi149.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T05:05:13.760 INFO:teuthology.orchestra.run.smithi114.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T05:05:13.762 INFO:teuthology.orchestra.run.smithi149.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T05:05:13.762 INFO:teuthology.orchestra.run.smithi149.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T05:05:13.764 INFO:teuthology.orchestra.run.smithi149.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T05:05:13.767 INFO:teuthology.orchestra.run.smithi114.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T05:05:13.797 INFO:teuthology.orchestra.run.smithi149.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T05:05:13.800 INFO:teuthology.orchestra.run.smithi149.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T05:05:13.800 INFO:teuthology.orchestra.run.smithi149.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T05:05:13.808 INFO:teuthology.orchestra.run.smithi149.stdout:Get:62 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-326-g8921c8ef-1jammy [356 kB] 2024-09-16T05:05:13.841 INFO:teuthology.orchestra.run.smithi114.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T05:05:13.845 INFO:teuthology.orchestra.run.smithi114.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T05:05:13.847 INFO:teuthology.orchestra.run.smithi114.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T05:05:13.857 INFO:teuthology.orchestra.run.smithi114.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T05:05:13.860 INFO:teuthology.orchestra.run.smithi149.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T05:05:13.860 INFO:teuthology.orchestra.run.smithi149.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T05:05:13.862 INFO:teuthology.orchestra.run.smithi114.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T05:05:13.864 INFO:teuthology.orchestra.run.smithi149.stdout:Get:65 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-326-g8921c8ef-1jammy [32.8 kB] 2024-09-16T05:05:13.865 INFO:teuthology.orchestra.run.smithi114.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T05:05:13.865 INFO:teuthology.orchestra.run.smithi149.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-326-g8921c8ef-1jammy [180 kB] 2024-09-16T05:05:13.866 INFO:teuthology.orchestra.run.smithi114.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T05:05:13.917 INFO:teuthology.orchestra.run.smithi149.stdout:Get:67 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-326-g8921c8ef-1jammy [67.8 kB] 2024-09-16T05:05:13.919 INFO:teuthology.orchestra.run.smithi149.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-326-g8921c8ef-1jammy [333 kB] 2024-09-16T05:05:13.923 INFO:teuthology.orchestra.run.smithi114.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T05:05:13.925 INFO:teuthology.orchestra.run.smithi114.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T05:05:13.925 INFO:teuthology.orchestra.run.smithi149.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T05:05:13.926 INFO:teuthology.orchestra.run.smithi149.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T05:05:13.927 INFO:teuthology.orchestra.run.smithi149.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T05:05:13.928 INFO:teuthology.orchestra.run.smithi149.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T05:05:13.929 INFO:teuthology.orchestra.run.smithi149.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T05:05:13.929 INFO:teuthology.orchestra.run.smithi149.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T05:05:13.932 INFO:teuthology.orchestra.run.smithi114.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T05:05:13.935 INFO:teuthology.orchestra.run.smithi149.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T05:05:13.973 INFO:teuthology.orchestra.run.smithi149.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-326-g8921c8ef-1jammy [6,875 kB] 2024-09-16T05:05:14.005 INFO:teuthology.orchestra.run.smithi114.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T05:05:14.006 INFO:teuthology.orchestra.run.smithi114.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T05:05:14.007 INFO:teuthology.orchestra.run.smithi149.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T05:05:14.007 INFO:teuthology.orchestra.run.smithi114.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T05:05:14.010 INFO:teuthology.orchestra.run.smithi149.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T05:05:14.011 INFO:teuthology.orchestra.run.smithi149.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T05:05:14.012 INFO:teuthology.orchestra.run.smithi114.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T05:05:14.014 INFO:teuthology.orchestra.run.smithi114.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T05:05:14.031 INFO:teuthology.orchestra.run.smithi114.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T05:05:14.032 INFO:teuthology.orchestra.run.smithi114.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T05:05:14.086 INFO:teuthology.orchestra.run.smithi114.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T05:05:14.089 INFO:teuthology.orchestra.run.smithi149.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T05:05:14.092 INFO:teuthology.orchestra.run.smithi114.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T05:05:14.097 INFO:teuthology.orchestra.run.smithi114.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T05:05:14.097 INFO:teuthology.orchestra.run.smithi149.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T05:05:14.097 INFO:teuthology.orchestra.run.smithi149.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T05:05:14.098 INFO:teuthology.orchestra.run.smithi149.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T05:05:14.109 INFO:teuthology.orchestra.run.smithi149.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T05:05:14.111 INFO:teuthology.orchestra.run.smithi149.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T05:05:14.118 INFO:teuthology.orchestra.run.smithi149.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T05:05:14.170 INFO:teuthology.orchestra.run.smithi149.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T05:05:14.172 INFO:teuthology.orchestra.run.smithi149.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T05:05:14.173 INFO:teuthology.orchestra.run.smithi149.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T05:05:14.252 INFO:teuthology.orchestra.run.smithi149.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T05:05:14.253 INFO:teuthology.orchestra.run.smithi149.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T05:05:14.270 INFO:teuthology.orchestra.run.smithi149.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T05:05:14.271 INFO:teuthology.orchestra.run.smithi149.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T05:05:14.272 INFO:teuthology.orchestra.run.smithi149.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T05:05:14.277 INFO:teuthology.orchestra.run.smithi149.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T05:05:14.278 INFO:teuthology.orchestra.run.smithi149.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T05:05:14.586 INFO:teuthology.orchestra.run.smithi149.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-326-g8921c8ef-1jammy [112 kB] 2024-09-16T05:05:14.589 INFO:teuthology.orchestra.run.smithi149.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-326-g8921c8ef-1jammy [472 kB] 2024-09-16T05:05:14.643 INFO:teuthology.orchestra.run.smithi149.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-326-g8921c8ef-1jammy [25.9 MB] 2024-09-16T05:05:14.925 INFO:teuthology.orchestra.run.smithi114.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-326-g8921c8ef-1jammy [130 kB] 2024-09-16T05:05:14.927 INFO:teuthology.orchestra.run.smithi114.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-326-g8921c8ef-1jammy [39.0 kB] 2024-09-16T05:05:14.928 INFO:teuthology.orchestra.run.smithi114.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-326-g8921c8ef-1jammy [13.6 MB] 2024-09-16T05:05:15.226 INFO:teuthology.orchestra.run.smithi114.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-326-g8921c8ef-1jammy [91.5 kB] 2024-09-16T05:05:15.398 INFO:teuthology.orchestra.run.smithi149.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-326-g8921c8ef-1jammy [5,145 kB] 2024-09-16T05:05:15.463 INFO:teuthology.orchestra.run.smithi149.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-326-g8921c8ef-1jammy [245 kB] 2024-09-16T05:05:15.466 INFO:teuthology.orchestra.run.smithi149.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-326-g8921c8ef-1jammy [124 kB] 2024-09-16T05:05:15.467 INFO:teuthology.orchestra.run.smithi149.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-326-g8921c8ef-1jammy [1,555 kB] 2024-09-16T05:05:15.480 INFO:teuthology.orchestra.run.smithi149.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-326-g8921c8ef-1jammy [6,143 kB] 2024-09-16T05:05:15.571 INFO:teuthology.orchestra.run.smithi149.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-326-g8921c8ef-1jammy [22.8 MB] 2024-09-16T05:05:15.786 INFO:teuthology.orchestra.run.smithi114.stdout:Fetched 181 MB in 4s (51.2 MB/s) 2024-09-16T05:05:15.946 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T05:05:16.017 INFO:teuthology.orchestra.run.smithi149.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-326-g8921c8ef-1jammy [14.1 kB] 2024-09-16T05:05:16.017 INFO:teuthology.orchestra.run.smithi149.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-326-g8921c8ef-1jammy [877 kB] 2024-09-16T05:05:16.027 INFO:teuthology.orchestra.run.smithi149.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-326-g8921c8ef-1jammy [2,462 kB] 2024-09-16T05:05:16.036 INFO:teuthology.orchestra.run.smithi114.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T05:05:16.041 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T05:05:16.054 INFO:teuthology.orchestra.run.smithi149.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-326-g8921c8ef-1jammy [794 kB] 2024-09-16T05:05:16.062 INFO:teuthology.orchestra.run.smithi149.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-326-g8921c8ef-1jammy [152 kB] 2024-09-16T05:05:16.064 INFO:teuthology.orchestra.run.smithi149.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-326-g8921c8ef-1jammy [2,275 kB] 2024-09-16T05:05:16.084 INFO:teuthology.orchestra.run.smithi149.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-326-g8921c8ef-1jammy [8,625 kB] 2024-09-16T05:05:16.128 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T05:05:16.225 INFO:teuthology.orchestra.run.smithi149.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-326-g8921c8ef-1jammy [14.2 kB] 2024-09-16T05:05:16.226 INFO:teuthology.orchestra.run.smithi149.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-326-g8921c8ef-1jammy [51.4 MB] 2024-09-16T05:05:16.465 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T05:05:16.491 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T05:05:16.531 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T05:05:16.842 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T05:05:16.869 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T05:05:16.909 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T05:05:17.327 INFO:teuthology.orchestra.run.smithi149.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-326-g8921c8ef-1jammy [130 kB] 2024-09-16T05:05:17.335 INFO:teuthology.orchestra.run.smithi149.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-326-g8921c8ef-1jammy [39.0 kB] 2024-09-16T05:05:17.335 INFO:teuthology.orchestra.run.smithi149.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-326-g8921c8ef-1jammy [13.6 MB] 2024-09-16T05:05:17.346 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T05:05:17.373 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T05:05:17.421 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:05:17.523 INFO:teuthology.orchestra.run.smithi149.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-326-g8921c8ef-1jammy [91.5 kB] 2024-09-16T05:05:17.858 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T05:05:17.884 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T05:05:17.924 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:05:18.096 INFO:teuthology.orchestra.run.smithi149.stdout:Fetched 181 MB in 6s (32.9 MB/s) 2024-09-16T05:05:18.261 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T05:05:18.269 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T05:05:18.295 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T05:05:18.335 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:05:18.361 INFO:teuthology.orchestra.run.smithi149.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T05:05:18.368 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T05:05:18.476 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T05:05:18.655 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T05:05:18.681 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T05:05:18.713 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T05:05:18.804 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T05:05:18.831 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T05:05:18.871 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T05:05:19.016 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package librados2. 2024-09-16T05:05:19.042 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../007-librados2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:19.082 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:19.182 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T05:05:19.209 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T05:05:19.249 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T05:05:19.560 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package librbd1. 2024-09-16T05:05:19.586 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../008-librbd1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:19.627 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:19.686 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T05:05:19.712 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T05:05:19.760 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:05:20.097 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T05:05:20.123 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:20.155 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T05:05:20.163 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:20.181 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T05:05:20.222 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:05:20.407 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-rados. 2024-09-16T05:05:20.422 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../010-python3-rados_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:20.457 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:20.542 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T05:05:20.568 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T05:05:20.608 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:05:20.760 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T05:05:20.786 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:05:20.826 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:20.911 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T05:05:20.938 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T05:05:20.978 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T05:05:21.079 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T05:05:21.094 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:21.137 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:21.273 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package librados2. 2024-09-16T05:05:21.299 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../007-librados2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:21.339 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:21.456 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T05:05:21.482 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:05:21.523 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:21.800 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package librbd1. 2024-09-16T05:05:21.817 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T05:05:21.826 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../008-librbd1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:21.833 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T05:05:21.866 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:21.867 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T05:05:22.181 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T05:05:22.208 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T05:05:22.254 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T05:05:22.311 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T05:05:22.337 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:22.377 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:22.523 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T05:05:22.550 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:22.589 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:22.664 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-rados. 2024-09-16T05:05:22.692 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../010-python3-rados_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:22.730 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:22.926 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T05:05:22.952 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T05:05:22.992 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T05:05:23.016 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T05:05:23.042 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:05:23.074 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:23.253 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T05:05:23.268 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T05:05:23.294 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T05:05:23.304 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T05:05:23.308 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:23.351 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:23.614 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T05:05:23.641 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T05:05:23.663 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T05:05:23.681 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T05:05:23.689 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:05:23.729 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:24.017 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package lua5.1. 2024-09-16T05:05:24.024 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T05:05:24.039 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T05:05:24.044 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T05:05:24.074 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T05:05:24.084 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T05:05:24.343 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T05:05:24.370 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T05:05:24.370 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package lua-any. 2024-09-16T05:05:24.386 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-16T05:05:24.410 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T05:05:24.428 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T05:05:24.706 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T05:05:24.706 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package zip. 2024-09-16T05:05:24.722 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-16T05:05:24.733 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:24.764 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T05:05:24.770 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:25.051 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package unzip. 2024-09-16T05:05:25.077 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T05:05:25.109 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T05:05:25.324 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T05:05:25.350 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T05:05:25.390 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T05:05:25.659 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T05:05:25.675 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T05:05:25.709 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T05:05:25.730 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package luarocks. 2024-09-16T05:05:25.756 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T05:05:25.796 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T05:05:25.954 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T05:05:25.969 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T05:05:26.003 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T05:05:26.170 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package librgw2. 2024-09-16T05:05:26.196 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../025-librgw2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:26.237 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:26.324 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package lua5.1. 2024-09-16T05:05:26.350 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T05:05:26.390 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T05:05:26.727 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package lua-any. 2024-09-16T05:05:26.742 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-16T05:05:26.785 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T05:05:26.814 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T05:05:26.841 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:26.881 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:27.012 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package zip. 2024-09-16T05:05:27.027 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-16T05:05:27.062 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T05:05:27.175 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T05:05:27.202 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T05:05:27.233 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T05:05:27.349 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package unzip. 2024-09-16T05:05:27.375 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T05:05:27.415 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T05:05:27.545 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T05:05:27.571 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:27.612 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:27.836 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package luarocks. 2024-09-16T05:05:27.862 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T05:05:27.902 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T05:05:28.048 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-common. 2024-09-16T05:05:28.074 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../029-ceph-common_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:28.114 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:28.310 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package librgw2. 2024-09-16T05:05:28.337 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../025-librgw2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:28.376 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:28.896 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T05:05:28.911 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:28.945 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:29.290 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T05:05:29.317 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T05:05:29.357 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T05:05:29.436 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-base. 2024-09-16T05:05:29.462 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../030-ceph-base_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:29.504 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:29.668 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T05:05:29.694 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:29.734 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:30.082 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T05:05:30.097 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T05:05:30.104 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package ceph-common. 2024-09-16T05:05:30.131 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../029-ceph-common_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:30.140 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T05:05:30.162 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:30.577 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T05:05:30.604 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T05:05:30.693 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T05:05:31.022 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T05:05:31.048 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T05:05:31.088 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T05:05:31.366 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T05:05:31.393 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T05:05:31.441 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T05:05:31.626 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package ceph-base. 2024-09-16T05:05:31.652 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../030-ceph-base_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:31.702 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:31.710 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T05:05:31.736 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T05:05:31.777 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T05:05:32.088 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T05:05:32.114 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T05:05:32.154 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T05:05:32.280 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T05:05:32.309 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T05:05:32.347 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T05:05:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-portend. 2024-09-16T05:05:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-16T05:05:32.508 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T05:05:32.600 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T05:05:32.626 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T05:05:32.658 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T05:05:32.810 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T05:05:32.836 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T05:05:32.876 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T05:05:32.944 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T05:05:32.970 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T05:05:33.019 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T05:05:33.148 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T05:05:33.174 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T05:05:33.214 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T05:05:33.322 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T05:05:33.348 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T05:05:33.388 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T05:05:33.560 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T05:05:33.586 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T05:05:33.618 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T05:05:33.699 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T05:05:33.726 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T05:05:33.791 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T05:05:33.949 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T05:05:33.963 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T05:05:34.004 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T05:05:34.035 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T05:05:34.050 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T05:05:34.085 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T05:05:34.274 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T05:05:34.300 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T05:05:34.332 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T05:05:34.363 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-portend. 2024-09-16T05:05:34.389 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-16T05:05:34.429 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T05:05:34.635 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-mako. 2024-09-16T05:05:34.661 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T05:05:34.701 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T05:05:34.741 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T05:05:34.767 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T05:05:34.807 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T05:05:35.029 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T05:05:35.055 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T05:05:35.096 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T05:05:35.127 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T05:05:35.142 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T05:05:35.185 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T05:05:35.407 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T05:05:35.433 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T05:05:35.474 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T05:05:35.580 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T05:05:35.607 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T05:05:35.772 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-webob. 2024-09-16T05:05:35.788 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T05:05:35.799 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T05:05:35.968 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T05:05:36.084 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T05:05:36.144 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T05:05:36.150 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T05:05:36.296 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T05:05:36.322 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T05:05:36.357 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T05:05:36.411 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T05:05:36.438 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T05:05:36.469 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T05:05:36.657 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T05:05:36.683 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T05:05:36.723 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T05:05:36.755 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-mako. 2024-09-16T05:05:36.781 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T05:05:36.841 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T05:05:36.993 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-paste. 2024-09-16T05:05:37.008 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T05:05:37.043 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T05:05:37.175 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T05:05:37.201 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T05:05:37.241 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T05:05:37.371 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T05:05:37.386 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T05:05:37.429 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T05:05:37.544 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T05:05:37.559 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T05:05:37.602 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T05:05:37.699 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T05:05:37.725 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T05:05:37.757 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T05:05:37.863 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-webob. 2024-09-16T05:05:37.878 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T05:05:37.913 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T05:05:38.035 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T05:05:38.061 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T05:05:38.102 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T05:05:38.208 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T05:05:38.234 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T05:05:38.278 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T05:05:38.438 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T05:05:38.464 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T05:05:38.504 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T05:05:38.586 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T05:05:38.613 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T05:05:38.652 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T05:05:38.868 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T05:05:38.894 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T05:05:38.934 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T05:05:38.955 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-paste. 2024-09-16T05:05:38.982 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T05:05:39.022 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T05:05:39.270 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-idna. 2024-09-16T05:05:39.296 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T05:05:39.319 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T05:05:39.337 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T05:05:39.345 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T05:05:39.385 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T05:05:39.614 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T05:05:39.629 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T05:05:39.664 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T05:05:39.679 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T05:05:39.706 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T05:05:39.746 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T05:05:39.926 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-requests. 2024-09-16T05:05:39.952 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T05:05:39.984 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T05:05:40.041 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T05:05:40.056 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T05:05:40.090 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T05:05:40.270 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T05:05:40.286 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T05:05:40.328 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T05:05:40.360 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T05:05:40.386 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T05:05:40.427 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T05:05:40.632 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T05:05:40.658 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:05:40.690 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:40.765 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T05:05:40.791 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T05:05:40.831 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T05:05:41.078 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T05:05:41.105 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:41.144 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:41.159 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-idna. 2024-09-16T05:05:41.185 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T05:05:41.227 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T05:05:41.464 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T05:05:41.491 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T05:05:41.546 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T05:05:41.572 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T05:05:41.589 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T05:05:41.612 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T05:05:41.883 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T05:05:41.910 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T05:05:41.940 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-requests. 2024-09-16T05:05:41.950 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T05:05:41.967 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T05:05:42.006 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T05:05:42.261 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T05:05:42.288 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:42.326 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T05:05:42.327 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:42.353 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T05:05:42.393 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T05:05:42.679 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T05:05:42.706 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:05:42.731 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T05:05:42.745 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:42.757 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:42.797 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:43.142 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T05:05:43.168 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:43.200 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:43.418 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T05:05:43.444 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:43.484 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:43.495 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T05:05:43.521 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T05:05:43.561 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T05:05:43.864 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T05:05:43.890 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T05:05:43.930 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T05:05:44.250 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T05:05:44.276 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:44.316 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:44.545 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph. 2024-09-16T05:05:44.560 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../066-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:44.603 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:44.661 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T05:05:44.677 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:44.711 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:44.923 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T05:05:44.949 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:45.415 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:46.257 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T05:05:46.284 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:46.323 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:46.519 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T05:05:46.545 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:46.585 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:46.988 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package cephadm. 2024-09-16T05:05:47.015 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../069-cephadm_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:47.054 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:47.324 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T05:05:47.351 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T05:05:47.382 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T05:05:47.402 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package ceph. 2024-09-16T05:05:47.428 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../066-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:47.468 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:47.735 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T05:05:47.762 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T05:05:47.779 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T05:05:47.801 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T05:05:47.805 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:47.846 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:48.174 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T05:05:48.200 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:48.232 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:48.635 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package cephadm. 2024-09-16T05:05:48.662 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../069-cephadm_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:05:48.701 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:48.881 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-babel. 2024-09-16T05:05:48.908 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T05:05:48.947 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T05:05:49.012 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T05:05:49.039 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T05:05:49.079 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T05:05:49.276 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T05:05:49.302 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T05:05:49.342 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T05:05:49.406 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T05:05:49.422 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T05:05:49.465 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T05:05:49.703 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T05:05:49.730 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:05:49.770 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:50.131 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T05:05:50.158 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T05:05:50.198 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T05:05:50.495 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-babel. 2024-09-16T05:05:50.501 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-routes. 2024-09-16T05:05:50.522 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T05:05:50.527 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T05:05:50.561 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T05:05:50.567 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T05:05:50.881 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T05:05:50.903 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T05:05:50.907 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T05:05:50.930 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:05:50.947 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T05:05:50.970 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:51.317 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T05:05:51.344 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:05:51.383 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:51.720 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T05:05:51.748 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T05:05:51.786 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T05:05:52.089 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-routes. 2024-09-16T05:05:52.116 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T05:05:52.156 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T05:05:52.300 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T05:05:52.316 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T05:05:52.358 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T05:05:52.467 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T05:05:52.494 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:05:52.525 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:52.870 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T05:05:52.897 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T05:05:52.936 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T05:05:53.315 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T05:05:53.331 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T05:05:53.373 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T05:05:53.651 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T05:05:53.678 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T05:05:53.709 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T05:05:53.797 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T05:05:53.813 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T05:05:53.855 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T05:05:54.342 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T05:05:54.369 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T05:05:54.408 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T05:05:54.467 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T05:05:54.496 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:05:54.619 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:54.754 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T05:05:54.781 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T05:05:54.820 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T05:05:55.073 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T05:05:55.093 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T05:05:55.122 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T05:05:55.556 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T05:05:55.583 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T05:05:55.622 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T05:05:55.839 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T05:05:55.867 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:05:55.934 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T05:05:55.961 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T05:05:55.983 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:56.000 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T05:05:56.337 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T05:05:56.364 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T05:05:56.403 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T05:05:56.931 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T05:05:56.959 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-16T05:05:56.998 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T05:05:57.111 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T05:05:57.138 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T05:05:57.178 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T05:05:57.310 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T05:05:57.337 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T05:05:57.376 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T05:05:57.481 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T05:05:57.508 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T05:05:57.547 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T05:05:57.696 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T05:05:57.724 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T05:05:57.762 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T05:05:57.867 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T05:05:57.897 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T05:05:57.934 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T05:05:58.032 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T05:05:58.060 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T05:05:58.090 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T05:05:58.220 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T05:05:58.248 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-16T05:05:58.278 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T05:05:58.410 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T05:05:58.438 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T05:05:58.477 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T05:05:58.599 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T05:05:58.626 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T05:05:58.665 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T05:05:58.994 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T05:05:59.021 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T05:05:59.060 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T05:05:59.165 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T05:05:59.182 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:05:59.232 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:05:59.371 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T05:05:59.398 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T05:05:59.437 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T05:05:59.535 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T05:05:59.563 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T05:05:59.601 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T05:05:59.749 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T05:05:59.776 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T05:05:59.815 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T05:05:59.871 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T05:05:59.898 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T05:05:59.929 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T05:06:00.182 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package jq. 2024-09-16T05:06:00.209 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T05:06:00.248 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T05:06:00.496 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T05:06:00.501 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package socat. 2024-09-16T05:06:00.512 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:06:00.529 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T05:06:00.554 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:00.567 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T05:06:00.865 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T05:06:00.879 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T05:06:00.893 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T05:06:00.903 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T05:06:00.931 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T05:06:00.945 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T05:06:01.235 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T05:06:01.262 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T05:06:01.293 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T05:06:01.299 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-test. 2024-09-16T05:06:01.315 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../097-ceph-test_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:06:01.374 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:01.554 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package jq. 2024-09-16T05:06:01.570 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T05:06:01.612 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T05:06:01.881 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package socat. 2024-09-16T05:06:01.897 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T05:06:01.939 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T05:06:02.218 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T05:06:02.234 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T05:06:02.276 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T05:06:02.647 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package ceph-test. 2024-09-16T05:06:02.663 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../097-ceph-test_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:06:02.705 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:03.671 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T05:06:03.698 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:06:03.737 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:04.058 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T05:06:04.086 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:06:04.124 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:04.402 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T05:06:04.418 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T05:06:04.451 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T05:06:04.756 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T05:06:04.784 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T05:06:04.822 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T05:06:04.942 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T05:06:04.958 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:06:04.992 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:05.193 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T05:06:05.221 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T05:06:05.238 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T05:06:05.254 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:06:05.260 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T05:06:05.287 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:05.565 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T05:06:05.593 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T05:06:05.632 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T05:06:05.681 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package pkg-config. 2024-09-16T05:06:05.709 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T05:06:05.747 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T05:06:05.961 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T05:06:05.989 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T05:06:06.009 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T05:06:06.025 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T05:06:06.027 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T05:06:06.059 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T05:06:06.398 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T05:06:06.425 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T05:06:06.464 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T05:06:06.479 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T05:06:06.495 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T05:06:06.528 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T05:06:06.773 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T05:06:06.801 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T05:06:06.839 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T05:06:06.928 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package pkg-config. 2024-09-16T05:06:06.955 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T05:06:06.994 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T05:06:07.205 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T05:06:07.221 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T05:06:07.255 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T05:06:07.376 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T05:06:07.404 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T05:06:07.442 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T05:06:07.675 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T05:06:07.703 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T05:06:07.741 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T05:06:07.754 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T05:06:07.782 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T05:06:07.820 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T05:06:08.044 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T05:06:08.072 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T05:06:08.111 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T05:06:08.148 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-py. 2024-09-16T05:06:08.175 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-16T05:06:08.214 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T05:06:08.431 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T05:06:08.458 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T05:06:08.497 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T05:06:08.552 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T05:06:08.579 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T05:06:08.618 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T05:06:08.767 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T05:06:08.783 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T05:06:08.816 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T05:06:09.097 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T05:06:09.125 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T05:06:09.153 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-py. 2024-09-16T05:06:09.163 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T05:06:09.180 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-16T05:06:09.219 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T05:06:09.449 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-toml. 2024-09-16T05:06:09.477 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-16T05:06:09.507 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T05:06:09.565 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T05:06:09.592 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T05:06:09.631 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T05:06:09.760 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T05:06:09.787 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T05:06:09.826 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T05:06:10.051 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T05:06:10.067 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T05:06:10.118 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T05:06:10.147 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T05:06:10.174 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T05:06:10.213 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T05:06:10.429 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-toml. 2024-09-16T05:06:10.456 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-16T05:06:10.457 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T05:06:10.473 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T05:06:10.495 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T05:06:10.507 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T05:06:10.832 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T05:06:10.859 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T05:06:10.898 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T05:06:11.268 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T05:06:11.269 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package radosgw. 2024-09-16T05:06:11.295 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T05:06:11.298 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../116-radosgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:06:11.334 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T05:06:11.336 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:11.596 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T05:06:11.623 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T05:06:11.662 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T05:06:12.122 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T05:06:12.138 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:06:12.180 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:12.366 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package radosgw. 2024-09-16T05:06:12.394 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../116-radosgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:06:12.432 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:12.510 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T05:06:12.760 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T05:06:12.911 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T05:06:13.180 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T05:06:13.194 INFO:teuthology.orchestra.run.smithi149.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T05:06:13.222 INFO:teuthology.orchestra.run.smithi149.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:06:13.260 INFO:teuthology.orchestra.run.smithi149.stdout:Unpacking rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:13.507 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T05:06:13.556 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T05:06:13.790 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T05:06:13.891 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T05:06:13.946 INFO:teuthology.orchestra.run.smithi114.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T05:06:14.030 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:14.127 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T05:06:14.427 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T05:06:14.566 INFO:teuthology.orchestra.run.smithi114.stdout:Adding system user cephadm....done 2024-09-16T05:06:14.748 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T05:06:14.817 INFO:teuthology.orchestra.run.smithi149.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T05:06:14.885 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:15.258 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T05:06:15.384 INFO:teuthology.orchestra.run.smithi149.stdout:Adding system user cephadm....done 2024-09-16T05:06:15.517 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T05:06:15.532 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T05:06:15.635 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T05:06:15.800 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T05:06:15.895 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T05:06:16.051 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T05:06:16.137 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T05:06:16.177 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T05:06:16.272 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T05:06:16.437 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T05:06:16.590 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T05:06:16.679 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T05:06:16.831 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T05:06:16.950 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T05:06:17.132 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T05:06:17.218 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T05:06:17.502 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T05:06:17.736 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T05:06:17.761 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T05:06:18.021 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T05:06:18.028 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T05:06:18.255 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T05:06:18.281 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T05:06:18.497 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T05:06:18.549 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T05:06:18.765 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T05:06:18.808 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:19.016 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:19.092 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T05:06:19.243 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T05:06:19.275 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T05:06:19.394 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T05:06:19.462 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T05:06:19.545 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T05:06:19.563 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T05:06:19.664 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T05:06:19.805 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T05:06:19.923 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T05:06:20.206 INFO:teuthology.orchestra.run.smithi114.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T05:06:20.235 INFO:teuthology.orchestra.run.smithi114.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T05:06:20.291 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T05:06:20.322 INFO:teuthology.orchestra.run.smithi149.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T05:06:20.355 INFO:teuthology.orchestra.run.smithi149.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T05:06:20.417 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T05:06:20.426 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T05:06:20.552 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T05:06:20.785 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T05:06:20.920 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T05:06:20.936 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T05:06:21.046 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T05:06:21.221 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T05:06:21.314 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T05:06:21.480 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T05:06:21.573 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T05:06:21.790 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T05:06:21.891 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T05:06:21.916 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T05:06:22.017 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T05:06:22.568 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T05:06:22.735 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T05:06:22.819 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T05:06:22.970 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T05:06:23.012 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T05:06:23.104 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T05:06:23.138 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T05:06:23.264 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T05:06:23.405 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T05:06:23.556 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T05:06:23.790 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T05:06:23.949 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T05:06:24.041 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T05:06:24.200 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T05:06:24.426 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T05:06:24.569 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T05:06:24.736 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T05:06:24.845 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T05:06:25.070 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T05:06:25.171 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T05:06:25.321 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T05:06:25.423 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T05:06:25.447 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:25.548 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:25.732 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T05:06:25.841 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T05:06:26.033 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T05:06:26.167 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T05:06:27.139 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T05:06:27.323 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T05:06:27.391 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:06:27.542 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T05:06:27.817 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T05:06:27.916 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:06:27.989 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T05:06:28.050 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T05:06:28.310 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T05:06:28.328 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T05:06:28.436 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T05:06:28.554 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T05:06:28.621 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T05:06:28.847 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T05:06:28.881 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T05:06:29.106 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T05:06:29.132 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:06:29.266 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T05:06:29.357 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:06:29.483 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T05:06:29.593 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T05:06:29.784 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T05:06:29.877 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T05:06:30.027 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T05:06:30.137 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T05:06:30.295 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T05:06:30.404 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T05:06:30.529 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T05:06:30.657 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T05:06:30.780 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T05:06:30.915 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T05:06:31.016 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T05:06:31.048 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T05:06:31.174 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T05:06:31.325 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T05:06:31.451 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T05:06:31.467 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T05:06:31.584 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T05:06:31.744 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T05:06:31.894 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T05:06:32.045 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T05:06:32.195 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T05:06:32.438 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T05:06:32.455 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T05:06:32.547 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T05:06:32.581 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T05:06:32.673 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T05:06:32.691 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T05:06:32.763 INFO:teuthology.orchestra.run.smithi114.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T05:06:32.763 INFO:teuthology.orchestra.run.smithi149.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T05:06:32.983 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T05:06:33.008 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T05:06:33.126 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T05:06:33.134 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T05:06:33.510 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T05:06:33.545 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T05:06:33.770 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T05:06:33.820 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T05:06:33.896 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T05:06:33.947 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T05:06:34.147 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:06:34.214 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:06:34.273 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T05:06:34.324 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T05:06:34.607 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T05:06:34.633 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T05:06:34.925 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T05:06:34.967 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T05:06:35.226 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T05:06:35.269 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T05:06:35.327 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T05:06:35.395 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T05:06:35.603 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T05:06:35.663 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T05:06:35.863 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T05:06:35.939 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T05:06:36.224 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T05:06:36.300 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T05:06:36.626 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T05:06:36.727 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T05:06:36.911 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T05:06:37.029 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T05:06:37.255 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T05:06:37.365 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T05:06:37.381 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:37.491 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:37.507 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:37.633 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T05:06:37.635 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:37.784 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T05:06:37.944 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:38.079 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:38.187 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:38.205 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T05:06:38.338 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:38.464 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T05:06:38.506 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:38.632 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:38.741 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:38.750 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:38.842 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:38.961 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T05:06:38.961 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T05:06:38.968 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:39.191 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T05:06:39.191 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T05:06:39.401 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:39.502 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:39.620 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:39.661 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:39.754 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T05:06:39.804 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:39.930 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:40.056 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T05:06:40.933 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:41.050 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:41.151 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:41.277 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:41.293 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:41.429 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:41.452 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:41.603 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:41.746 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:41.872 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:41.963 INFO:teuthology.orchestra.run.smithi149.stdout:Adding group ceph....done 2024-09-16T05:06:42.298 INFO:teuthology.orchestra.run.smithi149.stdout:Adding system user ceph....done 2024-09-16T05:06:42.397 INFO:teuthology.orchestra.run.smithi114.stdout:Adding group ceph....done 2024-09-16T05:06:42.441 INFO:teuthology.orchestra.run.smithi149.stdout:Setting system user ceph properties....done 2024-09-16T05:06:42.454 INFO:teuthology.orchestra.run.smithi149.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T05:06:42.592 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T05:06:42.784 INFO:teuthology.orchestra.run.smithi114.stdout:Adding system user ceph....done 2024-09-16T05:06:42.924 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T05:06:42.960 INFO:teuthology.orchestra.run.smithi114.stdout:Setting system user ceph properties....done 2024-09-16T05:06:42.973 INFO:teuthology.orchestra.run.smithi114.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T05:06:43.113 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T05:06:43.380 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:43.477 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T05:06:43.507 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:43.940 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:43.966 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:06:43.966 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:06:44.041 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:44.495 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:44.498 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:06:44.498 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:06:44.937 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T05:06:44.976 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:45.376 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T05:06:45.426 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:45.646 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T05:06:45.647 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T05:06:45.862 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:46.076 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T05:06:46.076 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T05:06:46.121 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:46.336 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T05:06:46.336 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T05:06:46.522 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:46.733 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T05:06:46.733 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T05:06:46.797 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:47.072 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T05:06:47.072 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T05:06:47.173 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:47.423 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T05:06:47.423 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T05:06:47.548 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:47.674 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:47.817 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:47.908 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:48.017 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:48.026 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T05:06:48.026 INFO:teuthology.orchestra.run.smithi149.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T05:06:48.177 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:48.410 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T05:06:48.410 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T05:06:48.452 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:48.612 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:48.738 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:48.879 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:48.889 INFO:teuthology.orchestra.run.smithi149.stdout:Setting up ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:49.047 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:49.173 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:49.233 INFO:teuthology.orchestra.run.smithi149.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T05:06:49.316 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:06:49.334 INFO:teuthology.orchestra.run.smithi149.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T05:06:49.478 INFO:teuthology.orchestra.run.smithi149.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T05:06:49.660 INFO:teuthology.orchestra.run.smithi114.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T05:06:49.786 INFO:teuthology.orchestra.run.smithi114.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T05:06:49.938 INFO:teuthology.orchestra.run.smithi114.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T05:06:50.236 INFO:teuthology.orchestra.run.smithi149.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T05:06:50.930 INFO:teuthology.orchestra.run.smithi149.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T05:06:50.936 DEBUG:teuthology.parallel:result is None 2024-09-16T05:06:50.997 INFO:teuthology.orchestra.run.smithi114.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T05:06:51.662 INFO:teuthology.orchestra.run.smithi114.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T05:06:51.666 DEBUG:teuthology.parallel:result is None 2024-09-16T05:06:51.666 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:06:51.808 DEBUG:teuthology.orchestra.run.smithi114:> dpkg-query -W -f '${Version}' ceph 2024-09-16T05:06:51.827 INFO:teuthology.orchestra.run.smithi114.stdout:19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:06:51.827 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:06:51.828 INFO:teuthology.task.install:The correct ceph version 19.1.1-326-g8921c8ef-1jammy is installed. 2024-09-16T05:06:51.829 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:06:51.977 DEBUG:teuthology.orchestra.run.smithi149:> dpkg-query -W -f '${Version}' ceph 2024-09-16T05:06:51.996 INFO:teuthology.orchestra.run.smithi149.stdout:19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:06:51.997 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:06:51.997 INFO:teuthology.task.install:The correct ceph version 19.1.1-326-g8921c8ef-1jammy is installed. 2024-09-16T05:06:51.999 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T05:06:51.999 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:06:51.999 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T05:06:52.013 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:06:52.013 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T05:06:52.051 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T05:06:52.051 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:06:52.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T05:06:52.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T05:06:52.125 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:06:52.125 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T05:06:52.138 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T05:06:52.194 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T05:06:52.194 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:06:52.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T05:06:52.208 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T05:06:52.260 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:06:52.261 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T05:06:52.273 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T05:06:52.325 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T05:06:52.325 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:06:52.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T05:06:52.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T05:06:52.392 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:06:52.392 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T05:06:52.406 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T05:06:52.462 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T05:06:52.571 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T05:06:52.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /var/log/ceph 2024-09-16T05:06:52.575 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 777 /var/log/ceph 2024-09-16T05:06:52.588 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T05:06:52.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T05:06:52.631 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T05:06:52.645 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T05:06:52.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T05:06:52.687 DEBUG:teuthology.orchestra.run.smithi149:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T05:06:52.708 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T05:06:52.708 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T05:06:52.708 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906097', 'branch': 'squid', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_api_tests}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7906097', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 700, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 5616, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'sleep_before_teardown': 0, 'subset': '15927/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4', 'targets': {'smithi114.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNE/0il3+C8uzd5E8kQ/LerQQAwQhbXmAw7gc4pFg2UaFFfODWF4PiaiInerRO/H6G9S6HOuXtl8cL8olrhm2gU=', 'smithi149.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPSsAMjpZuRE7q2ekaDwBf90GoBlnSKGa6anDr0+/5foCmP6kr4QubL5rgiMiV8eCrnDEsjJbY9sSPD6taahvPw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['rados/test.sh unit_test_scan']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_21:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T05:06:52.709 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T05:06:52.743 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T05:06:52.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T05:06:52.791 DEBUG:teuthology.orchestra.run.smithi149:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T05:06:52.803 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:06:52.803 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T05:06:52.851 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T05:06:52.851 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-09-16T05:06:52.898 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T05:06:52.898 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:06:52.898 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-09-16T05:06:52.898 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:06:52.898 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-16 05:01:31.393330382 +0000 2024-09-16T05:06:52.899 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-16 05:01:31.169332383 +0000 2024-09-16T05:06:52.899 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-16 05:01:31.169332383 +0000 2024-09-16T05:06:52.899 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2024-09-16T05:06:52.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T05:06:52.953 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-16T05:06:52.954 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-16T05:06:52.954 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000351951 s, 1.5 MB/s 2024-09-16T05:06:52.954 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T05:06:53.003 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-09-16T05:06:53.054 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T05:06:53.054 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:06:53.054 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-09-16T05:06:53.054 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:06:53.054 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-16 05:01:31.841326379 +0000 2024-09-16T05:06:53.055 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-16 05:01:31.621328346 +0000 2024-09-16T05:06:53.055 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-16 05:01:31.621328346 +0000 2024-09-16T05:06:53.055 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2024-09-16T05:06:53.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T05:06:53.113 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-16T05:06:53.113 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-16T05:06:53.113 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000453723 s, 1.1 MB/s 2024-09-16T05:06:53.114 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T05:06:53.163 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-09-16T05:06:53.210 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T05:06:53.210 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:06:53.210 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 920 Links: 1 2024-09-16T05:06:53.210 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:06:53.210 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-16 05:01:32.357321772 +0000 2024-09-16T05:06:53.210 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-16 05:01:32.077324273 +0000 2024-09-16T05:06:53.210 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-16 05:01:32.077324273 +0000 2024-09-16T05:06:53.210 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2024-09-16T05:06:53.210 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T05:06:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-16T05:06:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-16T05:06:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000390828 s, 1.3 MB/s 2024-09-16T05:06:53.265 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T05:06:53.315 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-09-16T05:06:53.362 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T05:06:53.362 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:06:53.362 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 936 Links: 1 2024-09-16T05:06:53.362 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:06:53.362 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-16 05:01:32.841317448 +0000 2024-09-16T05:06:53.362 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-16 05:01:32.589319699 +0000 2024-09-16T05:06:53.362 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-16 05:01:32.589319699 +0000 2024-09-16T05:06:53.362 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2024-09-16T05:06:53.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T05:06:53.416 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-16T05:06:53.416 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-16T05:06:53.416 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000281812 s, 1.8 MB/s 2024-09-16T05:06:53.417 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T05:06:53.467 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T05:06:53.467 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:06:53.467 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T05:06:53.474 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T05:06:53.475 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_1 2024-09-16T05:06:53.524 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T05:06:53.524 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:06:53.524 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 881 Links: 1 2024-09-16T05:06:53.524 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:06:53.524 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-09-16 05:01:33.136025235 +0000 2024-09-16T05:06:53.524 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-09-16 05:01:32.904027307 +0000 2024-09-16T05:06:53.524 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-09-16 05:01:32.904027307 +0000 2024-09-16T05:06:53.524 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2024-09-16T05:06:53.525 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T05:06:53.577 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-09-16T05:06:53.577 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-09-16T05:06:53.577 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000387638 s, 1.3 MB/s 2024-09-16T05:06:53.578 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T05:06:53.628 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_2 2024-09-16T05:06:53.674 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T05:06:53.674 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:06:53.675 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 897 Links: 1 2024-09-16T05:06:53.675 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:06:53.675 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-09-16 05:01:33.680020376 +0000 2024-09-16T05:06:53.675 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-09-16 05:01:33.352023305 +0000 2024-09-16T05:06:53.675 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-09-16 05:01:33.352023305 +0000 2024-09-16T05:06:53.675 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2024-09-16T05:06:53.675 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T05:06:53.728 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-09-16T05:06:53.728 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-09-16T05:06:53.729 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000413895 s, 1.2 MB/s 2024-09-16T05:06:53.730 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T05:06:53.780 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_3 2024-09-16T05:06:53.826 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T05:06:53.826 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:06:53.826 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 920 Links: 1 2024-09-16T05:06:53.826 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:06:53.827 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-09-16 05:01:34.172015982 +0000 2024-09-16T05:06:53.827 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-09-16 05:01:33.904018374 +0000 2024-09-16T05:06:53.827 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-09-16 05:01:33.904018374 +0000 2024-09-16T05:06:53.827 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2024-09-16T05:06:53.827 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T05:06:53.880 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-09-16T05:06:53.881 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-09-16T05:06:53.881 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.00044149 s, 1.2 MB/s 2024-09-16T05:06:53.882 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T05:06:53.932 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_4 2024-09-16T05:06:53.979 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T05:06:53.979 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:06:53.979 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 930 Links: 1 2024-09-16T05:06:53.979 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:06:53.979 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-09-16 05:01:34.656011658 +0000 2024-09-16T05:06:53.979 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-09-16 05:01:34.396013981 +0000 2024-09-16T05:06:53.979 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-09-16 05:01:34.396013981 +0000 2024-09-16T05:06:53.979 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2024-09-16T05:06:53.979 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T05:06:54.033 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-09-16T05:06:54.033 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-09-16T05:06:54.033 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000386487 s, 1.3 MB/s 2024-09-16T05:06:54.034 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T05:06:54.084 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T05:06:54.085 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi149.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T05:06:54.085 INFO:tasks.ceph:Generating config... 2024-09-16T05:06:54.086 INFO:tasks.ceph:[client] debug monc = 20 2024-09-16T05:06:54.086 INFO:tasks.ceph:[client] debug ms = 1 2024-09-16T05:06:54.086 INFO:tasks.ceph:[global] debug ms = 1 2024-09-16T05:06:54.086 INFO:tasks.ceph:[global] debug objecter = 20 2024-09-16T05:06:54.086 INFO:tasks.ceph:[global] debug rados = 20 2024-09-16T05:06:54.086 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T05:06:54.086 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-16T05:06:54.086 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-16T05:06:54.086 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-16T05:06:54.087 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T05:06:54.087 INFO:tasks.ceph:[global] ms type = async 2024-09-16T05:06:54.087 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T05:06:54.087 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T05:06:54.087 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T05:06:54.087 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T05:06:54.087 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T05:06:54.087 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-16T05:06:54.087 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T05:06:54.087 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T05:06:54.087 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T05:06:54.087 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-16T05:06:54.087 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-16T05:06:54.087 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T05:06:54.088 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-16T05:06:54.088 INFO:tasks.ceph:[mon] paxos min = 10 2024-09-16T05:06:54.088 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-16T05:06:54.088 INFO:tasks.ceph:[mon] paxos trim min = 10 2024-09-16T05:06:54.088 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-16T05:06:54.088 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-16T05:06:54.088 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-09-16T05:06:54.088 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T05:06:54.088 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T05:06:54.088 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T05:06:54.088 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T05:06:54.088 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T05:06:54.088 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T05:06:54.088 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T05:06:54.089 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T05:06:54.089 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T05:06:54.089 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T05:06:54.089 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T05:06:54.089 INFO:tasks.ceph:[osd] osd class default list = * 2024-09-16T05:06:54.089 INFO:tasks.ceph:[osd] osd class load list = * 2024-09-16T05:06:54.089 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T05:06:54.089 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T05:06:54.089 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T05:06:54.089 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T05:06:54.089 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T05:06:54.089 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T05:06:54.089 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T05:06:54.089 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T05:06:54.090 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T05:06:54.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T05:06:54.126 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T05:06:54.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T05:06:54.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T05:06:54.287 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.114'), ('mon.c', '[v2:172.21.15.114:3301,v1:172.21.15.114:6790]'), ('mon.b', '172.21.15.149')] 2024-09-16T05:06:54.288 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.114,[v2:172.21.15.114:3301,v1:172.21.15.114:6790],172.21.15.149', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-16T05:06:54.289 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:06:54.289 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T05:06:54.337 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.114 --addv c '[v2:172.21.15.114:3301,v1:172.21.15.114:6790]' --add b 172.21.15.149 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:06:54.419 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:06:54.419 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool: generated fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:06:54.420 INFO:teuthology.orchestra.run.smithi114.stdout:setting min_mon_release = quincy 2024-09-16T05:06:54.420 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 0 2024-09-16T05:06:54.420 INFO:teuthology.orchestra.run.smithi114.stdout:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:06:54.420 INFO:teuthology.orchestra.run.smithi114.stdout:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:06:54.420 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:06:54.420 INFO:teuthology.orchestra.run.smithi114.stdout:min_mon_release 17 (quincy) 2024-09-16T05:06:54.420 INFO:teuthology.orchestra.run.smithi114.stdout:election_strategy: 1 2024-09-16T05:06:54.420 INFO:teuthology.orchestra.run.smithi114.stdout:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:06:54.420 INFO:teuthology.orchestra.run.smithi114.stdout:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:06:54.420 INFO:teuthology.orchestra.run.smithi114.stdout:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:06:54.421 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T05:06:54.422 DEBUG:teuthology.orchestra.run.smithi114:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T05:06:54.469 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c6dbfe9e-ab8c-4165-ae76-3f2273766e2c... 2024-09-16T05:06:54.471 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T05:06:54.515 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T05:06:54.544 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-09-16T05:06:54.544 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2024-09-16T05:06:54.544 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T05:06:54.544 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2024-09-16T05:06:54.544 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.544 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2024-09-16T05:06:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2024-09-16T05:06:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2024-09-16T05:06:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2024-09-16T05:06:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2024-09-16T05:06:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2024-09-16T05:06:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T05:06:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2024-09-16T05:06:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T05:06:54.545 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T05:06:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2024-09-16T05:06:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2024-09-16T05:06:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2024-09-16T05:06:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2024-09-16T05:06:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2024-09-16T05:06:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2024-09-16T05:06:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2024-09-16T05:06:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2024-09-16T05:06:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T05:06:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T05:06:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: mon_allow_pool_size_one = true 2024-09-16T05:06:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T05:06:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2024-09-16T05:06:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2024-09-16T05:06:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2024-09-16T05:06:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2024-09-16T05:06:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2024-09-16T05:06:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = "172.21.15.114,[v2:172.21.15.114:3301,v1:172.21.15.114:6790],172.21.15.149" 2024-09-16T05:06:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-09-16T05:06:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: debug objecter = 20 2024-09-16T05:06:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: debug rados = 20 2024-09-16T05:06:54.547 INFO:teuthology.orchestra.run.smithi114.stdout: mon client directed command retry = 5 2024-09-16T05:06:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: mon election default strategy = 1 2024-09-16T05:06:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: ms bind msgr1 = False 2024-09-16T05:06:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: ms bind msgr2 = True 2024-09-16T05:06:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: ms inject socket failures = 5000 2024-09-16T05:06:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: ms type = async 2024-09-16T05:06:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:06:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.548 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2024-09-16T05:06:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2024-09-16T05:06:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2024-09-16T05:06:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2024-09-16T05:06:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock profile = high_recovery_ops 2024-09-16T05:06:54.548 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2024-09-16T05:06:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2024-09-16T05:06:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2024-09-16T05:06:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2024-09-16T05:06:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2024-09-16T05:06:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug trim objects = true 2024-09-16T05:06:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2024-09-16T05:06:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2024-09-16T05:06:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2024-09-16T05:06:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.549 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2024-09-16T05:06:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2024-09-16T05:06:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2024-09-16T05:06:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2024-09-16T05:06:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2024-09-16T05:06:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2024-09-16T05:06:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: bdev async discard = True 2024-09-16T05:06:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: bdev enable discard = True 2024-09-16T05:06:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore allocator = stupid 2024-09-16T05:06:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore block size = 96636764160 2024-09-16T05:06:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore fsck on mount = True 2024-09-16T05:06:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore zero block detection = True 2024-09-16T05:06:54.550 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluefs = 20 2024-09-16T05:06:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluestore = 20 2024-09-16T05:06:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-09-16T05:06:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 20 2024-09-16T05:06:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: debug rocksdb = 10 2024-09-16T05:06:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T05:06:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd full ratio = 0.9 2024-09-16T05:06:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd nearfull ratio = 0.8 2024-09-16T05:06:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: osd class default list = * 2024-09-16T05:06:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: osd class load list = * 2024-09-16T05:06:54.551 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify cached snaps = True 2024-09-16T05:06:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify missing on start = True 2024-09-16T05:06:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: osd failsafe full ratio = 0.95 2024-09-16T05:06:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock override recovery settings = True 2024-09-16T05:06:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: osd objectstore = bluestore 2024-09-16T05:06:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: osd op queue = debug_random 2024-09-16T05:06:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: osd op queue cut off = debug_random 2024-09-16T05:06:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.552 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2024-09-16T05:06:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-09-16T05:06:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2024-09-16T05:06:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-09-16T05:06:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-09-16T05:06:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-09-16T05:06:54.552 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-09-16T05:06:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2024-09-16T05:06:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.553 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2024-09-16T05:06:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-09-16T05:06:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-09-16T05:06:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2024-09-16T05:06:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-09-16T05:06:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2024-09-16T05:06:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2024-09-16T05:06:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-09-16T05:06:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2024-09-16T05:06:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2024-09-16T05:06:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-09-16T05:06:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.553 INFO:teuthology.orchestra.run.smithi114.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T05:06:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T05:06:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T05:06:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T05:06:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T05:06:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T05:06:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: # 1m isn't quite enough 2024-09-16T05:06:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: mon_down_mkfs_grace = 2m 2024-09-16T05:06:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_filestore_osds = false 2024-09-16T05:06:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: mon min osdmap epochs = 25 2024-09-16T05:06:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: mon osdmap full prune interval = 2 2024-09-16T05:06:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: mon osdmap full prune min = 15 2024-09-16T05:06:54.554 INFO:teuthology.orchestra.run.smithi114.stdout: mon osdmap full prune txsize = 2 2024-09-16T05:06:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T05:06:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: mon scrub inject missing keys = 0.05 2024-09-16T05:06:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: mon scrub interval = 300 2024-09-16T05:06:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on pool no app = False 2024-09-16T05:06:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: paxos min = 10 2024-09-16T05:06:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: paxos service trim min = 5 2024-09-16T05:06:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: paxos trim min = 10 2024-09-16T05:06:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:06:54.555 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2024-09-16T05:06:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2024-09-16T05:06:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2024-09-16T05:06:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2024-09-16T05:06:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T05:06:54.555 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T05:06:54.556 INFO:teuthology.orchestra.run.smithi114.stdout: debug monc = 20 2024-09-16T05:06:54.556 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-09-16T05:06:54.556 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2024-09-16T05:06:54.556 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.c] 2024-09-16T05:06:54.556 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.b] 2024-09-16T05:06:54.559 INFO:teuthology.orchestra.run.smithi149.stdout:[global] 2024-09-16T05:06:54.559 INFO:teuthology.orchestra.run.smithi149.stdout: chdir = "" 2024-09-16T05:06:54.559 INFO:teuthology.orchestra.run.smithi149.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T05:06:54.559 INFO:teuthology.orchestra.run.smithi149.stdout: auth supported = cephx 2024-09-16T05:06:54.559 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.559 INFO:teuthology.orchestra.run.smithi149.stdout: filestore xattr use omap = true 2024-09-16T05:06:54.559 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.559 INFO:teuthology.orchestra.run.smithi149.stdout: mon clock drift allowed = 1.000 2024-09-16T05:06:54.559 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.559 INFO:teuthology.orchestra.run.smithi149.stdout: osd crush chooseleaf type = 0 2024-09-16T05:06:54.560 INFO:teuthology.orchestra.run.smithi149.stdout: auth debug = true 2024-09-16T05:06:54.560 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.560 INFO:teuthology.orchestra.run.smithi149.stdout: ms die on old message = true 2024-09-16T05:06:54.560 INFO:teuthology.orchestra.run.smithi149.stdout: ms die on bug = true 2024-09-16T05:06:54.560 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.560 INFO:teuthology.orchestra.run.smithi149.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T05:06:54.560 INFO:teuthology.orchestra.run.smithi149.stdout: mon pg warn max object skew = 0 2024-09-16T05:06:54.560 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.560 INFO:teuthology.orchestra.run.smithi149.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T05:06:54.560 INFO:teuthology.orchestra.run.smithi149.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T05:06:54.560 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.560 INFO:teuthology.orchestra.run.smithi149.stdout: osd pool default size = 2 2024-09-16T05:06:54.561 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.561 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd allow primary affinity = true 2024-09-16T05:06:54.561 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd allow pg remap = true 2024-09-16T05:06:54.561 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on legacy crush tunables = false 2024-09-16T05:06:54.561 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on crush straw calc version zero = false 2024-09-16T05:06:54.561 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on no sortbitwise = false 2024-09-16T05:06:54.561 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on osd down out interval zero = false 2024-09-16T05:06:54.561 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on too few osds = false 2024-09-16T05:06:54.561 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T05:06:54.561 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T05:06:54.562 INFO:teuthology.orchestra.run.smithi149.stdout: mon_allow_pool_size_one = true 2024-09-16T05:06:54.562 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.562 INFO:teuthology.orchestra.run.smithi149.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T05:06:54.562 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.562 INFO:teuthology.orchestra.run.smithi149.stdout: osd default data pool replay window = 5 2024-09-16T05:06:54.562 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.562 INFO:teuthology.orchestra.run.smithi149.stdout: mon allow pool delete = true 2024-09-16T05:06:54.562 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.562 INFO:teuthology.orchestra.run.smithi149.stdout: mon cluster log file level = debug 2024-09-16T05:06:54.562 INFO:teuthology.orchestra.run.smithi149.stdout: debug asserts on shutdown = true 2024-09-16T05:06:54.562 INFO:teuthology.orchestra.run.smithi149.stdout: mon health detail to clog = false 2024-09-16T05:06:54.562 INFO:teuthology.orchestra.run.smithi149.stdout: mon host = "172.21.15.114,[v2:172.21.15.114:3301,v1:172.21.15.114:6790],172.21.15.149" 2024-09-16T05:06:54.562 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2024-09-16T05:06:54.563 INFO:teuthology.orchestra.run.smithi149.stdout: debug objecter = 20 2024-09-16T05:06:54.563 INFO:teuthology.orchestra.run.smithi149.stdout: debug rados = 20 2024-09-16T05:06:54.563 INFO:teuthology.orchestra.run.smithi149.stdout: mon client directed command retry = 5 2024-09-16T05:06:54.563 INFO:teuthology.orchestra.run.smithi149.stdout: mon election default strategy = 1 2024-09-16T05:06:54.563 INFO:teuthology.orchestra.run.smithi149.stdout: ms bind msgr1 = False 2024-09-16T05:06:54.563 INFO:teuthology.orchestra.run.smithi149.stdout: ms bind msgr2 = True 2024-09-16T05:06:54.563 INFO:teuthology.orchestra.run.smithi149.stdout: ms inject socket failures = 5000 2024-09-16T05:06:54.563 INFO:teuthology.orchestra.run.smithi149.stdout: ms type = async 2024-09-16T05:06:54.563 INFO:teuthology.orchestra.run.smithi149.stdout: fsid = c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:06:54.563 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.563 INFO:teuthology.orchestra.run.smithi149.stdout:[osd] 2024-09-16T05:06:54.563 INFO:teuthology.orchestra.run.smithi149.stdout: osd journal size = 100 2024-09-16T05:06:54.563 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.563 INFO:teuthology.orchestra.run.smithi149.stdout: osd scrub load threshold = 5.0 2024-09-16T05:06:54.564 INFO:teuthology.orchestra.run.smithi149.stdout: osd scrub max interval = 600 2024-09-16T05:06:54.564 INFO:teuthology.orchestra.run.smithi149.stdout: osd mclock profile = high_recovery_ops 2024-09-16T05:06:54.564 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.564 INFO:teuthology.orchestra.run.smithi149.stdout: osd recover clone overlap = true 2024-09-16T05:06:54.564 INFO:teuthology.orchestra.run.smithi149.stdout: osd recovery max chunk = 1048576 2024-09-16T05:06:54.564 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.565 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug shutdown = true 2024-09-16T05:06:54.565 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug op order = true 2024-09-16T05:06:54.565 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug verify stray on activate = true 2024-09-16T05:06:54.565 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug trim objects = true 2024-09-16T05:06:54.565 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.565 INFO:teuthology.orchestra.run.smithi149.stdout: osd open classes on start = true 2024-09-16T05:06:54.565 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug pg log writeout = true 2024-09-16T05:06:54.565 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.565 INFO:teuthology.orchestra.run.smithi149.stdout: osd deep scrub update digest min age = 30 2024-09-16T05:06:54.565 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.565 INFO:teuthology.orchestra.run.smithi149.stdout: osd map max advance = 10 2024-09-16T05:06:54.565 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.565 INFO:teuthology.orchestra.run.smithi149.stdout: journal zero on create = true 2024-09-16T05:06:54.566 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.566 INFO:teuthology.orchestra.run.smithi149.stdout: filestore ondisk finisher threads = 3 2024-09-16T05:06:54.566 INFO:teuthology.orchestra.run.smithi149.stdout: filestore apply finisher threads = 3 2024-09-16T05:06:54.566 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.566 INFO:teuthology.orchestra.run.smithi149.stdout: bdev debug aio = true 2024-09-16T05:06:54.566 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug misdirected ops = true 2024-09-16T05:06:54.566 INFO:teuthology.orchestra.run.smithi149.stdout: bdev async discard = True 2024-09-16T05:06:54.566 INFO:teuthology.orchestra.run.smithi149.stdout: bdev enable discard = True 2024-09-16T05:06:54.566 INFO:teuthology.orchestra.run.smithi149.stdout: bluestore allocator = stupid 2024-09-16T05:06:54.566 INFO:teuthology.orchestra.run.smithi149.stdout: bluestore block size = 96636764160 2024-09-16T05:06:54.566 INFO:teuthology.orchestra.run.smithi149.stdout: bluestore fsck on mount = True 2024-09-16T05:06:54.566 INFO:teuthology.orchestra.run.smithi149.stdout: bluestore zero block detection = True 2024-09-16T05:06:54.566 INFO:teuthology.orchestra.run.smithi149.stdout: debug bluefs = 20 2024-09-16T05:06:54.567 INFO:teuthology.orchestra.run.smithi149.stdout: debug bluestore = 20 2024-09-16T05:06:54.567 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2024-09-16T05:06:54.567 INFO:teuthology.orchestra.run.smithi149.stdout: debug osd = 20 2024-09-16T05:06:54.567 INFO:teuthology.orchestra.run.smithi149.stdout: debug rocksdb = 10 2024-09-16T05:06:54.567 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T05:06:54.567 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd full ratio = 0.9 2024-09-16T05:06:54.567 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd nearfull ratio = 0.8 2024-09-16T05:06:54.567 INFO:teuthology.orchestra.run.smithi149.stdout: osd class default list = * 2024-09-16T05:06:54.567 INFO:teuthology.orchestra.run.smithi149.stdout: osd class load list = * 2024-09-16T05:06:54.567 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug verify cached snaps = True 2024-09-16T05:06:54.567 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug verify missing on start = True 2024-09-16T05:06:54.568 INFO:teuthology.orchestra.run.smithi149.stdout: osd failsafe full ratio = 0.95 2024-09-16T05:06:54.568 INFO:teuthology.orchestra.run.smithi149.stdout: osd mclock override recovery settings = True 2024-09-16T05:06:54.568 INFO:teuthology.orchestra.run.smithi149.stdout: osd objectstore = bluestore 2024-09-16T05:06:54.568 INFO:teuthology.orchestra.run.smithi149.stdout: osd op queue = debug_random 2024-09-16T05:06:54.568 INFO:teuthology.orchestra.run.smithi149.stdout: osd op queue cut off = debug_random 2024-09-16T05:06:54.568 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.568 INFO:teuthology.orchestra.run.smithi149.stdout:[mgr] 2024-09-16T05:06:54.568 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2024-09-16T05:06:54.568 INFO:teuthology.orchestra.run.smithi149.stdout: debug mgr = 20 2024-09-16T05:06:54.568 INFO:teuthology.orchestra.run.smithi149.stdout: debug mon = 20 2024-09-16T05:06:54.568 INFO:teuthology.orchestra.run.smithi149.stdout: debug auth = 20 2024-09-16T05:06:54.568 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min pgs per osd = 4 2024-09-16T05:06:54.568 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min bytes per osd = 10 2024-09-16T05:06:54.568 INFO:teuthology.orchestra.run.smithi149.stdout: mgr/telemetry/nag = false 2024-09-16T05:06:54.569 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.569 INFO:teuthology.orchestra.run.smithi149.stdout:[mon] 2024-09-16T05:06:54.569 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2024-09-16T05:06:54.569 INFO:teuthology.orchestra.run.smithi149.stdout: debug mon = 20 2024-09-16T05:06:54.569 INFO:teuthology.orchestra.run.smithi149.stdout: debug paxos = 20 2024-09-16T05:06:54.569 INFO:teuthology.orchestra.run.smithi149.stdout: debug auth = 20 2024-09-16T05:06:54.569 INFO:teuthology.orchestra.run.smithi149.stdout: mon data avail warn = 5 2024-09-16T05:06:54.569 INFO:teuthology.orchestra.run.smithi149.stdout: mon mgr mkfs grace = 240 2024-09-16T05:06:54.569 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min pgs per osd = 4 2024-09-16T05:06:54.569 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd reporter subtree level = osd 2024-09-16T05:06:54.570 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd prime pg temp = true 2024-09-16T05:06:54.570 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min bytes per osd = 10 2024-09-16T05:06:54.570 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.570 INFO:teuthology.orchestra.run.smithi149.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T05:06:54.570 INFO:teuthology.orchestra.run.smithi149.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T05:06:54.570 INFO:teuthology.orchestra.run.smithi149.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T05:06:54.570 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.570 INFO:teuthology.orchestra.run.smithi149.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T05:06:54.570 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T05:06:54.570 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T05:06:54.570 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.570 INFO:teuthology.orchestra.run.smithi149.stdout: # 1m isn't quite enough 2024-09-16T05:06:54.570 INFO:teuthology.orchestra.run.smithi149.stdout: mon_down_mkfs_grace = 2m 2024-09-16T05:06:54.570 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.570 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_filestore_osds = false 2024-09-16T05:06:54.571 INFO:teuthology.orchestra.run.smithi149.stdout: mon min osdmap epochs = 25 2024-09-16T05:06:54.571 INFO:teuthology.orchestra.run.smithi149.stdout: mon osdmap full prune interval = 2 2024-09-16T05:06:54.571 INFO:teuthology.orchestra.run.smithi149.stdout: mon osdmap full prune min = 15 2024-09-16T05:06:54.571 INFO:teuthology.orchestra.run.smithi149.stdout: mon osdmap full prune txsize = 2 2024-09-16T05:06:54.571 INFO:teuthology.orchestra.run.smithi149.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T05:06:54.571 INFO:teuthology.orchestra.run.smithi149.stdout: mon scrub inject missing keys = 0.05 2024-09-16T05:06:54.571 INFO:teuthology.orchestra.run.smithi149.stdout: mon scrub interval = 300 2024-09-16T05:06:54.571 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on pool no app = False 2024-09-16T05:06:54.571 INFO:teuthology.orchestra.run.smithi149.stdout: paxos min = 10 2024-09-16T05:06:54.572 INFO:teuthology.orchestra.run.smithi149.stdout: paxos service trim min = 5 2024-09-16T05:06:54.572 INFO:teuthology.orchestra.run.smithi149.stdout: paxos trim min = 10 2024-09-16T05:06:54.572 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-09-16T05:06:54.572 INFO:teuthology.orchestra.run.smithi149.stdout:[client] 2024-09-16T05:06:54.572 INFO:teuthology.orchestra.run.smithi149.stdout: rgw cache enabled = true 2024-09-16T05:06:54.572 INFO:teuthology.orchestra.run.smithi149.stdout: rgw enable ops log = true 2024-09-16T05:06:54.572 INFO:teuthology.orchestra.run.smithi149.stdout: rgw enable usage log = true 2024-09-16T05:06:54.572 INFO:teuthology.orchestra.run.smithi149.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T05:06:54.572 INFO:teuthology.orchestra.run.smithi149.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T05:06:54.572 INFO:teuthology.orchestra.run.smithi149.stdout: debug monc = 20 2024-09-16T05:06:54.572 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2024-09-16T05:06:54.572 INFO:teuthology.orchestra.run.smithi149.stdout:[mon.a] 2024-09-16T05:06:54.572 INFO:teuthology.orchestra.run.smithi149.stdout:[mon.c] 2024-09-16T05:06:54.572 INFO:teuthology.orchestra.run.smithi149.stdout:[mon.b] 2024-09-16T05:06:54.573 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T05:06:54.573 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T05:06:54.645 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T05:06:54.645 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:06:54.645 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T05:06:54.693 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:06:54.694 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T05:06:54.741 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T05:06:54.741 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:06:54.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T05:06:54.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T05:06:54.807 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:06:54.807 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:06:54.854 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi149.front.sepia.ceph.com 2024-09-16T05:06:54.854 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:06:54.854 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T05:06:54.854 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T05:06:54.878 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:06:54.878 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:06:54.927 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T05:06:54.927 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T05:06:54.927 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T05:06:55.011 INFO:teuthology.orchestra.run.smithi149.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T05:06:55.020 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T05:06:55.021 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T05:06:55.021 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T05:06:55.095 INFO:teuthology.orchestra.run.smithi149.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T05:06:55.104 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T05:06:55.105 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi149.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T05:06:55.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T05:06:55.118 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T05:06:55.118 INFO:tasks.ceph:role: osd.0 2024-09-16T05:06:55.118 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T05:06:55.118 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T05:06:55.181 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T05:06:55.182 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T05:06:55.182 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T05:06:55.182 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T05:06:55.182 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T05:06:55.182 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-09-16T05:06:55.182 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T05:06:55.182 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T05:06:55.182 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T05:06:55.182 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T05:06:55.674 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-09-16T05:06:55.675 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-09-16T05:06:55.675 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T05:06:55.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T05:06:55.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:06:55.843 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T05:06:55.843 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T05:06:55.902 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T05:06:55.902 INFO:tasks.ceph:role: osd.1 2024-09-16T05:06:55.902 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T05:06:55.902 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T05:06:55.965 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T05:06:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T05:06:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T05:06:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T05:06:55.965 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T05:06:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-09-16T05:06:55.965 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T05:06:55.965 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T05:06:55.966 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T05:06:55.966 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T05:06:56.404 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-09-16T05:06:56.406 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-09-16T05:06:56.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T05:06:56.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T05:06:56.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:06:56.546 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T05:06:56.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T05:06:56.600 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T05:06:56.600 INFO:tasks.ceph:role: osd.2 2024-09-16T05:06:56.600 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T05:06:56.600 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T05:06:56.663 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T05:06:56.663 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T05:06:56.663 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T05:06:56.663 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T05:06:56.663 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T05:06:56.663 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-09-16T05:06:56.663 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T05:06:56.663 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T05:06:56.663 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T05:06:56.664 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T05:06:57.125 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-09-16T05:06:57.127 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-09-16T05:06:57.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T05:06:57.273 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T05:06:57.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:06:57.287 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T05:06:57.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:06:57.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:06:57.394+0000 7f1de61137c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T05:06:57.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:06:57.398+0000 7f1de61137c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T05:06:57.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:06:57.398+0000 7f1de61137c0 -1 bdev(0x556a9c1d1800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T05:06:57.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:06:57.398+0000 7f1de61137c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T05:07:00.823 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T05:07:00.836 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:07:00.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:00.918+0000 7f5c54d247c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T05:07:00.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:00.918+0000 7f5c54d247c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T05:07:00.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:00.918+0000 7f5c54d247c0 -1 bdev(0x5624b1801800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T05:07:00.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:00.918+0000 7f5c54d247c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T05:07:04.617 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T05:07:04.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:07:04.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:04.758+0000 7f6ef2e2a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T05:07:04.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:04.758+0000 7f6ef2e2a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T05:07:04.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:04.758+0000 7f6ef2e2a7c0 -1 bdev(0x55ec9a8cd800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T05:07:04.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:04.758+0000 7f6ef2e2a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T05:07:08.144 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T05:07:08.199 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T05:07:08.214 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T05:07:08.214 INFO:tasks.ceph:role: osd.3 2024-09-16T05:07:08.214 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi149.front.sepia.ceph.com 2024-09-16T05:07:08.214 DEBUG:teuthology.orchestra.run.smithi149:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T05:07:08.278 INFO:teuthology.orchestra.run.smithi149.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T05:07:08.278 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T05:07:08.279 INFO:teuthology.orchestra.run.smithi149.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T05:07:08.279 INFO:teuthology.orchestra.run.smithi149.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T05:07:08.279 INFO:teuthology.orchestra.run.smithi149.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T05:07:08.279 INFO:teuthology.orchestra.run.smithi149.stdout: = sunit=0 swidth=0 blks 2024-09-16T05:07:08.279 INFO:teuthology.orchestra.run.smithi149.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T05:07:08.279 INFO:teuthology.orchestra.run.smithi149.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T05:07:08.279 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T05:07:08.279 INFO:teuthology.orchestra.run.smithi149.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T05:07:08.762 INFO:teuthology.orchestra.run.smithi149.stdout:Discarding blocks...Done. 2024-09-16T05:07:08.764 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi149.front.sepia.ceph.com -o noatime 2024-09-16T05:07:08.765 DEBUG:teuthology.orchestra.run.smithi149:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T05:07:08.910 DEBUG:teuthology.orchestra.run.smithi149:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T05:07:08.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:07:08.922 INFO:teuthology.orchestra.run.smithi149.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T05:07:08.922 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T05:07:08.979 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T05:07:08.979 INFO:tasks.ceph:role: osd.4 2024-09-16T05:07:08.979 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi149.front.sepia.ceph.com 2024-09-16T05:07:08.979 DEBUG:teuthology.orchestra.run.smithi149:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T05:07:09.045 INFO:teuthology.orchestra.run.smithi149.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T05:07:09.045 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T05:07:09.045 INFO:teuthology.orchestra.run.smithi149.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T05:07:09.046 INFO:teuthology.orchestra.run.smithi149.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T05:07:09.046 INFO:teuthology.orchestra.run.smithi149.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T05:07:09.046 INFO:teuthology.orchestra.run.smithi149.stdout: = sunit=0 swidth=0 blks 2024-09-16T05:07:09.046 INFO:teuthology.orchestra.run.smithi149.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T05:07:09.046 INFO:teuthology.orchestra.run.smithi149.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T05:07:09.046 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T05:07:09.046 INFO:teuthology.orchestra.run.smithi149.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T05:07:09.494 INFO:teuthology.orchestra.run.smithi149.stdout:Discarding blocks...Done. 2024-09-16T05:07:09.496 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi149.front.sepia.ceph.com -o noatime 2024-09-16T05:07:09.496 DEBUG:teuthology.orchestra.run.smithi149:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T05:07:09.587 DEBUG:teuthology.orchestra.run.smithi149:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T05:07:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:07:09.597 INFO:teuthology.orchestra.run.smithi149.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T05:07:09.597 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T05:07:09.648 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T05:07:09.648 INFO:tasks.ceph:role: osd.5 2024-09-16T05:07:09.648 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi149.front.sepia.ceph.com 2024-09-16T05:07:09.648 DEBUG:teuthology.orchestra.run.smithi149:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T05:07:09.709 INFO:teuthology.orchestra.run.smithi149.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T05:07:09.709 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T05:07:09.709 INFO:teuthology.orchestra.run.smithi149.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T05:07:09.709 INFO:teuthology.orchestra.run.smithi149.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T05:07:09.709 INFO:teuthology.orchestra.run.smithi149.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T05:07:09.709 INFO:teuthology.orchestra.run.smithi149.stdout: = sunit=0 swidth=0 blks 2024-09-16T05:07:09.710 INFO:teuthology.orchestra.run.smithi149.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T05:07:09.710 INFO:teuthology.orchestra.run.smithi149.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T05:07:09.710 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T05:07:09.710 INFO:teuthology.orchestra.run.smithi149.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T05:07:10.165 INFO:teuthology.orchestra.run.smithi149.stdout:Discarding blocks...Done. 2024-09-16T05:07:10.167 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi149.front.sepia.ceph.com -o noatime 2024-09-16T05:07:10.167 DEBUG:teuthology.orchestra.run.smithi149:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-16T05:07:10.263 DEBUG:teuthology.orchestra.run.smithi149:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T05:07:10.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:07:10.273 INFO:teuthology.orchestra.run.smithi149.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T05:07:10.274 DEBUG:teuthology.orchestra.run.smithi149:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:07:10.390 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:10.389+0000 7f1f4c27f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T05:07:10.396 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:10.393+0000 7f1f4c27f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T05:07:10.396 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:10.393+0000 7f1f4c27f7c0 -1 bdev(0x557f9949f800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T05:07:10.396 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:10.393+0000 7f1f4c27f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T05:07:13.789 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T05:07:13.803 DEBUG:teuthology.orchestra.run.smithi149:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:07:13.883 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:13.881+0000 7f5b5a64b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T05:07:13.883 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:13.881+0000 7f5b5a64b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T05:07:13.884 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:13.881+0000 7f5b5a64b7c0 -1 bdev(0x5615fef95800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T05:07:13.884 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:13.881+0000 7f5b5a64b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T05:07:17.046 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T05:07:17.104 DEBUG:teuthology.orchestra.run.smithi149:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:07:17.189 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:17.185+0000 7fa766d207c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T05:07:17.190 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:17.189+0000 7fa766d207c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T05:07:17.190 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:17.189+0000 7fa766d207c0 -1 bdev(0x55786eecd800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T05:07:17.190 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:17.189+0000 7fa766d207c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T05:07:20.349 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T05:07:20.407 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T05:07:20.407 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:07:20.407 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T05:07:20.422 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:07:20.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T05:07:20.477 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:07:20.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T05:07:20.533 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:07:20.533 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T05:07:20.547 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:07:20.547 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T05:07:20.602 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:07:20.602 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T05:07:20.662 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:07:20.662 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T05:07:20.721 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:07:20.721 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T05:07:20.770 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T05:07:20.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T05:07:20.773 DEBUG:teuthology.orchestra.run.smithi149:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T05:07:20.816 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.0] 2024-09-16T05:07:20.816 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDxvOdmtc+xFxAAZtiaAYOx7RGIfZhip8E56g== 2024-09-16T05:07:20.816 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.1] 2024-09-16T05:07:20.816 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQD0vOdm8o7bNhAAkeWJRlAhTvrczqBswZdF2w== 2024-09-16T05:07:20.816 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.2] 2024-09-16T05:07:20.816 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQD4vOdmUwNuLRAAObnANGWAET93xHZZsa0fMg== 2024-09-16T05:07:20.817 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr.x] 2024-09-16T05:07:20.817 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDvvOdmBke1ABAA5GRqjuwX7CB4v1+SnPSJxw== 2024-09-16T05:07:20.817 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.3] 2024-09-16T05:07:20.817 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQD+vOdm9wdXFxAA4LPE9iivRMK1yuNfD3lrrg== 2024-09-16T05:07:20.817 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.4] 2024-09-16T05:07:20.817 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQABvedmccOyNBAA6kjtpW/EeHYy3uIRiSZi+g== 2024-09-16T05:07:20.817 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.5] 2024-09-16T05:07:20.817 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAFvedmfyxfCxAAw5TPl9XNIWqrWu8994w0jQ== 2024-09-16T05:07:20.817 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-09-16T05:07:20.817 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDvvOdm+o21BRAAy1RhVfXIw8tVyEbwPvLCww== 2024-09-16T05:07:20.824 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.0] 2024-09-16T05:07:20.824 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDxvOdmtc+xFxAAZtiaAYOx7RGIfZhip8E56g== 2024-09-16T05:07:20.824 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.1] 2024-09-16T05:07:20.824 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQD0vOdm8o7bNhAAkeWJRlAhTvrczqBswZdF2w== 2024-09-16T05:07:20.824 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.2] 2024-09-16T05:07:20.824 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQD4vOdmUwNuLRAAObnANGWAET93xHZZsa0fMg== 2024-09-16T05:07:20.824 INFO:teuthology.orchestra.run.smithi149.stdout:[mgr.x] 2024-09-16T05:07:20.824 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDvvOdmBke1ABAA5GRqjuwX7CB4v1+SnPSJxw== 2024-09-16T05:07:20.825 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.3] 2024-09-16T05:07:20.825 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQD+vOdm9wdXFxAA4LPE9iivRMK1yuNfD3lrrg== 2024-09-16T05:07:20.825 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.4] 2024-09-16T05:07:20.825 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQABvedmccOyNBAA6kjtpW/EeHYy3uIRiSZi+g== 2024-09-16T05:07:20.825 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.5] 2024-09-16T05:07:20.825 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQAFvedmfyxfCxAAw5TPl9XNIWqrWu8994w0jQ== 2024-09-16T05:07:20.825 INFO:teuthology.orchestra.run.smithi149.stdout:[client.0] 2024-09-16T05:07:20.825 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDvvOdm+o21BRAAy1RhVfXIw8tVyEbwPvLCww== 2024-09-16T05:07:20.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:07:20.862 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:07:20.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:07:20.950 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:07:20.999 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:07:21.031 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:07:21.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T05:07:21.114 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T05:07:21.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:07:21.196 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:07:21.236 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:07:21.279 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:07:21.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:07:21.362 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:07:21.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T05:07:21.447 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T05:07:21.493 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T05:07:21.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T05:07:21.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T05:07:21.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T05:07:21.608 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T05:07:21.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T05:07:21.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T05:07:21.759 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T05:07:21.774 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T05:07:21.870 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T05:07:21.885 DEBUG:teuthology.orchestra.run.smithi114:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:07:21.887 DEBUG:teuthology.orchestra.run.smithi149:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:07:21.933 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T05:07:21.933 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T05:07:21.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T05:07:21.936 INFO:tasks.ceph.mon.a:Started 2024-09-16T05:07:21.936 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T05:07:21.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T05:07:21.938 INFO:tasks.ceph.mon.c:Started 2024-09-16T05:07:21.938 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T05:07:21.938 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T05:07:21.979 INFO:tasks.ceph.mon.b:Started 2024-09-16T05:07:21.979 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T05:07:21.979 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T05:07:21.979 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T05:07:21.981 INFO:tasks.ceph.mgr.x:Started 2024-09-16T05:07:21.981 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T05:07:21.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T05:07:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 10 monclient: get_monmap_and_config 2024-09-16T05:07:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 10 monclient: build_initial_monmap 2024-09-16T05:07:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 10 monclient: monmap: 2024-09-16T05:07:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:22.102037+0000 2024-09-16T05:07:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:22.102037+0000 2024-09-16T05:07:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:22.100 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 1 Processor -- start 2024-09-16T05:07:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 1 -- start start 2024-09-16T05:07:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 10 monclient: init 2024-09-16T05:07:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-16T05:07:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 0x7fb9ec09dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 10 monclient(hunting): picked mon.noname-b con 0x7fb9ec0a0ec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 0x7fb9ec09e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 10 monclient(hunting): picked mon.noname-c con 0x7fb9ec09e000 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 0x7fb9ec13aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 10 monclient(hunting): picked mon.noname-a con 0x7fb9ec138630 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb9ec0994b0 con 0x7fb9ec138630 2024-09-16T05:07:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fb9ec09a760 con 0x7fb9ec09e000 2024-09-16T05:07:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb9ec0a1870 con 0x7fb9ec0a0ec0 2024-09-16T05:07:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9f0f86640 10 monclient(hunting): authenticate will time out at 926.730936s 2024-09-16T05:07:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ea59c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 0x7fb9ec09dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ea59c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 0x7fb9ec09dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:46406/0 (socket says 172.21.15.114:46406) 2024-09-16T05:07:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ead9d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 0x7fb9ec13aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ea59c640 1 -- 172.21.15.114:0/696886415 learned_addr learned my addr 172.21.15.114:0/696886415 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ea59c640 10 monclient(hunting): get_auth_request con 0x7fb9ec0a0ec0 auth_method 0 2024-09-16T05:07:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ead9d640 10 monclient(hunting): get_auth_request con 0x7fb9ec138630 auth_method 0 2024-09-16T05:07:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ea59c640 1 -- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 msgr2=0x7fb9ec09dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:07:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ea59c640 1 -- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 msgr2=0x7fb9ec09dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:07:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 0x7fb9ec09dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:07:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9e9d9b640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 0x7fb9ec09e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 0x7fb9ec09dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:07:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9e9d9b640 10 monclient(hunting): get_auth_request con 0x7fb9ec09e000 auth_method 0 2024-09-16T05:07:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ead9d640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 msgr2=0x7fb9ec13aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:07:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ead9d640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 msgr2=0x7fb9ec13aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:07:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ead9d640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 0x7fb9ec13aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:07:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9ead9d640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 0x7fb9ec13aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:07:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9e9d9b640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 msgr2=0x7fb9ec09e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:07:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9e9d9b640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 msgr2=0x7fb9ec09e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:07:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9e9d9b640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 0x7fb9ec09e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:07:22.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.098+0000 7fb9e9d9b640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 0x7fb9ec09e3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:07:22.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ead9d640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 0x7fb9ec13aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:22.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 0x7fb9ec09dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:22.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ead9d640 10 monclient(hunting): get_auth_request con 0x7fb9ec138630 auth_method 0 2024-09-16T05:07:22.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:22.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:22.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ead9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:22.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ea59c640 10 monclient(hunting): get_auth_request con 0x7fb9ec0a0ec0 auth_method 0 2024-09-16T05:07:22.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:22.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:22.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ea59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:22.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ead9d640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 msgr2=0x7fb9ec13aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:07:22.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ead9d640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 msgr2=0x7fb9ec13aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:07:22.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ead9d640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 0x7fb9ec13aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:07:22.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9e9d9b640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 0x7fb9ec09e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:22.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9e9d9b640 10 monclient(hunting): get_auth_request con 0x7fb9ec09e000 auth_method 0 2024-09-16T05:07:22.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ea59c640 1 -- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 msgr2=0x7fb9ec09dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:07:22.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:22.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:22.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9e9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:22.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ea59c640 1 -- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 msgr2=0x7fb9ec09dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:07:22.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 0x7fb9ec09dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:07:22.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ead9d640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 0x7fb9ec13aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:07:22.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 0x7fb9ec09dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:07:22.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9e9d9b640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 msgr2=0x7fb9ec09e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:07:22.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9e9d9b640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 msgr2=0x7fb9ec09e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:07:22.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9e9d9b640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 0x7fb9ec09e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:07:22.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.302+0000 7fb9e9d9b640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 0x7fb9ec09e3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:07:22.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 0x7fb9ec09dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:22.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ead9d640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 0x7fb9ec13aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:22.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ea59c640 10 monclient(hunting): get_auth_request con 0x7fb9ec0a0ec0 auth_method 0 2024-09-16T05:07:22.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ea59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ead9d640 10 monclient(hunting): get_auth_request con 0x7fb9ec138630 auth_method 0 2024-09-16T05:07:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ead9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9e9d9b640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 0x7fb9ec09e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9e9d9b640 10 monclient(hunting): get_auth_request con 0x7fb9ec09e000 auth_method 0 2024-09-16T05:07:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ea59c640 1 -- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 msgr2=0x7fb9ec09dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:07:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9e9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ea59c640 1 -- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 msgr2=0x7fb9ec09dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:07:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 0x7fb9ec09dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:07:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ead9d640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 msgr2=0x7fb9ec13aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:07:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ead9d640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 msgr2=0x7fb9ec13aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:07:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ead9d640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 0x7fb9ec13aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:07:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 0x7fb9ec09dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:07:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9ead9d640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 0x7fb9ec13aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:07:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9e9d9b640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 msgr2=0x7fb9ec09e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:07:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9e9d9b640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 msgr2=0x7fb9ec09e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:07:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9e9d9b640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 0x7fb9ec09e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:07:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:22.706+0000 7fb9e9d9b640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 0x7fb9ec09e3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:07:23.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 0x7fb9ec09dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:23.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ead9d640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 0x7fb9ec13aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:23.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ea59c640 10 monclient(hunting): get_auth_request con 0x7fb9ec0a0ec0 auth_method 0 2024-09-16T05:07:23.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:23.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:23.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ea59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:23.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ead9d640 10 monclient(hunting): get_auth_request con 0x7fb9ec138630 auth_method 0 2024-09-16T05:07:23.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:23.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:23.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ead9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:23.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9e9d9b640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 0x7fb9ec09e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:23.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9e9d9b640 10 monclient(hunting): get_auth_request con 0x7fb9ec09e000 auth_method 0 2024-09-16T05:07:23.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:23.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:23.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9e9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ea59c640 1 -- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 msgr2=0x7fb9ec09dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:07:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ea59c640 1 -- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 msgr2=0x7fb9ec09dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:07:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 0x7fb9ec09dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:07:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ead9d640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 msgr2=0x7fb9ec13aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:07:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ead9d640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 msgr2=0x7fb9ec13aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:07:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ead9d640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 0x7fb9ec13aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:07:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9e9d9b640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 msgr2=0x7fb9ec09e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:07:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9e9d9b640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 msgr2=0x7fb9ec09e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:07:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9e9d9b640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 0x7fb9ec09e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:07:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 0x7fb9ec09dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:07:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9ead9d640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 0x7fb9ec13aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:07:23.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:23.510+0000 7fb9e9d9b640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 0x7fb9ec09e3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:07:25.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 10 monclient(hunting): tick 2024-09-16T05:07:25.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:07:25.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 1 monclient(hunting): continuing hunt 2024-09-16T05:07:25.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:07:25.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 1 -- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 msgr2=0x7fb9ec09dac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:07:25.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9ec0a0ec0 0x7fb9ec09dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:25.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 msgr2=0x7fb9ec09e3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:07:25.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.149:3300/0 conn(0x7fb9ec09e000 0x7fb9ec09e3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:25.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 1 -- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 msgr2=0x7fb9ec13aa20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:07:25.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 1 --2- 172.21.15.114:0/696886415 >> v2:172.21.15.114:3300/0 conn(0x7fb9ec138630 0x7fb9ec13aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:25.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 10 monclient: _add_conns ranks=[1,0] 2024-09-16T05:07:25.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb9d80036a0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:25.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb9d8007310 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:25.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 1 -- 172.21.15.114:0/696886415 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9ec09e920 con 0x7fb9d8007310 2024-09-16T05:07:25.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 1 -- 172.21.15.114:0/696886415 --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9d800c090 con 0x7fb9d80036a0 2024-09-16T05:07:25.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:25.098+0000 7fb9e3fff640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:27.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.330+0000 7fb9e959a640 1 -- 172.21.15.114:0/696886415 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 95784754 0 0) 0x7fb9ec09e920 con 0x7fb9d8007310 2024-09-16T05:07:27.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.330+0000 7fb9e959a640 10 monclient(hunting): _negotiate 2024-09-16T05:07:27.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.330+0000 7fb9e959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.330+0000 7fb9e959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.330+0000 7fb9e959a640 10 monclient(hunting): my global_id is 4098 2024-09-16T05:07:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.330+0000 7fb9e959a640 1 -- 172.21.15.114:0/696886415 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9cc003530 con 0x7fb9d8007310 2024-09-16T05:07:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.330+0000 7fb9e959a640 1 -- 172.21.15.114:0/696886415 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 899431336 0 0) 0x7fb9d800c090 con 0x7fb9d80036a0 2024-09-16T05:07:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.330+0000 7fb9e959a640 10 monclient(hunting): _negotiate 2024-09-16T05:07:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.330+0000 7fb9e959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.330+0000 7fb9e959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.330+0000 7fb9e959a640 10 monclient(hunting): my global_id is 4102 2024-09-16T05:07:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.330+0000 7fb9e959a640 1 -- 172.21.15.114:0/696886415 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9ec09e920 con 0x7fb9d80036a0 2024-09-16T05:07:27.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.330+0000 7fb9e959a640 1 -- 172.21.15.114:0/696886415 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3686471068 0 0) 0x7fb9cc003530 con 0x7fb9d8007310 2024-09-16T05:07:27.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 1 -- 172.21.15.114:0/696886415 >> v1:172.21.15.149:6789/0 conn(0x7fb9d80036a0 legacy=0x7fb9d8005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:27.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient: _finish_hunting 0 2024-09-16T05:07:27.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:07:27.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:07:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 1 -- 172.21.15.114:0/696886415 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d800d2d0 con 0x7fb9d8007310 2024-09-16T05:07:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient: _finish_auth 0 2024-09-16T05:07:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient: _check_auth_tickets 2024-09-16T05:07:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 1 -- 172.21.15.114:0/696886415 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7fb9d400d5d0 con 0x7fb9d8007310 2024-09-16T05:07:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9f0f86640 5 monclient: authenticate success, global_id 4098 2024-09-16T05:07:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9f0f86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:07:27.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient: dump: 2024-09-16T05:07:27.341 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:27.341 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:27.341 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.341 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.341 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:27.341 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:27.341 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:27.341 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:27.341 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:27.341 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:27.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:07:27.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:27.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 1 -- 172.21.15.114:0/696886415 >> v1:172.21.15.114:6789/0 conn(0x7fb9d8007310 legacy=0x7fb9d80097d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:27.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ead9d640 1 -- 172.21.15.114:0/696886415 reap_dead start 2024-09-16T05:07:27.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:07:27.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9cc004c80 0x7fb9cc005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:27.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient(hunting): picked mon.b con 0x7fb9cc004c80 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:27.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9cc0037b0 0x7fb9cc003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:27.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient(hunting): picked mon.c con 0x7fb9cc0037b0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:27.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9cc0099c0 0x7fb9cc00bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:27.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient(hunting): picked mon.a con 0x7fb9cc0099c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:27.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:27.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 1 -- 172.21.15.114:0/696886415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc00c3a0 con 0x7fb9cc0099c0 2024-09-16T05:07:27.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:27.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 1 -- 172.21.15.114:0/696886415 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc00c550 con 0x7fb9cc004c80 2024-09-16T05:07:27.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:27.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 1 -- 172.21.15.114:0/696886415 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb9cc00c750 con 0x7fb9cc0037b0 2024-09-16T05:07:27.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:27.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e959a640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:07:27.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9f0f86640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:27.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e9d9b640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9cc0037b0 0x7fb9cc003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:27.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9cc004c80 0x7fb9cc005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:27.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 10 monclient(hunting): get_auth_request con 0x7fb9cc004c80 auth_method 0 2024-09-16T05:07:27.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:27.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:27.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:27.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e9d9b640 10 monclient(hunting): get_auth_request con 0x7fb9cc0037b0 auth_method 0 2024-09-16T05:07:27.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:27.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:27.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:27.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ead9d640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9cc0099c0 0x7fb9cc00bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:27.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ead9d640 10 monclient(hunting): get_auth_request con 0x7fb9cc0099c0 auth_method 0 2024-09-16T05:07:27.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:27.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:27.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ead9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:27.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:27.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:27.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:07:27.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:27.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:27.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9e9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:07:27.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:27.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:27.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:07:27.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-09-16T05:07:27.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 1 -- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9cc0037b0 msgr2=0x7fb9cc003c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:27.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9cc0037b0 0x7fb9cc003c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 1 -- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9cc0099c0 msgr2=0x7fb9cc00bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:27.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9cc0099c0 0x7fb9cc00bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 10 monclient: _finish_hunting 0 2024-09-16T05:07:27.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 1 monclient: found mon.b 2024-09-16T05:07:27.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:27.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:27.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 1 -- 172.21.15.114:0/696886415 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9cc003530 con 0x7fb9cc004c80 2024-09-16T05:07:27.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 10 monclient: _finish_auth 0 2024-09-16T05:07:27.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 10 monclient: _check_auth_tickets 2024-09-16T05:07:27.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.334+0000 7fb9ea59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:27.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9cc004c80 0x7fb9cc005060 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0096d0 tx=0x7fb9dc0281b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=375162e023203270 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:27.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9e959a640 1 -- 172.21.15.114:0/696886415 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc0b7070 con 0x7fb9cc004c80 2024-09-16T05:07:27.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:27.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9e959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:27.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9e959a640 10 monclient: dump: 2024-09-16T05:07:27.347 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:27.348 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:27.348 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.348 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.348 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:27.348 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:27.348 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:27.348 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:27.348 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:27.348 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:27.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9f0f86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:27.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9e959a640 1 -- 172.21.15.114:0/696886415 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9dc0b1be0 con 0x7fb9cc004c80 2024-09-16T05:07:27.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9e959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:27.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9f0f86640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:27.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9f0f86640 4 set_mon_vals no callback set 2024-09-16T05:07:27.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9e959a640 1 -- 172.21.15.114:0/696886415 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc01bbc0 con 0x7fb9cc004c80 2024-09-16T05:07:27.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:27.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9e959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:27.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9e959a640 10 monclient: dump: 2024-09-16T05:07:27.349 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:27.349 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:27.349 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.349 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.349 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:27.349 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:27.349 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:27.349 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:27.349 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:27.349 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:27.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9f0f86640 10 monclient: shutdown 2024-09-16T05:07:27.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9f0f86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:27.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9f0f86640 1 -- 172.21.15.114:0/696886415 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9cc004c80 msgr2=0x7fb9cc005060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:27.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9f0f86640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9cc004c80 0x7fb9cc005060 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0096d0 tx=0x7fb9dc0281b0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9f0f86640 1 -- 172.21.15.114:0/696886415 shutdown_connections 2024-09-16T05:07:27.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9f0f86640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9cc0099c0 0x7fb9cc00bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9f0f86640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9cc0037b0 0x7fb9cc003c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9f0f86640 1 --2- 172.21.15.114:0/696886415 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9cc004c80 0x7fb9cc005060 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9f0f86640 1 -- 172.21.15.114:0/696886415 >> 172.21.15.114:0/696886415 conn(0x7fb9ec0a1fb0 msgr2=0x7fb9ec0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:27.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9f0f86640 1 -- 172.21.15.114:0/696886415 shutdown_connections 2024-09-16T05:07:27.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.338+0000 7fb9f0f86640 1 -- 172.21.15.114:0/696886415 wait complete. 2024-09-16T05:07:27.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 10 monclient: build_initial_monmap 2024-09-16T05:07:27.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 10 monclient: monmap: 2024-09-16T05:07:27.356 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:27.356 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:27.357 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 1 librados: starting msgr at 2024-09-16T05:07:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 1 librados: starting objecter 2024-09-16T05:07:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 1 Processor -- start 2024-09-16T05:07:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 1 -- start start 2024-09-16T05:07:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 1 librados: setting wanted keys 2024-09-16T05:07:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 1 librados: calling monclient init 2024-09-16T05:07:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 10 monclient: init 2024-09-16T05:07:27.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:27.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:07:27.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9cc0037b0 0x7fb9ec138090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:27.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 10 monclient(hunting): picked mon.noname-c con 0x7fb9cc0037b0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:27.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9cc004c80 0x7fb9ec098b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:27.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 10 monclient(hunting): picked mon.noname-a con 0x7fb9cc004c80 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:27.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9cc0099c0 0x7fb9ec08f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:27.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 10 monclient(hunting): picked mon.noname-b con 0x7fb9cc0099c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:27.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:27.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb9ec0a1c70 con 0x7fb9cc004c80 2024-09-16T05:07:27.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:27.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb9ec099a70 con 0x7fb9cc0099c0 2024-09-16T05:07:27.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:27.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb9ec0a1df0 con 0x7fb9cc0037b0 2024-09-16T05:07:27.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:27.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9f0f86640 10 monclient(hunting): authenticate will time out at 931.974390s 2024-09-16T05:07:27.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9e9d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9cc004c80 0x7fb9ec098b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:27.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9e9d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9cc004c80 0x7fb9ec098b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56870/0 (socket says 172.21.15.114:56870) 2024-09-16T05:07:27.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9e9d9b640 1 -- 172.21.15.114:0/675846074 learned_addr learned my addr 172.21.15.114:0/675846074 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:27.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9e9d9b640 10 monclient(hunting): get_auth_request con 0x7fb9cc004c80 auth_method 0 2024-09-16T05:07:27.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:27.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:27.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/675846074 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9cc0037b0 0x7fb9ec138090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 10 monclient(hunting): get_auth_request con 0x7fb9cc0037b0 auth_method 0 2024-09-16T05:07:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9e9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9e9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ead9d640 1 --2- 172.21.15.114:0/675846074 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9cc0099c0 0x7fb9ec08f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9e9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ead9d640 10 monclient(hunting): get_auth_request con 0x7fb9cc0099c0 auth_method 0 2024-09-16T05:07:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:27.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:27.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:27.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:27.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:27.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:27.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:27.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 10 monclient(hunting): handle_auth_done global_id 4100 payload 824 2024-09-16T05:07:27.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 1 -- 172.21.15.114:0/675846074 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9cc0099c0 msgr2=0x7fb9ec08f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:27.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/675846074 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9cc0099c0 0x7fb9ec08f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 1 -- 172.21.15.114:0/675846074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9cc004c80 msgr2=0x7fb9ec098b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:27.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/675846074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9cc004c80 0x7fb9ec098b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 10 monclient: _finish_hunting 0 2024-09-16T05:07:27.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 1 monclient: found mon.noname-c 2024-09-16T05:07:27.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:27.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:27.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9ea59c640 1 -- 172.21.15.114:0/675846074 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9ec09c6a0 con 0x7fb9cc0037b0 2024-09-16T05:07:27.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.342+0000 7fb9e9d9b640 1 --2- 172.21.15.114:0/675846074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9cc004c80 0x7fb9ec098b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:27.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9ea59c640 1 --2- 172.21.15.114:0/675846074 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9cc0037b0 0x7fb9ec138090 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0b9000 tx=0x7fb9cc005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:27.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 10 client.?.objecter ms_handle_connect 0x7fb9cc0037b0 2024-09-16T05:07:27.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:27.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 1 -- 172.21.15.114:0/675846074 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc0ca030 con 0x7fb9cc0037b0 2024-09-16T05:07:27.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:27.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:27.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 10 monclient: dump: 2024-09-16T05:07:27.362 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:27.362 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:27.362 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:27.363 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:07:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:07:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 1 -- 172.21.15.114:0/675846074 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9dc01bba0 con 0x7fb9cc0037b0 2024-09-16T05:07:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 1 -- 172.21.15.114:0/675846074 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc0b7040 con 0x7fb9cc0037b0 2024-09-16T05:07:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 10 monclient: dump: 2024-09-16T05:07:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:27.364 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9f0f86640 5 monclient: authenticate success, global_id 4100 2024-09-16T05:07:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9f0f86640 10 monclient: _renew_subs 2024-09-16T05:07:27.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9f0f86640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:27.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9f0f86640 1 -- 172.21.15.114:0/675846074 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9ec092b20 con 0x7fb9cc0037b0 2024-09-16T05:07:27.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e8d99640 4 set_mon_vals no callback set 2024-09-16T05:07:27.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9f0f86640 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:27.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9f0f86640 10 monclient: _renew_subs 2024-09-16T05:07:27.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9f0f86640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:27.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9f0f86640 1 -- 172.21.15.114:0/675846074 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9ec091290 con 0x7fb9cc0037b0 2024-09-16T05:07:27.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9f0f86640 1 librados: init done 2024-09-16T05:07:27.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 1 -- 172.21.15.114:0/675846074 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb9dc0cd070 con 0x7fb9cc0037b0 2024-09-16T05:07:27.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 1 -- 172.21.15.114:0/675846074 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb9dc0c69b0 con 0x7fb9cc0037b0 2024-09-16T05:07:27.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 10 client.4100.objecter ms_dispatch 0x7fb9ec000ba0 osd_map(1..1 src has 1..1) 2024-09-16T05:07:27.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2024-09-16T05:07:27.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 3 client.4100.objecter handle_osd_map decoding full epoch 1 2024-09-16T05:07:27.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.346+0000 7fb9e2ffd640 20 client.4100.objecter dump_active .. 0 homeless 2024-09-16T05:07:27.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.350+0000 7fb9f0f86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:27.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.350+0000 7fb9f0f86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:27.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.350+0000 7fb9f0f86640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:27.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.350+0000 7fb9f0f86640 1 -- 172.21.15.114:0/675846074 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9ec09e730 con 0x7fb9cc0037b0 2024-09-16T05:07:27.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.350+0000 7fb9e2ffd640 1 -- 172.21.15.114:0/675846074 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fb9dc0d5020 con 0x7fb9cc0037b0 2024-09-16T05:07:27.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.350+0000 7fb9e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:27.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.350+0000 7fb9e2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:27.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.418+0000 7fb9f0f86640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T05:07:27.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.418+0000 7fb9f0f86640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T05:07:27.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.418+0000 7fb9f0f86640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:27.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.418+0000 7fb9f0f86640 1 -- 172.21.15.114:0/675846074 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fb9ec0a4ad0 con 0x7fb9cc0037b0 2024-09-16T05:07:27.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.422+0000 7fb9e2ffd640 1 -- 172.21.15.114:0/675846074 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fb9ec09c6a0 con 0x7fb9cc0037b0 2024-09-16T05:07:27.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.422+0000 7fb9e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T05:07:27.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.422+0000 7fb9e2ffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-16T05:07:27.424 INFO:teuthology.orchestra.run.smithi114.stderr:adjusted tunables profile to default 2024-09-16T05:07:27.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 10 librados: watch_flush enter 2024-09-16T05:07:27.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 10 librados: watch_flush exit 2024-09-16T05:07:27.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 20 client.4100.objecter shutdown clearing up homeless session... 2024-09-16T05:07:27.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 10 client.4100.objecter successfully canceled tick 2024-09-16T05:07:27.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 10 monclient: shutdown 2024-09-16T05:07:27.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:27.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 1 -- 172.21.15.114:0/675846074 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9cc0037b0 msgr2=0x7fb9ec138090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:27.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 1 --2- 172.21.15.114:0/675846074 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9cc0037b0 0x7fb9ec138090 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0b9000 tx=0x7fb9cc005840 comp rx=0 tx=0).stop 2024-09-16T05:07:27.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 1 -- 172.21.15.114:0/675846074 shutdown_connections 2024-09-16T05:07:27.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 1 --2- 172.21.15.114:0/675846074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9cc004c80 0x7fb9ec098b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 1 --2- 172.21.15.114:0/675846074 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9cc0099c0 0x7fb9ec08f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 1 --2- 172.21.15.114:0/675846074 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9cc0037b0 0x7fb9ec138090 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 1 -- 172.21.15.114:0/675846074 >> 172.21.15.114:0/675846074 conn(0x7fb9ec0a1fb0 msgr2=0x7fb9ec06f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:27.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 1 -- 172.21.15.114:0/675846074 shutdown_connections 2024-09-16T05:07:27.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 1 -- 172.21.15.114:0/675846074 wait complete. 2024-09-16T05:07:27.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:27.426+0000 7fb9f0f86640 1 librados: shutdown 2024-09-16T05:07:27.442 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T05:07:27.443 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T05:07:27.443 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:07:27.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T05:07:27.454 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:07:27.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T05:07:27.506 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T05:07:27.506 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T05:07:27.557 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:07:27.558 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T05:07:27.566 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:27.565+0000 7fb43baf8180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T05:07:27.569 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:07:27.569 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T05:07:27.625 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:07:27.625 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T05:07:27.681 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new 6ffdab41-0191-4764-ad62-922ec1091569 0 2024-09-16T05:07:27.843 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 10 monclient: get_monmap_and_config 2024-09-16T05:07:27.843 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 10 monclient: build_initial_monmap 2024-09-16T05:07:27.843 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 10 monclient: monmap: 2024-09-16T05:07:27.843 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 0 2024-09-16T05:07:27.843 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:27.843 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:07:27.843944+0000 2024-09-16T05:07:27.843 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:07:27.843944+0000 2024-09-16T05:07:27.843 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:27.843 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:27.843 INFO:teuthology.orchestra.run.smithi149.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:27.843 INFO:teuthology.orchestra.run.smithi149.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:27.843 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:27.844 INFO:teuthology.orchestra.run.smithi149.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:27.844 INFO:teuthology.orchestra.run.smithi149.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:27.844 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:27.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 1 Processor -- start 2024-09-16T05:07:27.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 1 -- start start 2024-09-16T05:07:27.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 10 monclient: init 2024-09-16T05:07:27.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:27.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-16T05:07:27.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2878137730 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:27.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f28780a5d00 0x7f287813c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:27.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 10 monclient(hunting): picked mon.noname-c con 0x7f28780a5d00 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:27.847 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f287813ca00 0x7f287813edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:27.847 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 10 monclient(hunting): picked mon.noname-b con 0x7f287813ca00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:27.847 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28780a55e0 con 0x7f2878137730 2024-09-16T05:07:27.847 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:27.847 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f287809c450 con 0x7f28780a5d00 2024-09-16T05:07:27.847 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:27.847 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f28780a3710 con 0x7f287813ca00 2024-09-16T05:07:27.847 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:27.847 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f287e277640 10 monclient(hunting): authenticate will time out at 923.910663s 2024-09-16T05:07:27.847 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f28780a5d00 0x7f287813c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:27.847 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f28780a5d00 0x7f287813c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:53144/0 (socket says 172.21.15.149:53144) 2024-09-16T05:07:27.847 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 1 -- 172.21.15.149:0/2759600476 learned_addr learned my addr 172.21.15.149:0/2759600476 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-09-16T05:07:27.847 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 10 monclient(hunting): get_auth_request con 0x7f28780a5d00 auth_method 0 2024-09-16T05:07:27.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:27.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:27.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:27.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:27.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:27.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:27.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2024-09-16T05:07:27.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 1 -- 172.21.15.149:0/2759600476 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f287813ca00 msgr2=0x7f287813edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:27.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 1 --2- 172.21.15.149:0/2759600476 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f287813ca00 0x7f287813edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 1 -- 172.21.15.149:0/2759600476 >> v1:172.21.15.114:6789/0 conn(0x7f2878137730 legacy=0x7f2878137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:27.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 10 monclient: _finish_hunting 0 2024-09-16T05:07:27.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 1 monclient: found mon.noname-c 2024-09-16T05:07:27.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:27.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:07:27.849 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.841+0000 7f28777fe640 1 -- 172.21.15.149:0/2759600476 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f287809be00 con 0x7f28780a5d00 2024-09-16T05:07:27.856 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f28777fe640 1 --2- 172.21.15.149:0/2759600476 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28780a5d00 0x7f287813c4c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f286c009870 tx=0x7f286c01c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77b26199618385e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:27.857 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2876ffd640 1 -- 172.21.15.149:0/2759600476 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286c0b0070 con 0x7f28780a5d00 2024-09-16T05:07:27.857 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2876ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:27.857 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2876ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:27.857 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2876ffd640 10 monclient: dump: 2024-09-16T05:07:27.857 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:27.857 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:27.857 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.857 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.857 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:27.857 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:27.857 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:27.858 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:27.858 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:27.858 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:27.858 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2876ffd640 10 monclient: _finish_auth 0 2024-09-16T05:07:27.858 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2876ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:07:27.858 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2876ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:27.858 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2876ffd640 1 -- 172.21.15.149:0/2759600476 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f286c0ab600 con 0x7f28780a5d00 2024-09-16T05:07:27.858 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2876ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:27.858 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2876ffd640 1 -- 172.21.15.149:0/2759600476 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286c0ab9d0 con 0x7f28780a5d00 2024-09-16T05:07:27.858 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 5 monclient: authenticate success, global_id 4111 2024-09-16T05:07:27.858 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:27.858 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 4 set_mon_vals no callback set 2024-09-16T05:07:27.858 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2876ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:27.859 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2876ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:27.859 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2876ffd640 10 monclient: dump: 2024-09-16T05:07:27.859 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:27.859 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:27.859 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.859 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.859 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:27.859 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:27.859 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:27.859 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:27.859 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:27.859 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:27.859 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 10 monclient: shutdown 2024-09-16T05:07:27.859 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:27.860 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 -- 172.21.15.149:0/2759600476 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28780a5d00 msgr2=0x7f287813c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:27.860 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 --2- 172.21.15.149:0/2759600476 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28780a5d00 0x7f287813c4c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f286c009870 tx=0x7f286c01c3e0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.860 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 -- 172.21.15.149:0/2759600476 shutdown_connections 2024-09-16T05:07:27.860 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 --2- 172.21.15.149:0/2759600476 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f287813ca00 0x7f287813edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.860 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 --2- 172.21.15.149:0/2759600476 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28780a5d00 0x7f287813c4c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.860 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 -- 172.21.15.149:0/2759600476 >> 172.21.15.149:0/2759600476 conn(0x7f28780a9c70 msgr2=0x7f28780aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:27.860 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 -- 172.21.15.149:0/2759600476 shutdown_connections 2024-09-16T05:07:27.860 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 -- 172.21.15.149:0/2759600476 wait complete. 2024-09-16T05:07:27.860 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 10 monclient: build_initial_monmap 2024-09-16T05:07:27.860 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 10 monclient: monmap: 2024-09-16T05:07:27.860 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 0 2024-09-16T05:07:27.860 INFO:teuthology.orchestra.run.smithi149.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:27.860 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 0.000000 2024-09-16T05:07:27.861 INFO:teuthology.orchestra.run.smithi149.stderr:created 0.000000 2024-09-16T05:07:27.861 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:27.861 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:27.861 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:27.861 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:27.861 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:27.861 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:27.861 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 librados: starting msgr at 2024-09-16T05:07:27.861 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 librados: starting objecter 2024-09-16T05:07:27.861 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 Processor -- start 2024-09-16T05:07:27.861 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 -- start start 2024-09-16T05:07:27.861 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 librados: setting wanted keys 2024-09-16T05:07:27.861 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 librados: calling monclient init 2024-09-16T05:07:27.862 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 10 monclient: init 2024-09-16T05:07:27.862 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:27.862 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:07:27.862 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28780a5d00 0x7f28781cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:27.862 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 10 monclient(hunting): picked mon.noname-c con 0x7f28780a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:27.862 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2878137730 0x7f2878136290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:27.862 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 10 monclient(hunting): picked mon.noname-a con 0x7f2878137730 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:27.862 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f287813ca00 0x7f287809fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:27.862 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 10 monclient(hunting): picked mon.noname-b con 0x7f287813ca00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:27.862 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:27.862 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2878141cd0 con 0x7f2878137730 2024-09-16T05:07:27.862 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:27.862 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f2878141b50 con 0x7f287813ca00 2024-09-16T05:07:27.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:27.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2878141e50 con 0x7f28780a5d00 2024-09-16T05:07:27.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:27.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287e277640 10 monclient(hunting): authenticate will time out at 923.915508s 2024-09-16T05:07:27.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287c814640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f287813ca00 0x7f287809fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:27.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287c814640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f287813ca00 0x7f287809fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:53152/0 (socket says 172.21.15.149:53152) 2024-09-16T05:07:27.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287c814640 1 -- 172.21.15.149:0/3142771937 learned_addr learned my addr 172.21.15.149:0/3142771937 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-09-16T05:07:27.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287c814640 10 monclient(hunting): get_auth_request con 0x7f287813ca00 auth_method 0 2024-09-16T05:07:27.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287c814640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:27.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287c814640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:27.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287c814640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:27.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287c814640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:27.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287c814640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:27.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287c814640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:27.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f28777fe640 1 --2- 172.21.15.149:0/3142771937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2878137730 0x7f2878136290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:27.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2877fff640 1 --2- 172.21.15.149:0/3142771937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28780a5d00 0x7f28781cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:27.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f28777fe640 10 monclient(hunting): get_auth_request con 0x7f2878137730 auth_method 0 2024-09-16T05:07:27.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f28777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:27.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f28777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:27.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f28777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:27.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2877fff640 10 monclient(hunting): get_auth_request con 0x7f28780a5d00 auth_method 0 2024-09-16T05:07:27.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2877fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:27.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2877fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:27.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f2877fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:27.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287c814640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-09-16T05:07:27.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287c814640 1 -- 172.21.15.149:0/3142771937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28780a5d00 msgr2=0x7f28781cefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:27.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.845+0000 7f287c814640 1 --2- 172.21.15.149:0/3142771937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28780a5d00 0x7f28781cefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287c814640 1 -- 172.21.15.149:0/3142771937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2878137730 msgr2=0x7f2878136290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:27.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287c814640 1 --2- 172.21.15.149:0/3142771937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2878137730 0x7f2878136290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287c814640 10 monclient: _finish_hunting 0 2024-09-16T05:07:27.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287c814640 1 monclient: found mon.noname-b 2024-09-16T05:07:27.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287c814640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:27.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287c814640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.149:3300/0 2024-09-16T05:07:27.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287c814640 1 -- 172.21.15.149:0/3142771937 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f287811fec0 con 0x7f287813ca00 2024-09-16T05:07:27.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287c814640 1 --2- 172.21.15.149:0/3142771937 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f287813ca00 0x7f287809fa90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f287000b9e0 tx=0x7f287000bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:27.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 10 client.?.objecter ms_handle_connect 0x7f287813ca00 2024-09-16T05:07:27.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:27.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 1 -- 172.21.15.149:0/3142771937 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2870019070 con 0x7f287813ca00 2024-09-16T05:07:27.866 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:27.866 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:27.866 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 10 monclient: dump: 2024-09-16T05:07:27.866 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:27.866 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:27.866 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.866 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.866 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:27.866 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:27.866 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:27.866 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:27.866 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:27.866 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:27.866 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 10 monclient: _finish_auth 0 2024-09-16T05:07:27.867 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:07:27.867 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:27.867 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287e277640 5 monclient: authenticate success, global_id 4114 2024-09-16T05:07:27.867 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287e277640 10 monclient: _renew_subs 2024-09-16T05:07:27.867 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 1 -- 172.21.15.149:0/3142771937 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f287000cb40 con 0x7f287813ca00 2024-09-16T05:07:27.867 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287e277640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:27.867 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287e277640 1 -- 172.21.15.149:0/3142771937 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f287813e810 con 0x7f287813ca00 2024-09-16T05:07:27.867 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:27.867 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 1 -- 172.21.15.149:0/3142771937 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2870014440 con 0x7f287813ca00 2024-09-16T05:07:27.867 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:27.867 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:27.867 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 10 monclient: dump: 2024-09-16T05:07:27.867 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:27.867 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:27.867 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.868 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:27.868 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:27.868 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:27.868 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:27.868 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:27.868 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:27.868 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:27.868 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287e277640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:27.868 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287e277640 10 monclient: _renew_subs 2024-09-16T05:07:27.868 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f28767fc640 4 set_mon_vals no callback set 2024-09-16T05:07:27.868 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287e277640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:27.868 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287e277640 1 -- 172.21.15.149:0/3142771937 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2878120660 con 0x7f287813ca00 2024-09-16T05:07:27.868 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287e277640 1 librados: init done 2024-09-16T05:07:27.868 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 1 -- 172.21.15.149:0/3142771937 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f287001c070 con 0x7f287813ca00 2024-09-16T05:07:27.869 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 1 -- 172.21.15.149:0/3142771937 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f287001ebc0 con 0x7f287813ca00 2024-09-16T05:07:27.869 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 10 client.4114.objecter ms_dispatch 0x7f2878000ba0 osd_map(2..2 src has 1..2) 2024-09-16T05:07:27.869 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2024-09-16T05:07:27.869 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2024-09-16T05:07:27.869 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f2874ff9640 20 client.4114.objecter dump_active .. 0 homeless 2024-09-16T05:07:27.869 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287e277640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:27.869 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287e277640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:27.869 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287e277640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:27.869 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.849+0000 7f287e277640 1 -- 172.21.15.149:0/3142771937 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2878137b10 con 0x7f287813ca00 2024-09-16T05:07:27.870 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.861+0000 7f2874ff9640 1 -- 172.21.15.149:0/3142771937 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f2870023070 con 0x7f287813ca00 2024-09-16T05:07:27.870 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.861+0000 7f2874ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:27.870 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.861+0000 7f2874ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:27.933 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.929+0000 7f287e277640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6ffdab41-0191-4764-ad62-922ec1091569", "id": 0}] 2024-09-16T05:07:27.933 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.929+0000 7f287e277640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6ffdab41-0191-4764-ad62-922ec1091569", "id": 0}] 2024-09-16T05:07:27.933 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.929+0000 7f287e277640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:27.933 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.929+0000 7f287e277640 1 -- 172.21.15.149:0/3142771937 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6ffdab41-0191-4764-ad62-922ec1091569", "id": 0} v 0) -- 0x7f28780a1730 con 0x7f287813ca00 2024-09-16T05:07:27.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f2874ff9640 1 -- 172.21.15.149:0/3142771937 <== mon.1 v2:172.21.15.149:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6ffdab41-0191-4764-ad62-922ec1091569", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f2870014630 con 0x7f287813ca00 2024-09-16T05:07:27.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f2874ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6ffdab41-0191-4764-ad62-922ec1091569", "id": 0}] 2024-09-16T05:07:27.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f2874ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:27.939 INFO:teuthology.orchestra.run.smithi149.stdout:0 2024-09-16T05:07:27.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 10 librados: watch_flush enter 2024-09-16T05:07:27.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 10 librados: watch_flush exit 2024-09-16T05:07:27.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 20 client.4114.objecter shutdown clearing up homeless session... 2024-09-16T05:07:27.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 10 client.4114.objecter successfully canceled tick 2024-09-16T05:07:27.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 10 monclient: shutdown 2024-09-16T05:07:27.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:27.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 1 -- 172.21.15.149:0/3142771937 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f287813ca00 msgr2=0x7f287809fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:27.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 1 --2- 172.21.15.149:0/3142771937 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f287813ca00 0x7f287809fa90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f287000b9e0 tx=0x7f287000bea0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 1 -- 172.21.15.149:0/3142771937 shutdown_connections 2024-09-16T05:07:27.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 1 --2- 172.21.15.149:0/3142771937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2878137730 0x7f2878136290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 1 --2- 172.21.15.149:0/3142771937 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f287813ca00 0x7f287809fa90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 1 --2- 172.21.15.149:0/3142771937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28780a5d00 0x7f28781cefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:27.943 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 1 -- 172.21.15.149:0/3142771937 >> 172.21.15.149:0/3142771937 conn(0x7f28780a9c70 msgr2=0x7f28780a76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:27.943 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 1 -- 172.21.15.149:0/3142771937 shutdown_connections 2024-09-16T05:07:27.943 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 1 -- 172.21.15.149:0/3142771937 wait complete. 2024-09-16T05:07:27.943 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:27.937+0000 7f287e277640 1 librados: shutdown 2024-09-16T05:07:27.956 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new 7de579ee-87a3-4463-87a5-6980b56cd113 1 2024-09-16T05:07:28.069 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:28.065+0000 7fb43baf8180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T05:07:28.074 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 10 monclient: get_monmap_and_config 2024-09-16T05:07:28.075 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 10 monclient: build_initial_monmap 2024-09-16T05:07:28.075 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 10 monclient: monmap: 2024-09-16T05:07:28.075 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 0 2024-09-16T05:07:28.075 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.075 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:07:28.076097+0000 2024-09-16T05:07:28.075 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:07:28.076097+0000 2024-09-16T05:07:28.075 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:28.075 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.075 INFO:teuthology.orchestra.run.smithi149.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:28.075 INFO:teuthology.orchestra.run.smithi149.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:28.075 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:28.075 INFO:teuthology.orchestra.run.smithi149.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:28.075 INFO:teuthology.orchestra.run.smithi149.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:28.076 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.078 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 1 Processor -- start 2024-09-16T05:07:28.078 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 1 -- start start 2024-09-16T05:07:28.078 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 10 monclient: init 2024-09-16T05:07:28.078 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:28.078 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-16T05:07:28.078 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ad409e310 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:28.078 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8ad4131350 0x7f8ad4139040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.078 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 10 monclient(hunting): picked mon.noname-a con 0x7f8ad4131350 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:28.078 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ad4139580 0x7f8ad413b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.078 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 10 monclient(hunting): picked mon.noname-b con 0x7f8ad4139580 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:28.078 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ad40a0630 con 0x7f8ad409e310 2024-09-16T05:07:28.078 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.078 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f8ad4131780 con 0x7f8ad4131350 2024-09-16T05:07:28.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8ad409e6f0 con 0x7f8ad4139580 2024-09-16T05:07:28.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:28.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8adbe4e640 10 monclient(hunting): authenticate will time out at 924.142801s 2024-09-16T05:07:28.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ad93e9640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8ad4131350 0x7f8ad4139040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ad4139580 0x7f8ad413b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ad93e9640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8ad4131350 0x7f8ad4139040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.149:60496/0 (socket says 172.21.15.149:60496) 2024-09-16T05:07:28.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ad4139580 0x7f8ad413b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.149:42668/0 (socket says 172.21.15.149:42668) 2024-09-16T05:07:28.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ad9bea640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8ad409e310 0x7f8ad4130c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.149:39950/0 (socket says 172.21.15.149:39950) 2024-09-16T05:07:28.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ad93e9640 1 -- 172.21.15.149:0/1185014553 learned_addr learned my addr 172.21.15.149:0/1185014553 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-09-16T05:07:28.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ad93e9640 10 monclient(hunting): get_auth_request con 0x7f8ad4131350 auth_method 0 2024-09-16T05:07:28.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ad93e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ad93e9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ad93e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.080 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 10 monclient(hunting): get_auth_request con 0x7f8ad4139580 auth_method 0 2024-09-16T05:07:28.080 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.080 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.080 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.080 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ad93e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:28.080 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ad93e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:28.080 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ad93e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:28.080 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:28.080 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:28.080 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:28.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 10 monclient(hunting): handle_auth_done global_id 4106 payload 274 2024-09-16T05:07:28.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 1 -- 172.21.15.149:0/1185014553 >> v2:172.21.15.114:3300/0 conn(0x7f8ad4131350 msgr2=0x7f8ad4139040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:28.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 1 --2- 172.21.15.149:0/1185014553 >> v2:172.21.15.114:3300/0 conn(0x7f8ad4131350 0x7f8ad4139040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 1 -- 172.21.15.149:0/1185014553 >> v1:172.21.15.114:6789/0 conn(0x7f8ad409e310 legacy=0x7f8ad4130c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 10 monclient: _finish_hunting 0 2024-09-16T05:07:28.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 1 monclient: found mon.noname-b 2024-09-16T05:07:28.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:28.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:28.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ada3eb640 1 -- 172.21.15.149:0/1185014553 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad40a0a70 con 0x7f8ad4139580 2024-09-16T05:07:28.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.073+0000 7f8ad93e9640 1 --2- 172.21.15.149:0/1185014553 >> v2:172.21.15.114:3300/0 conn(0x7f8ad4131350 0x7f8ad4139040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:28.087 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8ada3eb640 1 --2- 172.21.15.149:0/1185014553 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ad4139580 0x7f8ad413b970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4009d90 tx=0x7f8ac401c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73726a6beb89c5e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:28.087 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8ad8be8640 1 -- 172.21.15.149:0/1185014553 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40b2070 con 0x7f8ad4139580 2024-09-16T05:07:28.087 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8ad8be8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.087 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8ad8be8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:28.087 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8ad8be8640 10 monclient: dump: 2024-09-16T05:07:28.087 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.087 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.087 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.087 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.087 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.087 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.087 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.087 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.088 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.088 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.088 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8ad8be8640 10 monclient: _finish_auth 0 2024-09-16T05:07:28.088 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8ad8be8640 10 monclient: _check_auth_tickets 2024-09-16T05:07:28.088 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8ad8be8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:28.088 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8ad8be8640 1 -- 172.21.15.149:0/1185014553 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac40a57c0 con 0x7f8ad4139580 2024-09-16T05:07:28.088 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 5 monclient: authenticate success, global_id 4106 2024-09-16T05:07:28.088 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8ad8be8640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:28.088 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:28.088 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8ad8be8640 1 -- 172.21.15.149:0/1185014553 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40a5b90 con 0x7f8ad4139580 2024-09-16T05:07:28.088 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8ad8be8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.088 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 4 set_mon_vals no callback set 2024-09-16T05:07:28.088 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8ad8be8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:28.088 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8ad8be8640 10 monclient: dump: 2024-09-16T05:07:28.089 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.089 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.089 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.089 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.089 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.089 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.089 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.089 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.089 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.089 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.089 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 10 monclient: shutdown 2024-09-16T05:07:28.089 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:28.089 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 1 -- 172.21.15.149:0/1185014553 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ad4139580 msgr2=0x7f8ad413b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.089 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 1 --2- 172.21.15.149:0/1185014553 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ad4139580 0x7f8ad413b970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4009d90 tx=0x7f8ac401c800 comp rx=0 tx=0).stop 2024-09-16T05:07:28.090 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 1 -- 172.21.15.149:0/1185014553 shutdown_connections 2024-09-16T05:07:28.090 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 1 --2- 172.21.15.149:0/1185014553 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ad4139580 0x7f8ad413b970 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.090 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 1 --2- 172.21.15.149:0/1185014553 >> v2:172.21.15.114:3300/0 conn(0x7f8ad4131350 0x7f8ad4139040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.090 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 1 -- 172.21.15.149:0/1185014553 >> 172.21.15.149:0/1185014553 conn(0x7f8ad40a6860 msgr2=0x7f8ad40a6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:28.090 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 1 -- 172.21.15.149:0/1185014553 shutdown_connections 2024-09-16T05:07:28.090 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 1 -- 172.21.15.149:0/1185014553 wait complete. 2024-09-16T05:07:28.090 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 10 monclient: build_initial_monmap 2024-09-16T05:07:28.090 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 10 monclient: monmap: 2024-09-16T05:07:28.090 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 0 2024-09-16T05:07:28.090 INFO:teuthology.orchestra.run.smithi149.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:28.090 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 0.000000 2024-09-16T05:07:28.090 INFO:teuthology.orchestra.run.smithi149.stderr:created 0.000000 2024-09-16T05:07:28.090 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:28.090 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.091 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:28.091 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:28.091 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:28.091 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.091 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 1 librados: starting msgr at 2024-09-16T05:07:28.091 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 1 librados: starting objecter 2024-09-16T05:07:28.091 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.077+0000 7f8adbe4e640 1 Processor -- start 2024-09-16T05:07:28.091 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 1 -- start start 2024-09-16T05:07:28.091 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 1 librados: setting wanted keys 2024-09-16T05:07:28.091 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 1 librados: calling monclient init 2024-09-16T05:07:28.091 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient: init 2024-09-16T05:07:28.093 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:28.093 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:07:28.093 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ad409e310 0x7f8ad4099480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.093 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient(hunting): picked mon.noname-a con 0x7f8ad409e310 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:28.093 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ad4131350 0x7f8ad40999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.094 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient(hunting): picked mon.noname-c con 0x7f8ad4131350 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:28.094 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8ad4139580 0x7f8ad412b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.094 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient(hunting): picked mon.noname-b con 0x7f8ad4139580 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:28.094 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.094 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad4076160 con 0x7f8ad409e310 2024-09-16T05:07:28.094 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.094 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad4075fe0 con 0x7f8ad4139580 2024-09-16T05:07:28.095 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.095 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8ad40762e0 con 0x7f8ad4131350 2024-09-16T05:07:28.095 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:28.095 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient(hunting): authenticate will time out at 924.148455s 2024-09-16T05:07:28.095 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8ad4139580 0x7f8ad412b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.095 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8ad4139580 0x7f8ad412b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:53162/0 (socket says 172.21.15.149:53162) 2024-09-16T05:07:28.095 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 1 -- 172.21.15.149:0/3321929078 learned_addr learned my addr 172.21.15.149:0/3321929078 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-09-16T05:07:28.095 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 10 monclient(hunting): get_auth_request con 0x7f8ad4139580 auth_method 0 2024-09-16T05:07:28.095 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.095 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.095 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.095 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:28.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:28.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:28.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad9bea640 1 --2- 172.21.15.149:0/3321929078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ad409e310 0x7f8ad4099480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad93e9640 1 --2- 172.21.15.149:0/3321929078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ad4131350 0x7f8ad40999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad93e9640 10 monclient(hunting): get_auth_request con 0x7f8ad4131350 auth_method 0 2024-09-16T05:07:28.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad93e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad93e9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad93e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad9bea640 10 monclient(hunting): get_auth_request con 0x7f8ad409e310 auth_method 0 2024-09-16T05:07:28.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad9bea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad9bea640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad9bea640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-09-16T05:07:28.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 1 -- 172.21.15.149:0/3321929078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ad4131350 msgr2=0x7f8ad40999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 1 --2- 172.21.15.149:0/3321929078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ad4131350 0x7f8ad40999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 1 -- 172.21.15.149:0/3321929078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ad409e310 msgr2=0x7f8ad4099480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 1 --2- 172.21.15.149:0/3321929078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ad409e310 0x7f8ad4099480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 10 monclient: _finish_hunting 0 2024-09-16T05:07:28.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 1 monclient: found mon.noname-b 2024-09-16T05:07:28.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:28.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 1 -- 172.21.15.149:0/3321929078 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad412dd30 con 0x7f8ad4139580 2024-09-16T05:07:28.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ada3eb640 1 --2- 172.21.15.149:0/3321929078 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8ad4139580 0x7f8ad412b6f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8ac40b0040 tx=0x7f8ac40195a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:28.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 10 client.?.objecter ms_handle_connect 0x7f8ad4139580 2024-09-16T05:07:28.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:28.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 1 -- 172.21.15.149:0/3321929078 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40b2050 con 0x7f8ad4139580 2024-09-16T05:07:28.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:28.099 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 10 monclient: dump: 2024-09-16T05:07:28.099 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.099 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.099 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.099 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.100 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.100 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.100 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.100 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.100 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.100 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 10 monclient: _finish_auth 0 2024-09-16T05:07:28.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:07:28.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:28.101 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 5 monclient: authenticate success, global_id 4117 2024-09-16T05:07:28.101 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 1 -- 172.21.15.149:0/3321929078 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac40be070 con 0x7f8ad4139580 2024-09-16T05:07:28.101 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:28.101 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 1 -- 172.21.15.149:0/3321929078 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40ad890 con 0x7f8ad4139580 2024-09-16T05:07:28.101 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.101 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:28.101 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 10 monclient: dump: 2024-09-16T05:07:28.101 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.101 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.101 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.101 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.101 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.101 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.101 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.102 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.102 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.102 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.102 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad3fff640 4 set_mon_vals no callback set 2024-09-16T05:07:28.102 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient: _renew_subs 2024-09-16T05:07:28.102 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.102 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 1 -- 172.21.15.149:0/3321929078 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad4138eb0 con 0x7f8ad4139580 2024-09-16T05:07:28.102 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:28.102 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient: _renew_subs 2024-09-16T05:07:28.102 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.102 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 1 -- 172.21.15.149:0/3321929078 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad412d7a0 con 0x7f8ad4139580 2024-09-16T05:07:28.102 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 1 librados: init done 2024-09-16T05:07:28.102 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 1 -- 172.21.15.149:0/3321929078 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8ac40ada30 con 0x7f8ad4139580 2024-09-16T05:07:28.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 1 -- 172.21.15.149:0/3321929078 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f8ac40b9450 con 0x7f8ad4139580 2024-09-16T05:07:28.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 10 client.4117.objecter ms_dispatch 0x7f8ad4000ba0 osd_map(3..3 src has 1..3) 2024-09-16T05:07:28.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-09-16T05:07:28.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-09-16T05:07:28.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8ad27fc640 20 client.4117.objecter dump_active .. 0 homeless 2024-09-16T05:07:28.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:28.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:28.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.081+0000 7f8adbe4e640 1 -- 172.21.15.149:0/3321929078 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad4130c40 con 0x7f8ad4139580 2024-09-16T05:07:28.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.085+0000 7f8ad27fc640 1 -- 172.21.15.149:0/3321929078 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f8ac4016540 con 0x7f8ad4139580 2024-09-16T05:07:28.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.085+0000 7f8ad27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:28.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.085+0000 7f8ad27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:28.156 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.153+0000 7f8adbe4e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7de579ee-87a3-4463-87a5-6980b56cd113", "id": 1}] 2024-09-16T05:07:28.156 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.153+0000 7f8adbe4e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7de579ee-87a3-4463-87a5-6980b56cd113", "id": 1}] 2024-09-16T05:07:28.156 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.153+0000 7f8adbe4e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.156 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.153+0000 7f8adbe4e640 1 -- 172.21.15.149:0/3321929078 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7de579ee-87a3-4463-87a5-6980b56cd113", "id": 1} v 0) -- 0x7f8ad4099f00 con 0x7f8ad4139580 2024-09-16T05:07:28.160 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.157+0000 7f8ad27fc640 1 -- 172.21.15.149:0/3321929078 <== mon.1 v2:172.21.15.149:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7de579ee-87a3-4463-87a5-6980b56cd113", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f8ac40c3460 con 0x7f8ad4139580 2024-09-16T05:07:28.161 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.157+0000 7f8ad27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7de579ee-87a3-4463-87a5-6980b56cd113", "id": 1}] 2024-09-16T05:07:28.161 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.157+0000 7f8ad27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:28.161 INFO:teuthology.orchestra.run.smithi149.stdout:1 2024-09-16T05:07:28.162 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 10 librados: watch_flush enter 2024-09-16T05:07:28.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 10 librados: watch_flush exit 2024-09-16T05:07:28.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 20 client.4117.objecter shutdown clearing up homeless session... 2024-09-16T05:07:28.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 10 client.4117.objecter successfully canceled tick 2024-09-16T05:07:28.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 10 monclient: shutdown 2024-09-16T05:07:28.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:28.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 1 -- 172.21.15.149:0/3321929078 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8ad4139580 msgr2=0x7f8ad412b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 1 --2- 172.21.15.149:0/3321929078 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8ad4139580 0x7f8ad412b6f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8ac40b0040 tx=0x7f8ac40195a0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 1 -- 172.21.15.149:0/3321929078 shutdown_connections 2024-09-16T05:07:28.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 1 --2- 172.21.15.149:0/3321929078 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8ad4139580 0x7f8ad412b6f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 1 --2- 172.21.15.149:0/3321929078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ad4131350 0x7f8ad40999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 1 --2- 172.21.15.149:0/3321929078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ad409e310 0x7f8ad4099480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 1 -- 172.21.15.149:0/3321929078 >> 172.21.15.149:0/3321929078 conn(0x7f8ad40a6860 msgr2=0x7f8ad40a4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:28.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 1 -- 172.21.15.149:0/3321929078 shutdown_connections 2024-09-16T05:07:28.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 1 -- 172.21.15.149:0/3321929078 wait complete. 2024-09-16T05:07:28.164 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.161+0000 7f8adbe4e640 1 librados: shutdown 2024-09-16T05:07:28.178 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new 258c8bf4-58c5-4c13-a0e5-ef9931efd698 2 2024-09-16T05:07:28.245 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:28.241+0000 7fb43baf8180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T05:07:28.295 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 10 monclient: get_monmap_and_config 2024-09-16T05:07:28.295 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 10 monclient: build_initial_monmap 2024-09-16T05:07:28.296 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 10 monclient: monmap: 2024-09-16T05:07:28.296 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 0 2024-09-16T05:07:28.296 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.296 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:07:28.296727+0000 2024-09-16T05:07:28.296 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:07:28.296727+0000 2024-09-16T05:07:28.296 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:28.296 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.296 INFO:teuthology.orchestra.run.smithi149.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:28.296 INFO:teuthology.orchestra.run.smithi149.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:28.296 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:28.296 INFO:teuthology.orchestra.run.smithi149.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:28.297 INFO:teuthology.orchestra.run.smithi149.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:28.297 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.301 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 1 Processor -- start 2024-09-16T05:07:28.301 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 1 -- start start 2024-09-16T05:07:28.301 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 10 monclient: init 2024-09-16T05:07:28.301 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:28.301 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-16T05:07:28.301 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b5409fbd0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:28.301 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b54131e30 0x7f7b5413a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.301 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 10 monclient(hunting): picked mon.noname-b con 0x7f7b54131e30 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:28.302 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b5413a740 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:28.302 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b540a3eb0 con 0x7f7b5413a740 2024-09-16T05:07:28.302 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b54075d30 con 0x7f7b5409fbd0 2024-09-16T05:07:28.302 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.302 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7b540a05b0 con 0x7f7b54131e30 2024-09-16T05:07:28.302 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:28.302 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 10 monclient(hunting): authenticate will time out at 924.363319s 2024-09-16T05:07:28.302 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b537fe640 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f7b5409fbd0 0x7f7b5409cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.149:56902/0 (socket says 172.21.15.149:56902) 2024-09-16T05:07:28.302 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b537fe640 1 -- 172.21.15.149:0/2742530702 learned_addr learned my addr 172.21.15.149:0/2742530702 (peer_addr_for_me v1:172.21.15.149:0/0) 2024-09-16T05:07:28.302 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 1 -- 172.21.15.149:0/2742530702 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3745621594 0 0) 0x7f7b54075d30 con 0x7f7b5409fbd0 2024-09-16T05:07:28.302 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 10 monclient(hunting): _negotiate 2024-09-16T05:07:28.302 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.302 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.302 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 10 monclient(hunting): my global_id is 4120 2024-09-16T05:07:28.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 1 -- 172.21.15.149:0/2742530702 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b3c003530 con 0x7f7b5409fbd0 2024-09-16T05:07:28.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b52ffd640 1 --2- 172.21.15.149:0/2742530702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b54131e30 0x7f7b5413a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b52ffd640 10 monclient(hunting): get_auth_request con 0x7f7b54131e30 auth_method 0 2024-09-16T05:07:28.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 1 -- 172.21.15.149:0/2742530702 <== mon.1 v1:172.21.15.149:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2044393484 0 0) 0x7f7b3c003530 con 0x7f7b5409fbd0 2024-09-16T05:07:28.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 1 -- 172.21.15.149:0/2742530702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b54131e30 msgr2=0x7f7b5413a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:28.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 1 --2- 172.21.15.149:0/2742530702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b54131e30 0x7f7b5413a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 1 -- 172.21.15.149:0/2742530702 >> v1:172.21.15.114:6789/0 conn(0x7f7b5413a740 legacy=0x7f7b5413cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 10 monclient: _finish_hunting 0 2024-09-16T05:07:28.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 1 monclient: found mon.noname-c-legacy 2024-09-16T05:07:28.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:28.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.149:6789/0 2024-09-16T05:07:28.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 1 -- 172.21.15.149:0/2742530702 --> v1:172.21.15.149:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b5409d340 con 0x7f7b5409fbd0 2024-09-16T05:07:28.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 10 monclient: _finish_auth 0 2024-09-16T05:07:28.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 10 monclient: _check_auth_tickets 2024-09-16T05:07:28.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:28.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 1 -- 172.21.15.149:0/2742530702 <== mon.1 v1:172.21.15.149:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f7b44001680 con 0x7f7b5409fbd0 2024-09-16T05:07:28.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 5 monclient: authenticate success, global_id 4120 2024-09-16T05:07:28.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b5a1fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:28.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.293+0000 7f7b527fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T05:07:28.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient: dump: 2024-09-16T05:07:28.304 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.305 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.305 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.305 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.305 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.305 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.305 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.305 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.305 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.305 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] but i'm connected to v1:172.21.15.149:6789/0, reconnecting 2024-09-16T05:07:28.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:28.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 1 -- 172.21.15.149:0/2742530702 >> v1:172.21.15.149:6789/0 conn(0x7f7b5409fbd0 legacy=0x7f7b5409cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:07:28.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 1 --2- 172.21.15.149:0/2742530702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b54131e30 0x7f7b3c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient(hunting): picked mon.c con 0x7f7b54131e30 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:28.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 1 --2- 172.21.15.149:0/2742530702 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7b3c003ff0 0x7f7b3c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient(hunting): picked mon.b con 0x7f7b3c003ff0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:28.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 1 --2- 172.21.15.149:0/2742530702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b3c004da0 0x7f7b3c005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient(hunting): picked mon.a con 0x7f7b3c004da0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:28.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 1 -- 172.21.15.149:0/2742530702 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3c00bc50 con 0x7f7b3c004da0 2024-09-16T05:07:28.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 1 -- 172.21.15.149:0/2742530702 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3c00be00 con 0x7f7b3c003ff0 2024-09-16T05:07:28.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 1 -- 172.21.15.149:0/2742530702 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7b3c00c000 con 0x7f7b54131e30 2024-09-16T05:07:28.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:28.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:07:28.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b5a1fc640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:28.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 1 --2- 172.21.15.149:0/2742530702 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7b3c003ff0 0x7f7b3c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 10 monclient(hunting): get_auth_request con 0x7f7b3c003ff0 auth_method 0 2024-09-16T05:07:28.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:28.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:28.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:28.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:07:28.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b53fff640 1 --2- 172.21.15.149:0/2742530702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b3c004da0 0x7f7b3c005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b537fe640 1 --2- 172.21.15.149:0/2742530702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b54131e30 0x7f7b3c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b53fff640 10 monclient(hunting): get_auth_request con 0x7f7b3c004da0 auth_method 0 2024-09-16T05:07:28.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b53fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b53fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:28.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b537fe640 10 monclient(hunting): get_auth_request con 0x7f7b54131e30 auth_method 0 2024-09-16T05:07:28.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:28.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 10 monclient(hunting): handle_auth_done global_id 4120 payload 293 2024-09-16T05:07:28.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 1 -- 172.21.15.149:0/2742530702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b54131e30 msgr2=0x7f7b3c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:28.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 1 --2- 172.21.15.149:0/2742530702 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b54131e30 0x7f7b3c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 1 -- 172.21.15.149:0/2742530702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b3c004da0 msgr2=0x7f7b3c005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:28.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 1 --2- 172.21.15.149:0/2742530702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b3c004da0 0x7f7b3c005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:07:28.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 1 monclient: found mon.b 2024-09-16T05:07:28.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:28.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 1 -- 172.21.15.149:0/2742530702 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3c003530 con 0x7f7b3c003ff0 2024-09-16T05:07:28.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 10 monclient: _finish_auth 0 2024-09-16T05:07:28.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:07:28.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:28.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b52ffd640 1 --2- 172.21.15.149:0/2742530702 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7b3c003ff0 0x7f7b3c004810 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800ba10 tx=0x7f7b4801c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be04d8f3686e20fd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:28.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 1 -- 172.21.15.149:0/2742530702 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b480b3070 con 0x7f7b3c003ff0 2024-09-16T05:07:28.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:28.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient: dump: 2024-09-16T05:07:28.309 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.309 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.309 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.309 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.309 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.309 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.310 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.310 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.310 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.310 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 1 -- 172.21.15.149:0/2742530702 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b480a9070 con 0x7f7b3c003ff0 2024-09-16T05:07:28.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b5a1fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:28.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:28.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b5a1fc640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:28.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 1 -- 172.21.15.149:0/2742530702 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4800b610 con 0x7f7b3c003ff0 2024-09-16T05:07:28.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b5a1fc640 4 set_mon_vals no callback set 2024-09-16T05:07:28.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:28.311 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b527fc640 10 monclient: dump: 2024-09-16T05:07:28.311 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.311 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.311 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.311 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.311 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.311 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.311 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.311 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.311 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.311 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.311 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b5a1fc640 10 monclient: shutdown 2024-09-16T05:07:28.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b5a1fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:28.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b5a1fc640 1 -- 172.21.15.149:0/2742530702 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7b3c003ff0 msgr2=0x7f7b3c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b5a1fc640 1 --2- 172.21.15.149:0/2742530702 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7b3c003ff0 0x7f7b3c004810 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800ba10 tx=0x7f7b4801c660 comp rx=0 tx=0).stop 2024-09-16T05:07:28.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b53fff640 1 -- 172.21.15.149:0/2742530702 reap_dead start 2024-09-16T05:07:28.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b5a1fc640 1 -- 172.21.15.149:0/2742530702 shutdown_connections 2024-09-16T05:07:28.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b5a1fc640 1 -- 172.21.15.149:0/2742530702 >> 172.21.15.149:0/2742530702 conn(0x7f7b540a2fc0 msgr2=0x7f7b540a33a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:28.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b5a1fc640 1 -- 172.21.15.149:0/2742530702 shutdown_connections 2024-09-16T05:07:28.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.297+0000 7f7b5a1fc640 1 -- 172.21.15.149:0/2742530702 wait complete. 2024-09-16T05:07:28.317 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient: build_initial_monmap 2024-09-16T05:07:28.317 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient: monmap: 2024-09-16T05:07:28.317 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 0 2024-09-16T05:07:28.317 INFO:teuthology.orchestra.run.smithi149.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:28.317 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 0.000000 2024-09-16T05:07:28.317 INFO:teuthology.orchestra.run.smithi149.stderr:created 0.000000 2024-09-16T05:07:28.317 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:28.317 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.317 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:28.317 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:28.318 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:28.318 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.318 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 librados: starting msgr at 2024-09-16T05:07:28.318 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 librados: starting objecter 2024-09-16T05:07:28.318 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 Processor -- start 2024-09-16T05:07:28.318 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 -- start start 2024-09-16T05:07:28.318 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 librados: setting wanted keys 2024-09-16T05:07:28.318 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 librados: calling monclient init 2024-09-16T05:07:28.318 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient: init 2024-09-16T05:07:28.318 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:28.318 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:07:28.319 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b3c003ff0 0x7f7b5412b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.319 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient(hunting): picked mon.noname-a con 0x7f7b3c003ff0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:28.319 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b5412bcf0 0x7f7b5412c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.319 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient(hunting): picked mon.noname-c con 0x7f7b5412bcf0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:28.319 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7b541cd130 0x7f7b541cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.319 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient(hunting): picked mon.noname-b con 0x7f7b541cd130 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:28.319 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.319 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5409ffb0 con 0x7f7b3c003ff0 2024-09-16T05:07:28.319 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.319 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f7b540a0a00 con 0x7f7b541cd130 2024-09-16T05:07:28.319 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.319 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7b5413ec70 con 0x7f7b5412bcf0 2024-09-16T05:07:28.320 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:28.320 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient(hunting): authenticate will time out at 924.368945s 2024-09-16T05:07:28.320 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7b541cd130 0x7f7b541cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.320 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7b541cd130 0x7f7b541cd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:53184/0 (socket says 172.21.15.149:53184) 2024-09-16T05:07:28.320 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 1 -- 172.21.15.149:0/913203267 learned_addr learned my addr 172.21.15.149:0/913203267 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-09-16T05:07:28.320 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 10 monclient(hunting): get_auth_request con 0x7f7b541cd130 auth_method 0 2024-09-16T05:07:28.320 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.320 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.320 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.321 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b537fe640 1 --2- 172.21.15.149:0/913203267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b3c003ff0 0x7f7b5412b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.321 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:28.321 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:28.321 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b52ffd640 1 --2- 172.21.15.149:0/913203267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b5412bcf0 0x7f7b5412c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.321 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:28.321 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b537fe640 10 monclient(hunting): get_auth_request con 0x7f7b3c003ff0 auth_method 0 2024-09-16T05:07:28.321 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.321 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.321 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.321 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b52ffd640 10 monclient(hunting): get_auth_request con 0x7f7b5412bcf0 auth_method 0 2024-09-16T05:07:28.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:28.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:28.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:28.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-09-16T05:07:28.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 1 -- 172.21.15.149:0/913203267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b5412bcf0 msgr2=0x7f7b5412c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 1 --2- 172.21.15.149:0/913203267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b5412bcf0 0x7f7b5412c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 1 -- 172.21.15.149:0/913203267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b3c003ff0 msgr2=0x7f7b5412b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 1 --2- 172.21.15.149:0/913203267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b3c003ff0 0x7f7b5412b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 10 monclient: _finish_hunting 0 2024-09-16T05:07:28.323 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 1 monclient: found mon.noname-b 2024-09-16T05:07:28.323 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:28.323 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.323 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 1 -- 172.21.15.149:0/913203267 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b5412f280 con 0x7f7b541cd130 2024-09-16T05:07:28.323 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b52ffd640 1 --2- 172.21.15.149:0/913203267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b5412bcf0 0x7f7b5412c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:07:28.323 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b53fff640 1 --2- 172.21.15.149:0/913203267 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7b541cd130 0x7f7b541cd5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c00e780 tx=0x7f7b4c00ec40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:28.323 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 10 client.?.objecter ms_handle_connect 0x7f7b541cd130 2024-09-16T05:07:28.323 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:28.323 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 1 -- 172.21.15.149:0/913203267 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c003980 con 0x7f7b541cd130 2024-09-16T05:07:28.323 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.323 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:28.324 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 10 monclient: dump: 2024-09-16T05:07:28.324 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.324 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.324 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.324 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.324 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.324 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.324 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.324 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.324 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.325 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.325 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 10 monclient: _finish_auth 0 2024-09-16T05:07:28.325 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:07:28.325 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:28.325 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 1 -- 172.21.15.149:0/913203267 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b4c018070 con 0x7f7b541cd130 2024-09-16T05:07:28.325 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 5 monclient: authenticate success, global_id 4123 2024-09-16T05:07:28.325 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient: _renew_subs 2024-09-16T05:07:28.325 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.325 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 -- 172.21.15.149:0/913203267 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b5412e3b0 con 0x7f7b541cd130 2024-09-16T05:07:28.325 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:28.325 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 1 -- 172.21.15.149:0/913203267 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c003d60 con 0x7f7b541cd130 2024-09-16T05:07:28.325 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.325 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:28.326 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 10 monclient: dump: 2024-09-16T05:07:28.326 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.326 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.326 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.326 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.326 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.326 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.326 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.326 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.326 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.326 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.326 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b51ffb640 4 set_mon_vals no callback set 2024-09-16T05:07:28.327 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:28.327 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient: _renew_subs 2024-09-16T05:07:28.327 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.327 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 -- 172.21.15.149:0/913203267 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b5412ecc0 con 0x7f7b541cd130 2024-09-16T05:07:28.327 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 librados: init done 2024-09-16T05:07:28.327 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 1 -- 172.21.15.149:0/913203267 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7b4c0050d0 con 0x7f7b541cd130 2024-09-16T05:07:28.327 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 1 -- 172.21.15.149:0/913203267 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f7b4c00ccb0 con 0x7f7b541cd130 2024-09-16T05:07:28.327 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 10 client.4123.objecter ms_dispatch 0x7f7b54000ba0 osd_map(4..4 src has 1..4) 2024-09-16T05:07:28.327 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-09-16T05:07:28.327 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-09-16T05:07:28.327 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b3bfff640 20 client.4123.objecter dump_active .. 0 homeless 2024-09-16T05:07:28.327 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:28.328 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:28.328 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.328 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.301+0000 7f7b5a1fc640 1 -- 172.21.15.149:0/913203267 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b5412eec0 con 0x7f7b541cd130 2024-09-16T05:07:28.328 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.305+0000 7f7b3bfff640 1 -- 172.21.15.149:0/913203267 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f7b5412eec0 con 0x7f7b541cd130 2024-09-16T05:07:28.328 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.305+0000 7f7b3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:28.328 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.305+0000 7f7b3bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:28.377 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.373+0000 7f7b5a1fc640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "258c8bf4-58c5-4c13-a0e5-ef9931efd698", "id": 2}] 2024-09-16T05:07:28.377 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.373+0000 7f7b5a1fc640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "258c8bf4-58c5-4c13-a0e5-ef9931efd698", "id": 2}] 2024-09-16T05:07:28.377 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.373+0000 7f7b5a1fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.378 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.373+0000 7f7b5a1fc640 1 -- 172.21.15.149:0/913203267 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "osd new", "uuid": "258c8bf4-58c5-4c13-a0e5-ef9931efd698", "id": 2} v 0) -- 0x7f7b5409cc30 con 0x7f7b541cd130 2024-09-16T05:07:28.383 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b3bfff640 1 -- 172.21.15.149:0/913203267 <== mon.1 v2:172.21.15.149:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "258c8bf4-58c5-4c13-a0e5-ef9931efd698", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f7b4c00c790 con 0x7f7b541cd130 2024-09-16T05:07:28.383 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "258c8bf4-58c5-4c13-a0e5-ef9931efd698", "id": 2}] 2024-09-16T05:07:28.383 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b3bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:28.383 INFO:teuthology.orchestra.run.smithi149.stdout:2 2024-09-16T05:07:28.385 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 10 librados: watch_flush enter 2024-09-16T05:07:28.385 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 10 librados: watch_flush exit 2024-09-16T05:07:28.385 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 20 client.4123.objecter shutdown clearing up homeless session... 2024-09-16T05:07:28.385 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 10 client.4123.objecter successfully canceled tick 2024-09-16T05:07:28.385 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 10 monclient: shutdown 2024-09-16T05:07:28.385 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:28.385 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 1 -- 172.21.15.149:0/913203267 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7b541cd130 msgr2=0x7f7b541cd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.386 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 1 --2- 172.21.15.149:0/913203267 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7b541cd130 0x7f7b541cd5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c00e780 tx=0x7f7b4c00ec40 comp rx=0 tx=0).stop 2024-09-16T05:07:28.386 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 1 -- 172.21.15.149:0/913203267 shutdown_connections 2024-09-16T05:07:28.386 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 1 --2- 172.21.15.149:0/913203267 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7b541cd130 0x7f7b541cd5b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.386 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 1 --2- 172.21.15.149:0/913203267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b5412bcf0 0x7f7b5412c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.386 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 1 --2- 172.21.15.149:0/913203267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b3c003ff0 0x7f7b5412b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.386 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 1 -- 172.21.15.149:0/913203267 >> 172.21.15.149:0/913203267 conn(0x7f7b540a2fc0 msgr2=0x7f7b540a0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:28.386 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 1 -- 172.21.15.149:0/913203267 shutdown_connections 2024-09-16T05:07:28.386 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 1 -- 172.21.15.149:0/913203267 wait complete. 2024-09-16T05:07:28.386 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.381+0000 7f7b5a1fc640 1 librados: shutdown 2024-09-16T05:07:28.400 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new acc1e03f-8a6c-4c85-9525-88707597d732 3 2024-09-16T05:07:28.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 10 monclient: get_monmap_and_config 2024-09-16T05:07:28.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 10 monclient: build_initial_monmap 2024-09-16T05:07:28.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 10 monclient: monmap: 2024-09-16T05:07:28.511 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 0 2024-09-16T05:07:28.512 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.512 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:07:28.512458+0000 2024-09-16T05:07:28.512 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:07:28.512458+0000 2024-09-16T05:07:28.512 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:28.512 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.512 INFO:teuthology.orchestra.run.smithi149.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:28.512 INFO:teuthology.orchestra.run.smithi149.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:28.512 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:28.512 INFO:teuthology.orchestra.run.smithi149.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:28.512 INFO:teuthology.orchestra.run.smithi149.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:28.512 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 1 Processor -- start 2024-09-16T05:07:28.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 1 -- start start 2024-09-16T05:07:28.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 10 monclient: init 2024-09-16T05:07:28.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:28.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-16T05:07:28.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7b809d200 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:28.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7b80a3270 0x7fc7b8139490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 10 monclient(hunting): picked mon.noname-b con 0x7fc7b80a3270 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:28.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc7b809dcf0 0x7fc7b813b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 10 monclient(hunting): picked mon.noname-a con 0x7fc7b809dcf0 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:28.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7b80a1c80 con 0x7fc7b809d200 2024-09-16T05:07:28.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fc7b809f6e0 con 0x7fc7b809dcf0 2024-09-16T05:07:28.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc7b80a2fe0 con 0x7fc7b80a3270 2024-09-16T05:07:28.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:28.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bf1bc640 10 monclient(hunting): authenticate will time out at 924.579153s 2024-09-16T05:07:28.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bd759640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc7b809dcf0 0x7fc7b813b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7b7fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7b80a3270 0x7fc7b8139490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7b7fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7b80a3270 0x7fc7b8139490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.149:42702/0 (socket says 172.21.15.149:42702) 2024-09-16T05:07:28.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7b7fff640 1 -- 172.21.15.149:0/2409794113 learned_addr learned my addr 172.21.15.149:0/2409794113 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-09-16T05:07:28.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7b7fff640 10 monclient(hunting): get_auth_request con 0x7fc7b80a3270 auth_method 0 2024-09-16T05:07:28.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bd759640 10 monclient(hunting): get_auth_request con 0x7fc7b809dcf0 auth_method 0 2024-09-16T05:07:28.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bd759640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bd759640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bd759640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:28.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:28.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:28.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bd759640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:28.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bd759640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:28.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.509+0000 7fc7bd759640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:28.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b7fff640 10 monclient(hunting): handle_auth_done global_id 4118 payload 274 2024-09-16T05:07:28.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b7fff640 1 -- 172.21.15.149:0/2409794113 >> v2:172.21.15.114:3300/0 conn(0x7fc7b809dcf0 msgr2=0x7fc7b813b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:28.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b7fff640 1 --2- 172.21.15.149:0/2409794113 >> v2:172.21.15.114:3300/0 conn(0x7fc7b809dcf0 0x7fc7b813b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b7fff640 1 -- 172.21.15.149:0/2409794113 >> v1:172.21.15.114:6789/0 conn(0x7fc7b809d200 legacy=0x7fc7b809d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:07:28.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b7fff640 1 monclient: found mon.noname-b 2024-09-16T05:07:28.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:28.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:28.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b7fff640 1 -- 172.21.15.149:0/2409794113 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b8073690 con 0x7fc7b80a3270 2024-09-16T05:07:28.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7bd759640 1 --2- 172.21.15.149:0/2409794113 >> v2:172.21.15.114:3300/0 conn(0x7fc7b809dcf0 0x7fc7b813b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:28.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b7fff640 1 --2- 172.21.15.149:0/2409794113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7b80a3270 0x7fc7b8139490 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009870 tx=0x7fc7ac01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5841766502ba2a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:28.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b77fe640 1 -- 172.21.15.149:0/2409794113 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0b0070 con 0x7fc7b80a3270 2024-09-16T05:07:28.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:28.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b77fe640 10 monclient: dump: 2024-09-16T05:07:28.523 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.524 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.524 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.524 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.524 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.524 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.524 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.524 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.524 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.524 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.524 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b77fe640 10 monclient: _finish_auth 0 2024-09-16T05:07:28.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b77fe640 10 monclient: _check_auth_tickets 2024-09-16T05:07:28.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:28.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7bf1bc640 5 monclient: authenticate success, global_id 4118 2024-09-16T05:07:28.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b77fe640 1 -- 172.21.15.149:0/2409794113 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7ac0ab600 con 0x7fc7b80a3270 2024-09-16T05:07:28.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7bf1bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:28.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:28.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b77fe640 1 -- 172.21.15.149:0/2409794113 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0ab9d0 con 0x7fc7b80a3270 2024-09-16T05:07:28.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7bf1bc640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:28.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7bf1bc640 4 set_mon_vals no callback set 2024-09-16T05:07:28.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:28.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.513+0000 7fc7b77fe640 10 monclient: dump: 2024-09-16T05:07:28.526 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.526 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.526 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.526 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.526 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.526 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.526 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.526 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.526 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.526 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.526 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient: shutdown 2024-09-16T05:07:28.526 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:28.526 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 -- 172.21.15.149:0/2409794113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7b80a3270 msgr2=0x7fc7b8139490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.526 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 --2- 172.21.15.149:0/2409794113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7b80a3270 0x7fc7b8139490 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009870 tx=0x7fc7ac01c420 comp rx=0 tx=0).stop 2024-09-16T05:07:28.527 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 -- 172.21.15.149:0/2409794113 shutdown_connections 2024-09-16T05:07:28.527 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 --2- 172.21.15.149:0/2409794113 >> v2:172.21.15.114:3300/0 conn(0x7fc7b809dcf0 0x7fc7b813b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.527 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 --2- 172.21.15.149:0/2409794113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7b80a3270 0x7fc7b8139490 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.527 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 -- 172.21.15.149:0/2409794113 >> 172.21.15.149:0/2409794113 conn(0x7fc7b80a6310 msgr2=0x7fc7b80a2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:28.527 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 -- 172.21.15.149:0/2409794113 shutdown_connections 2024-09-16T05:07:28.527 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 -- 172.21.15.149:0/2409794113 wait complete. 2024-09-16T05:07:28.527 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient: build_initial_monmap 2024-09-16T05:07:28.527 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient: monmap: 2024-09-16T05:07:28.527 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 0 2024-09-16T05:07:28.527 INFO:teuthology.orchestra.run.smithi149.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:28.527 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 0.000000 2024-09-16T05:07:28.527 INFO:teuthology.orchestra.run.smithi149.stderr:created 0.000000 2024-09-16T05:07:28.527 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:28.527 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.528 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:28.528 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:28.528 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:28.528 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 librados: starting msgr at 2024-09-16T05:07:28.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 librados: starting objecter 2024-09-16T05:07:28.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 Processor -- start 2024-09-16T05:07:28.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 -- start start 2024-09-16T05:07:28.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 librados: setting wanted keys 2024-09-16T05:07:28.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 librados: calling monclient init 2024-09-16T05:07:28.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient: init 2024-09-16T05:07:28.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:28.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:07:28.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7b809d200 0x7fc7b81ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient(hunting): picked mon.noname-b con 0x7fc7b809d200 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:28.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b809dcf0 0x7fc7b813b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient(hunting): picked mon.noname-a con 0x7fc7b809dcf0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:28.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7b80a3270 0x7fc7b8122da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient(hunting): picked mon.noname-c con 0x7fc7b80a3270 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:28.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b813e8c0 con 0x7fc7b809dcf0 2024-09-16T05:07:28.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b813e740 con 0x7fc7b809d200 2024-09-16T05:07:28.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc7b813ea40 con 0x7fc7b80a3270 2024-09-16T05:07:28.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:28.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient(hunting): authenticate will time out at 924.585864s 2024-09-16T05:07:28.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7b809d200 0x7fc7b81ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7b809d200 0x7fc7b81ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:53190/0 (socket says 172.21.15.149:53190) 2024-09-16T05:07:28.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 1 -- 172.21.15.149:0/3481084073 learned_addr learned my addr 172.21.15.149:0/3481084073 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-09-16T05:07:28.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 10 monclient(hunting): get_auth_request con 0x7fc7b809d200 auth_method 0 2024-09-16T05:07:28.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:28.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:28.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:28.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b7fff640 1 --2- 172.21.15.149:0/3481084073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b809dcf0 0x7fc7b813b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b7fff640 10 monclient(hunting): get_auth_request con 0x7fc7b809dcf0 auth_method 0 2024-09-16T05:07:28.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bd759640 1 --2- 172.21.15.149:0/3481084073 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7b80a3270 0x7fc7b8122da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bd759640 10 monclient(hunting): get_auth_request con 0x7fc7b80a3270 auth_method 0 2024-09-16T05:07:28.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bd759640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bd759640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bd759640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-09-16T05:07:28.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 1 -- 172.21.15.149:0/3481084073 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7b80a3270 msgr2=0x7fc7b8122da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 1 --2- 172.21.15.149:0/3481084073 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7b80a3270 0x7fc7b8122da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 1 -- 172.21.15.149:0/3481084073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b809dcf0 msgr2=0x7fc7b813b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 1 --2- 172.21.15.149:0/3481084073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b809dcf0 0x7fc7b813b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 10 monclient: _finish_hunting 0 2024-09-16T05:07:28.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 1 monclient: found mon.noname-b 2024-09-16T05:07:28.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:28.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 1 -- 172.21.15.149:0/3481084073 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b80a5210 con 0x7fc7b809d200 2024-09-16T05:07:28.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b7fff640 1 --2- 172.21.15.149:0/3481084073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b809dcf0 0x7fc7b813b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:07:28.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bcf58640 1 --2- 172.21.15.149:0/3481084073 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7b809d200 0x7fc7b81ccde0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800c500 tx=0x7fc7a800c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:28.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 10 client.?.objecter ms_handle_connect 0x7fc7b809d200 2024-09-16T05:07:28.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:28.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 1 -- 172.21.15.149:0/3481084073 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8016020 con 0x7fc7b809d200 2024-09-16T05:07:28.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:28.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 10 monclient: dump: 2024-09-16T05:07:28.533 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.533 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.533 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.533 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.533 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.533 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.533 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.533 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.533 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.533 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.533 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 10 monclient: _finish_auth 0 2024-09-16T05:07:28.533 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 10 monclient: _check_auth_tickets 2024-09-16T05:07:28.533 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:28.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 5 monclient: authenticate success, global_id 4126 2024-09-16T05:07:28.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 1 -- 172.21.15.149:0/3481084073 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7a8005220 con 0x7fc7b809d200 2024-09-16T05:07:28.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient: _renew_subs 2024-09-16T05:07:28.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 -- 172.21.15.149:0/3481084073 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b8126770 con 0x7fc7b809d200 2024-09-16T05:07:28.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:28.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 1 -- 172.21.15.149:0/3481084073 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8007030 con 0x7fc7b809d200 2024-09-16T05:07:28.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:28.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 10 monclient: dump: 2024-09-16T05:07:28.534 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.534 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.534 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.534 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.535 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.535 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.535 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.535 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.535 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.535 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.535 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:28.535 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b6ffd640 4 set_mon_vals no callback set 2024-09-16T05:07:28.535 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient: _renew_subs 2024-09-16T05:07:28.535 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.535 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 -- 172.21.15.149:0/3481084073 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7b8126490 con 0x7fc7b809d200 2024-09-16T05:07:28.535 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 librados: init done 2024-09-16T05:07:28.535 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 1 -- 172.21.15.149:0/3481084073 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc7a80067e0 con 0x7fc7b809d200 2024-09-16T05:07:28.535 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 1 -- 172.21.15.149:0/3481084073 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fc7a8012d40 con 0x7fc7b809d200 2024-09-16T05:07:28.536 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 10 client.4126.objecter ms_dispatch 0x7fc7b8000ba0 osd_map(5..5 src has 1..5) 2024-09-16T05:07:28.536 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 3 client.4126.objecter handle_osd_map got epochs [5,5] > 0 2024-09-16T05:07:28.536 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 3 client.4126.objecter handle_osd_map decoding full epoch 5 2024-09-16T05:07:28.536 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7b57fa640 20 client.4126.objecter dump_active .. 0 homeless 2024-09-16T05:07:28.536 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:28.536 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:28.536 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.536 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.517+0000 7fc7bf1bc640 1 -- 172.21.15.149:0/3481084073 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7b809d5e0 con 0x7fc7b809d200 2024-09-16T05:07:28.536 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.521+0000 7fc7b57fa640 1 -- 172.21.15.149:0/3481084073 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fc7a8020020 con 0x7fc7b809d200 2024-09-16T05:07:28.536 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.521+0000 7fc7b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:28.536 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.521+0000 7fc7b57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:28.592 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.589+0000 7fc7bf1bc640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "acc1e03f-8a6c-4c85-9525-88707597d732", "id": 3}] 2024-09-16T05:07:28.593 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.589+0000 7fc7bf1bc640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "acc1e03f-8a6c-4c85-9525-88707597d732", "id": 3}] 2024-09-16T05:07:28.593 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.589+0000 7fc7bf1bc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.593 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.589+0000 7fc7bf1bc640 1 -- 172.21.15.149:0/3481084073 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "osd new", "uuid": "acc1e03f-8a6c-4c85-9525-88707597d732", "id": 3} v 0) -- 0x7fc7b8124a40 con 0x7fc7b809d200 2024-09-16T05:07:28.599 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7b57fa640 1 -- 172.21.15.149:0/3481084073 <== mon.1 v2:172.21.15.149:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "acc1e03f-8a6c-4c85-9525-88707597d732", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fc7b80a5210 con 0x7fc7b809d200 2024-09-16T05:07:28.599 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "acc1e03f-8a6c-4c85-9525-88707597d732", "id": 3}] 2024-09-16T05:07:28.599 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7b57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:28.600 INFO:teuthology.orchestra.run.smithi149.stdout:3 2024-09-16T05:07:28.601 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 10 librados: watch_flush enter 2024-09-16T05:07:28.601 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 10 librados: watch_flush exit 2024-09-16T05:07:28.601 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 20 client.4126.objecter shutdown clearing up homeless session... 2024-09-16T05:07:28.601 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 10 client.4126.objecter successfully canceled tick 2024-09-16T05:07:28.601 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 10 monclient: shutdown 2024-09-16T05:07:28.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:28.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 1 -- 172.21.15.149:0/3481084073 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7b809d200 msgr2=0x7fc7b81ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 1 --2- 172.21.15.149:0/3481084073 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7b809d200 0x7fc7b81ccde0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800c500 tx=0x7fc7a800c9c0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 1 -- 172.21.15.149:0/3481084073 shutdown_connections 2024-09-16T05:07:28.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 1 --2- 172.21.15.149:0/3481084073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b809dcf0 0x7fc7b813b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 1 --2- 172.21.15.149:0/3481084073 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7b80a3270 0x7fc7b8122da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 1 --2- 172.21.15.149:0/3481084073 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7b809d200 0x7fc7b81ccde0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 1 -- 172.21.15.149:0/3481084073 >> 172.21.15.149:0/3481084073 conn(0x7fc7b80a6310 msgr2=0x7fc7b8072890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:28.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 1 -- 172.21.15.149:0/3481084073 shutdown_connections 2024-09-16T05:07:28.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 1 -- 172.21.15.149:0/3481084073 wait complete. 2024-09-16T05:07:28.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.597+0000 7fc7bf1bc640 1 librados: shutdown 2024-09-16T05:07:28.616 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new 92cca4cf-19e9-44a1-9588-7c301a2c2065 4 2024-09-16T05:07:28.731 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 10 monclient: get_monmap_and_config 2024-09-16T05:07:28.731 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 10 monclient: build_initial_monmap 2024-09-16T05:07:28.731 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 10 monclient: monmap: 2024-09-16T05:07:28.731 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 0 2024-09-16T05:07:28.731 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.731 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:07:28.732423+0000 2024-09-16T05:07:28.732 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:07:28.732423+0000 2024-09-16T05:07:28.732 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:28.732 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.732 INFO:teuthology.orchestra.run.smithi149.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:28.732 INFO:teuthology.orchestra.run.smithi149.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:28.732 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:28.732 INFO:teuthology.orchestra.run.smithi149.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:28.732 INFO:teuthology.orchestra.run.smithi149.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:28.732 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.734 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 1 Processor -- start 2024-09-16T05:07:28.734 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 1 -- start start 2024-09-16T05:07:28.734 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 10 monclient: init 2024-09-16T05:07:28.734 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:28.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-16T05:07:28.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b040a3050 0x7f5b0406bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 10 monclient(hunting): picked mon.noname-b con 0x7f5b040a3050 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:28.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f5b0406c200 0x7f5b0406c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 10 monclient(hunting): picked mon.noname-c con 0x7f5b0406c200 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:28.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5b0413b7d0 0x7f5b0413dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 10 monclient(hunting): picked mon.noname-a con 0x7f5b0413b7d0 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:28.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f5b0406cb40 con 0x7f5b0413b7d0 2024-09-16T05:07:28.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f5b0409f1e0 con 0x7f5b0406c200 2024-09-16T05:07:28.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f5b040a8580 con 0x7f5b040a3050 2024-09-16T05:07:28.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:28.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b0b1b8640 10 monclient(hunting): authenticate will time out at 924.799162s 2024-09-16T05:07:28.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f5b0406c200 0x7f5b0406c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f5b0406c200 0x7f5b0406c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:53196/0 (socket says 172.21.15.149:53196) 2024-09-16T05:07:28.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 1 -- 172.21.15.149:0/3237400663 learned_addr learned my addr 172.21.15.149:0/3237400663 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-09-16T05:07:28.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 10 monclient(hunting): get_auth_request con 0x7f5b0406c200 auth_method 0 2024-09-16T05:07:28.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:28.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:28.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:28.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b09755640 1 --2- 172.21.15.149:0/3237400663 >> v2:172.21.15.114:3300/0 conn(0x7f5b0413b7d0 0x7f5b0413dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2024-09-16T05:07:28.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 1 -- 172.21.15.149:0/3237400663 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b040a3050 msgr2=0x7f5b0406bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:28.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b08f54640 1 --2- 172.21.15.149:0/3237400663 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b040a3050 0x7f5b0406bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 1 --2- 172.21.15.149:0/3237400663 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b040a3050 0x7f5b0406bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 1 -- 172.21.15.149:0/3237400663 >> v2:172.21.15.114:3300/0 conn(0x7f5b0413b7d0 msgr2=0x7f5b0413dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:28.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 1 --2- 172.21.15.149:0/3237400663 >> v2:172.21.15.114:3300/0 conn(0x7f5b0413b7d0 0x7f5b0413dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 10 monclient: _finish_hunting 0 2024-09-16T05:07:28.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 1 monclient: found mon.noname-c 2024-09-16T05:07:28.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:28.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b03fff640 1 -- 172.21.15.149:0/3237400663 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b0406d120 con 0x7f5b0406c200 2024-09-16T05:07:28.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b09755640 10 monclient: get_auth_request con 0x7f5b0413b7d0 auth_method 0 2024-09-16T05:07:28.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b09755640 1 --2- 172.21.15.149:0/3237400663 >> v2:172.21.15.114:3300/0 conn(0x7f5b0413b7d0 0x7f5b0413dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:07:28.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b08f54640 10 monclient: get_auth_request con 0x7f5b040a3050 auth_method 0 2024-09-16T05:07:28.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.729+0000 7f5b08f54640 1 --2- 172.21.15.149:0/3237400663 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b040a3050 0x7f5b0406bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:07:28.742 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b03fff640 1 --2- 172.21.15.149:0/3237400663 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b0406c200 0x7f5b0406c600 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5b0406f6e0 tx=0x7f5af401c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c60fdf645fea1a8e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:28.742 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b037fe640 1 -- 172.21.15.149:0/3237400663 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af40b0070 con 0x7f5b0406c200 2024-09-16T05:07:28.742 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.742 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:28.742 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b037fe640 10 monclient: dump: 2024-09-16T05:07:28.742 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.742 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.742 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.742 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.742 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.742 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.742 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.742 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.743 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.743 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.743 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b037fe640 10 monclient: _finish_auth 0 2024-09-16T05:07:28.743 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b037fe640 10 monclient: _check_auth_tickets 2024-09-16T05:07:28.743 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:28.743 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b037fe640 1 -- 172.21.15.149:0/3237400663 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af40ab600 con 0x7f5b0406c200 2024-09-16T05:07:28.743 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b037fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:28.743 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b037fe640 1 -- 172.21.15.149:0/3237400663 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af40ab9d0 con 0x7f5b0406c200 2024-09-16T05:07:28.743 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.743 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:28.743 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b037fe640 10 monclient: dump: 2024-09-16T05:07:28.743 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.743 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.743 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.744 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.744 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.744 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.744 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.744 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.744 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.744 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 5 monclient: authenticate success, global_id 4129 2024-09-16T05:07:28.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:28.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 4 set_mon_vals no callback set 2024-09-16T05:07:28.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 10 monclient: shutdown 2024-09-16T05:07:28.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:28.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 1 -- 172.21.15.149:0/3237400663 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b0406c200 msgr2=0x7f5b0406c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 1 --2- 172.21.15.149:0/3237400663 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b0406c200 0x7f5b0406c600 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5b0406f6e0 tx=0x7f5af401c3e0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 1 -- 172.21.15.149:0/3237400663 shutdown_connections 2024-09-16T05:07:28.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 1 --2- 172.21.15.149:0/3237400663 >> v2:172.21.15.114:3300/0 conn(0x7f5b0413b7d0 0x7f5b0413dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 1 --2- 172.21.15.149:0/3237400663 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b0406c200 0x7f5b0406c600 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 1 --2- 172.21.15.149:0/3237400663 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b040a3050 0x7f5b0406bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 1 -- 172.21.15.149:0/3237400663 >> 172.21.15.149:0/3237400663 conn(0x7f5b040a4140 msgr2=0x7f5b040a4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:28.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 1 -- 172.21.15.149:0/3237400663 shutdown_connections 2024-09-16T05:07:28.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 1 -- 172.21.15.149:0/3237400663 wait complete. 2024-09-16T05:07:28.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 10 monclient: build_initial_monmap 2024-09-16T05:07:28.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 10 monclient: monmap: 2024-09-16T05:07:28.745 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 0 2024-09-16T05:07:28.745 INFO:teuthology.orchestra.run.smithi149.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:28.745 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 0.000000 2024-09-16T05:07:28.745 INFO:teuthology.orchestra.run.smithi149.stderr:created 0.000000 2024-09-16T05:07:28.745 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:28.745 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.746 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:28.746 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:28.746 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:28.746 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.746 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 1 librados: starting msgr at 2024-09-16T05:07:28.746 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 1 librados: starting objecter 2024-09-16T05:07:28.746 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.733+0000 7f5b0b1b8640 1 Processor -- start 2024-09-16T05:07:28.746 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 1 -- start start 2024-09-16T05:07:28.746 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 1 librados: setting wanted keys 2024-09-16T05:07:28.746 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 1 librados: calling monclient init 2024-09-16T05:07:28.746 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient: init 2024-09-16T05:07:28.746 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:28.746 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:07:28.747 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b0406c200 0x7f5b04099da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.747 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient(hunting): picked mon.noname-c con 0x7f5b0406c200 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:28.747 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b040a3050 0x7f5b04072950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.747 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient(hunting): picked mon.noname-b con 0x7f5b040a3050 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:28.747 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b0413b7d0 0x7f5b04072e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.747 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient(hunting): picked mon.noname-a con 0x7f5b0413b7d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:28.747 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.747 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0409b140 con 0x7f5b0413b7d0 2024-09-16T05:07:28.747 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.747 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0409afc0 con 0x7f5b040a3050 2024-09-16T05:07:28.747 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.747 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f5b04140920 con 0x7f5b0406c200 2024-09-16T05:07:28.747 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:28.747 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient(hunting): authenticate will time out at 924.804497s 2024-09-16T05:07:28.748 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b040a3050 0x7f5b04072950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.748 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b040a3050 0x7f5b04072950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:53202/0 (socket says 172.21.15.149:53202) 2024-09-16T05:07:28.748 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 1 -- 172.21.15.149:0/3712721432 learned_addr learned my addr 172.21.15.149:0/3712721432 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-09-16T05:07:28.748 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 10 monclient(hunting): get_auth_request con 0x7f5b040a3050 auth_method 0 2024-09-16T05:07:28.748 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.748 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.748 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.748 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:28.748 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:28.748 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:28.748 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b08f54640 1 --2- 172.21.15.149:0/3712721432 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b0406c200 0x7f5b04099da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.748 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b08f54640 10 monclient(hunting): get_auth_request con 0x7f5b0406c200 auth_method 0 2024-09-16T05:07:28.748 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b08f54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b08f54640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b08f54640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b09755640 1 --2- 172.21.15.149:0/3712721432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b0413b7d0 0x7f5b04072e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b09755640 10 monclient(hunting): get_auth_request con 0x7f5b0413b7d0 auth_method 0 2024-09-16T05:07:28.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b09755640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b09755640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b09755640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-09-16T05:07:28.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 1 -- 172.21.15.149:0/3712721432 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b0406c200 msgr2=0x7f5b04099da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 1 --2- 172.21.15.149:0/3712721432 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b0406c200 0x7f5b04099da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 1 -- 172.21.15.149:0/3712721432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b0413b7d0 msgr2=0x7f5b04072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 1 --2- 172.21.15.149:0/3712721432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b0413b7d0 0x7f5b04072e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 10 monclient: _finish_hunting 0 2024-09-16T05:07:28.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 1 monclient: found mon.noname-b 2024-09-16T05:07:28.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:28.750 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.750 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 1 -- 172.21.15.149:0/3712721432 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b04092ec0 con 0x7f5b040a3050 2024-09-16T05:07:28.750 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b03fff640 1 --2- 172.21.15.149:0/3712721432 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b040a3050 0x7f5b04072950 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5af401c8c0 tx=0x7f5af4019180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:28.750 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 10 client.?.objecter ms_handle_connect 0x7f5b040a3050 2024-09-16T05:07:28.750 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:28.750 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 1 -- 172.21.15.149:0/3712721432 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af40b0040 con 0x7f5b040a3050 2024-09-16T05:07:28.750 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.750 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:28.750 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 10 monclient: dump: 2024-09-16T05:07:28.750 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.750 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.750 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.750 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.750 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.751 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.751 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.751 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.751 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.751 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.751 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 10 monclient: _finish_auth 0 2024-09-16T05:07:28.751 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 10 monclient: _check_auth_tickets 2024-09-16T05:07:28.751 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:28.751 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 1 -- 172.21.15.149:0/3712721432 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af40ab400 con 0x7f5b040a3050 2024-09-16T05:07:28.751 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:28.751 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 1 -- 172.21.15.149:0/3712721432 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af40acd70 con 0x7f5b040a3050 2024-09-16T05:07:28.751 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.751 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:28.751 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 10 monclient: dump: 2024-09-16T05:07:28.751 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.752 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.752 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.752 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.752 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.752 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.752 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.752 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.752 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.752 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.752 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 5 monclient: authenticate success, global_id 4132 2024-09-16T05:07:28.752 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b02ffd640 4 set_mon_vals no callback set 2024-09-16T05:07:28.752 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient: _renew_subs 2024-09-16T05:07:28.752 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.752 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 1 -- 172.21.15.149:0/3712721432 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b04093480 con 0x7f5b040a3050 2024-09-16T05:07:28.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:28.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient: _renew_subs 2024-09-16T05:07:28.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 1 -- 172.21.15.149:0/3712721432 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b04092640 con 0x7f5b040a3050 2024-09-16T05:07:28.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 1 librados: init done 2024-09-16T05:07:28.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 1 -- 172.21.15.149:0/3712721432 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5af40bb020 con 0x7f5b040a3050 2024-09-16T05:07:28.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 1 -- 172.21.15.149:0/3712721432 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f5af4006660 con 0x7f5b040a3050 2024-09-16T05:07:28.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 10 client.4132.objecter ms_dispatch 0x7f5b04000ba0 osd_map(6..6 src has 1..6) 2024-09-16T05:07:28.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 3 client.4132.objecter handle_osd_map got epochs [6,6] > 0 2024-09-16T05:07:28.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 3 client.4132.objecter handle_osd_map decoding full epoch 6 2024-09-16T05:07:28.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b017fa640 20 client.4132.objecter dump_active .. 0 homeless 2024-09-16T05:07:28.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:28.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:28.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.754 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.737+0000 7f5b0b1b8640 1 -- 172.21.15.149:0/3712721432 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b040a5a90 con 0x7f5b040a3050 2024-09-16T05:07:28.754 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.741+0000 7f5b017fa640 1 -- 172.21.15.149:0/3712721432 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f5af40160f0 con 0x7f5b040a3050 2024-09-16T05:07:28.754 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.741+0000 7f5b017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:28.754 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.741+0000 7f5b017fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:28.811 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.809+0000 7f5b0b1b8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "92cca4cf-19e9-44a1-9588-7c301a2c2065", "id": 4}] 2024-09-16T05:07:28.811 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.809+0000 7f5b0b1b8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "92cca4cf-19e9-44a1-9588-7c301a2c2065", "id": 4}] 2024-09-16T05:07:28.812 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.809+0000 7f5b0b1b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.812 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.809+0000 7f5b0b1b8640 1 -- 172.21.15.149:0/3712721432 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "osd new", "uuid": "92cca4cf-19e9-44a1-9588-7c301a2c2065", "id": 4} v 0) -- 0x7f5b0406c600 con 0x7f5b040a3050 2024-09-16T05:07:28.816 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.813+0000 7f5b017fa640 1 -- 172.21.15.149:0/3712721432 <== mon.1 v2:172.21.15.149:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "92cca4cf-19e9-44a1-9588-7c301a2c2065", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f5af4006020 con 0x7f5b040a3050 2024-09-16T05:07:28.817 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.813+0000 7f5b017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "92cca4cf-19e9-44a1-9588-7c301a2c2065", "id": 4}] 2024-09-16T05:07:28.817 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.813+0000 7f5b017fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:28.817 INFO:teuthology.orchestra.run.smithi149.stdout:4 2024-09-16T05:07:28.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 10 librados: watch_flush enter 2024-09-16T05:07:28.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 10 librados: watch_flush exit 2024-09-16T05:07:28.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 20 client.4132.objecter shutdown clearing up homeless session... 2024-09-16T05:07:28.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 10 client.4132.objecter successfully canceled tick 2024-09-16T05:07:28.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 10 monclient: shutdown 2024-09-16T05:07:28.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:28.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 1 -- 172.21.15.149:0/3712721432 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b040a3050 msgr2=0x7f5b04072950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 1 --2- 172.21.15.149:0/3712721432 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b040a3050 0x7f5b04072950 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5af401c8c0 tx=0x7f5af4019180 comp rx=0 tx=0).stop 2024-09-16T05:07:28.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 1 -- 172.21.15.149:0/3712721432 shutdown_connections 2024-09-16T05:07:28.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 1 --2- 172.21.15.149:0/3712721432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b0413b7d0 0x7f5b04072e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.821 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 1 --2- 172.21.15.149:0/3712721432 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b040a3050 0x7f5b04072950 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.821 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 1 --2- 172.21.15.149:0/3712721432 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b0406c200 0x7f5b04099da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.821 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 1 -- 172.21.15.149:0/3712721432 >> 172.21.15.149:0/3712721432 conn(0x7f5b040a4140 msgr2=0x7f5b0413ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:28.821 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 1 -- 172.21.15.149:0/3712721432 shutdown_connections 2024-09-16T05:07:28.821 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 1 -- 172.21.15.149:0/3712721432 wait complete. 2024-09-16T05:07:28.821 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.817+0000 7f5b0b1b8640 1 librados: shutdown 2024-09-16T05:07:28.825 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:28.825+0000 7fb43baf8180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T05:07:28.833 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new ac717f4e-83fc-46ea-9816-7e35ef75a674 5 2024-09-16T05:07:28.882 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:28.881+0000 7fb43baf8180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T05:07:28.936 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:28.933+0000 7fb43baf8180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T05:07:28.947 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 10 monclient: get_monmap_and_config 2024-09-16T05:07:28.947 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 10 monclient: build_initial_monmap 2024-09-16T05:07:28.947 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 10 monclient: monmap: 2024-09-16T05:07:28.947 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 0 2024-09-16T05:07:28.947 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.947 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:07:28.947988+0000 2024-09-16T05:07:28.947 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:07:28.947988+0000 2024-09-16T05:07:28.947 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:28.947 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.947 INFO:teuthology.orchestra.run.smithi149.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:28.948 INFO:teuthology.orchestra.run.smithi149.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:28.948 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:28.948 INFO:teuthology.orchestra.run.smithi149.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:28.948 INFO:teuthology.orchestra.run.smithi149.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:28.948 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.950 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 1 Processor -- start 2024-09-16T05:07:28.950 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 1 -- start start 2024-09-16T05:07:28.950 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 10 monclient: init 2024-09-16T05:07:28.950 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:28.950 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-16T05:07:28.950 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f6d90068250 0x7f6d90131820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.950 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 10 monclient(hunting): picked mon.noname-c con 0x7f6d90068250 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:28.950 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d90131d60 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:28.950 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d901321f0 0x7f6d90135470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.950 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 10 monclient(hunting): picked mon.noname-b con 0x7f6d901321f0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:28.950 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d90132640 con 0x7f6d90131d60 2024-09-16T05:07:28.951 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.951 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f6d900a30a0 con 0x7f6d90068250 2024-09-16T05:07:28.951 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.951 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6d90068630 con 0x7f6d901321f0 2024-09-16T05:07:28.951 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:28.951 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d94acb640 10 monclient(hunting): authenticate will time out at 925.015438s 2024-09-16T05:07:28.951 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8e59c640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f6d90068250 0x7f6d90131820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.951 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8dd9b640 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f6d90131d60 0x7f6d90134d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.149:56912/0 (socket says 172.21.15.149:56912) 2024-09-16T05:07:28.951 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8dd9b640 1 -- 172.21.15.149:0/131744070 learned_addr learned my addr 172.21.15.149:0/131744070 (peer_addr_for_me v1:172.21.15.149:0/0) 2024-09-16T05:07:28.951 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8e59c640 10 monclient(hunting): get_auth_request con 0x7f6d90068250 auth_method 0 2024-09-16T05:07:28.951 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.951 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.951 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.951 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:28.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:28.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:28.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8ed9d640 1 --2- 172.21.15.149:0/131744070 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d901321f0 0x7f6d90135470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8ed9d640 10 monclient(hunting): get_auth_request con 0x7f6d901321f0 auth_method 0 2024-09-16T05:07:28.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8d59a640 1 -- 172.21.15.149:0/131744070 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1213795046 0 0) 0x7f6d90132640 con 0x7f6d90131d60 2024-09-16T05:07:28.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:07:28.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8d59a640 10 monclient(hunting): my global_id is 4138 2024-09-16T05:07:28.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8d59a640 1 -- 172.21.15.149:0/131744070 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d78003530 con 0x7f6d90131d60 2024-09-16T05:07:28.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.945+0000 7f6d8e59c640 10 monclient(hunting): handle_auth_done global_id 4135 payload 274 2024-09-16T05:07:28.953 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8e59c640 1 -- 172.21.15.149:0/131744070 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d901321f0 msgr2=0x7f6d90135470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:28.953 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8e59c640 1 --2- 172.21.15.149:0/131744070 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d901321f0 0x7f6d90135470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.953 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8e59c640 1 -- 172.21.15.149:0/131744070 >> v1:172.21.15.149:6789/0 conn(0x7f6d90131d60 legacy=0x7f6d90134d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.953 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8e59c640 10 monclient: _finish_hunting 0 2024-09-16T05:07:28.953 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8e59c640 1 monclient: found mon.noname-c 2024-09-16T05:07:28.953 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:28.953 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.953 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8e59c640 1 -- 172.21.15.149:0/131744070 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d900a2490 con 0x7f6d90068250 2024-09-16T05:07:28.958 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8e59c640 1 --2- 172.21.15.149:0/131744070 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d90068250 0x7f6d90131820 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6d80009820 tx=0x7f6d8001c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c452272fc9cffc18 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:28.958 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8d59a640 1 -- 172.21.15.149:0/131744070 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d800b0070 con 0x7f6d90068250 2024-09-16T05:07:28.958 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.958 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:28.958 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8d59a640 10 monclient: dump: 2024-09-16T05:07:28.958 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.958 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.959 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.959 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.959 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.959 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.959 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.959 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.959 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.959 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8d59a640 10 monclient: _finish_auth 0 2024-09-16T05:07:28.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8d59a640 10 monclient: _check_auth_tickets 2024-09-16T05:07:28.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:28.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8d59a640 1 -- 172.21.15.149:0/131744070 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d800ab600 con 0x7f6d90068250 2024-09-16T05:07:28.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:28.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d94acb640 5 monclient: authenticate success, global_id 4135 2024-09-16T05:07:28.960 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8d59a640 1 -- 172.21.15.149:0/131744070 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d800ab9d0 con 0x7f6d90068250 2024-09-16T05:07:28.960 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d94acb640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:28.960 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d94acb640 4 set_mon_vals no callback set 2024-09-16T05:07:28.960 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.960 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:28.960 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d8d59a640 10 monclient: dump: 2024-09-16T05:07:28.960 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.960 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.960 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.960 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.960 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.960 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.960 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.960 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.961 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.961 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d94acb640 10 monclient: shutdown 2024-09-16T05:07:28.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d94acb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:28.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d94acb640 1 -- 172.21.15.149:0/131744070 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d90068250 msgr2=0x7f6d90131820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.949+0000 7f6d94acb640 1 --2- 172.21.15.149:0/131744070 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d90068250 0x7f6d90131820 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6d80009820 tx=0x7f6d8001c340 comp rx=0 tx=0).stop 2024-09-16T05:07:28.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 -- 172.21.15.149:0/131744070 shutdown_connections 2024-09-16T05:07:28.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 --2- 172.21.15.149:0/131744070 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d901321f0 0x7f6d90135470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 --2- 172.21.15.149:0/131744070 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d90068250 0x7f6d90131820 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 -- 172.21.15.149:0/131744070 >> 172.21.15.149:0/131744070 conn(0x7f6d900a6310 msgr2=0x7f6d900a2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:28.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 -- 172.21.15.149:0/131744070 shutdown_connections 2024-09-16T05:07:28.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 -- 172.21.15.149:0/131744070 wait complete. 2024-09-16T05:07:28.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient: build_initial_monmap 2024-09-16T05:07:28.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient: monmap: 2024-09-16T05:07:28.962 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 0 2024-09-16T05:07:28.962 INFO:teuthology.orchestra.run.smithi149.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:28.962 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 0.000000 2024-09-16T05:07:28.962 INFO:teuthology.orchestra.run.smithi149.stderr:created 0.000000 2024-09-16T05:07:28.962 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:28.962 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.962 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:28.962 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:28.962 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:28.962 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 librados: starting msgr at 2024-09-16T05:07:28.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 librados: starting objecter 2024-09-16T05:07:28.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 Processor -- start 2024-09-16T05:07:28.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 -- start start 2024-09-16T05:07:28.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 librados: setting wanted keys 2024-09-16T05:07:28.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 librados: calling monclient init 2024-09-16T05:07:28.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient: init 2024-09-16T05:07:28.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:28.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:07:28.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d90068250 0x7f6d9012eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient(hunting): picked mon.noname-b con 0x7f6d90068250 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:28.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d90131d60 0x7f6d90135e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient(hunting): picked mon.noname-c con 0x7f6d90131d60 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:28.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d901321f0 0x7f6d90136390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:28.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient(hunting): picked mon.noname-a con 0x7f6d901321f0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:28.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9013fcb0 con 0x7f6d901321f0 2024-09-16T05:07:28.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9013fb30 con 0x7f6d90068250 2024-09-16T05:07:28.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:28.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6d9013fe30 con 0x7f6d90131d60 2024-09-16T05:07:28.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:28.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient(hunting): authenticate will time out at 925.021646s 2024-09-16T05:07:28.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d90068250 0x7f6d9012eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:28.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d90068250 0x7f6d9012eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:53218/0 (socket says 172.21.15.149:53218) 2024-09-16T05:07:28.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 1 -- 172.21.15.149:0/3514927845 learned_addr learned my addr 172.21.15.149:0/3514927845 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-09-16T05:07:28.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 10 monclient(hunting): get_auth_request con 0x7f6d90068250 auth_method 0 2024-09-16T05:07:28.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:28.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:28.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:28.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:28.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:28.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:28.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 10 monclient(hunting): handle_auth_done global_id 4141 payload 824 2024-09-16T05:07:28.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 1 -- 172.21.15.149:0/3514927845 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d90131d60 msgr2=0x7f6d90135e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 1 --2- 172.21.15.149:0/3514927845 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d90131d60 0x7f6d90135e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 1 -- 172.21.15.149:0/3514927845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d901321f0 msgr2=0x7f6d90136390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:28.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 1 --2- 172.21.15.149:0/3514927845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d901321f0 0x7f6d90136390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:28.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 10 monclient: _finish_hunting 0 2024-09-16T05:07:28.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 1 monclient: found mon.noname-b 2024-09-16T05:07:28.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:28.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 1 -- 172.21.15.149:0/3514927845 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d901399b0 con 0x7f6d90068250 2024-09-16T05:07:28.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d8e59c640 1 --2- 172.21.15.149:0/3514927845 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d90068250 0x7f6d9012eab0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6d80009820 tx=0x7f6d800190e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:28.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 10 client.?.objecter ms_handle_connect 0x7f6d90068250 2024-09-16T05:07:28.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:28.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 1 -- 172.21.15.149:0/3514927845 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d800b8020 con 0x7f6d90068250 2024-09-16T05:07:28.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:28.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 10 monclient: dump: 2024-09-16T05:07:28.966 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.966 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.966 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.966 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.966 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.966 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.966 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.966 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.966 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.966 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 10 monclient: _finish_auth 0 2024-09-16T05:07:28.967 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 10 monclient: _check_auth_tickets 2024-09-16T05:07:28.967 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:28.967 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 5 monclient: authenticate success, global_id 4141 2024-09-16T05:07:28.967 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 1 -- 172.21.15.149:0/3514927845 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d800b4610 con 0x7f6d90068250 2024-09-16T05:07:28.967 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:28.967 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 1 -- 172.21.15.149:0/3514927845 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d800b0040 con 0x7f6d90068250 2024-09-16T05:07:28.967 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:28.967 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:28.967 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 10 monclient: dump: 2024-09-16T05:07:28.967 INFO:teuthology.orchestra.run.smithi149.stderr:epoch 1 2024-09-16T05:07:28.967 INFO:teuthology.orchestra.run.smithi149.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:28.967 INFO:teuthology.orchestra.run.smithi149.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.967 INFO:teuthology.orchestra.run.smithi149.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:28.967 INFO:teuthology.orchestra.run.smithi149.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:28.967 INFO:teuthology.orchestra.run.smithi149.stderr:election_strategy: 1 2024-09-16T05:07:28.968 INFO:teuthology.orchestra.run.smithi149.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:28.968 INFO:teuthology.orchestra.run.smithi149.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:28.968 INFO:teuthology.orchestra.run.smithi149.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:28.968 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:07:28.968 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7ffff640 4 set_mon_vals no callback set 2024-09-16T05:07:28.968 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient: _renew_subs 2024-09-16T05:07:28.968 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.968 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 -- 172.21.15.149:0/3514927845 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d90139c90 con 0x7f6d90068250 2024-09-16T05:07:28.968 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:28.968 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient: _renew_subs 2024-09-16T05:07:28.968 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.968 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 -- 172.21.15.149:0/3514927845 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d9006d180 con 0x7f6d90068250 2024-09-16T05:07:28.968 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 librados: init done 2024-09-16T05:07:28.969 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 1 -- 172.21.15.149:0/3514927845 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6d800bb020 con 0x7f6d90068250 2024-09-16T05:07:28.969 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 1 -- 172.21.15.149:0/3514927845 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f6d800bdae0 con 0x7f6d90068250 2024-09-16T05:07:28.969 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 10 client.4141.objecter ms_dispatch 0x7f6d90000ba0 osd_map(7..7 src has 1..7) 2024-09-16T05:07:28.969 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-09-16T05:07:28.969 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-09-16T05:07:28.969 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d7effd640 20 client.4141.objecter dump_active .. 0 homeless 2024-09-16T05:07:28.969 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:28.969 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:28.969 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:28.969 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.953+0000 7f6d94acb640 1 -- 172.21.15.149:0/3514927845 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d90131800 con 0x7f6d90068250 2024-09-16T05:07:28.969 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.957+0000 7f6d7effd640 1 -- 172.21.15.149:0/3514927845 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f6d800c2030 con 0x7f6d90068250 2024-09-16T05:07:28.970 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.957+0000 7f6d7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:28.970 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:28.957+0000 7f6d7effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:29.017 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:29.013+0000 7fb43baf8180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T05:07:29.024 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.021+0000 7f6d94acb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ac717f4e-83fc-46ea-9816-7e35ef75a674", "id": 5}] 2024-09-16T05:07:29.024 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.021+0000 7f6d94acb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ac717f4e-83fc-46ea-9816-7e35ef75a674", "id": 5}] 2024-09-16T05:07:29.024 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.021+0000 7f6d94acb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:29.024 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.021+0000 7f6d94acb640 1 -- 172.21.15.149:0/3514927845 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ac717f4e-83fc-46ea-9816-7e35ef75a674", "id": 5} v 0) -- 0x7f6d90138280 con 0x7f6d90068250 2024-09-16T05:07:29.030 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d7effd640 1 -- 172.21.15.149:0/3514927845 <== mon.1 v2:172.21.15.149:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ac717f4e-83fc-46ea-9816-7e35ef75a674", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f6d80016050 con 0x7f6d90068250 2024-09-16T05:07:29.030 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ac717f4e-83fc-46ea-9816-7e35ef75a674", "id": 5}] 2024-09-16T05:07:29.030 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d7effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:29.030 INFO:teuthology.orchestra.run.smithi149.stdout:5 2024-09-16T05:07:29.031 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 10 librados: watch_flush enter 2024-09-16T05:07:29.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 10 librados: watch_flush exit 2024-09-16T05:07:29.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 20 client.4141.objecter shutdown clearing up homeless session... 2024-09-16T05:07:29.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 10 client.4141.objecter successfully canceled tick 2024-09-16T05:07:29.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 10 monclient: shutdown 2024-09-16T05:07:29.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:29.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 1 -- 172.21.15.149:0/3514927845 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d90068250 msgr2=0x7f6d9012eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:29.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 1 --2- 172.21.15.149:0/3514927845 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d90068250 0x7f6d9012eab0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6d80009820 tx=0x7f6d800190e0 comp rx=0 tx=0).stop 2024-09-16T05:07:29.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 1 -- 172.21.15.149:0/3514927845 shutdown_connections 2024-09-16T05:07:29.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 1 --2- 172.21.15.149:0/3514927845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d901321f0 0x7f6d90136390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:29.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 1 --2- 172.21.15.149:0/3514927845 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d90131d60 0x7f6d90135e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:29.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 1 --2- 172.21.15.149:0/3514927845 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d90068250 0x7f6d9012eab0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:29.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 1 -- 172.21.15.149:0/3514927845 >> 172.21.15.149:0/3514927845 conn(0x7f6d900a6310 msgr2=0x7f6d901349b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:29.033 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 1 -- 172.21.15.149:0/3514927845 shutdown_connections 2024-09-16T05:07:29.033 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 1 -- 172.21.15.149:0/3514927845 wait complete. 2024-09-16T05:07:29.033 INFO:teuthology.orchestra.run.smithi149.stderr:2024-09-16T05:07:29.029+0000 7f6d94acb640 1 librados: shutdown 2024-09-16T05:07:29.046 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T05:07:29.047 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T05:07:29.049 INFO:tasks.ceph.osd.0:Started 2024-09-16T05:07:29.049 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T05:07:29.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T05:07:29.052 INFO:tasks.ceph.osd.1:Started 2024-09-16T05:07:29.052 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T05:07:29.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T05:07:29.054 INFO:tasks.ceph.osd.2:Started 2024-09-16T05:07:29.054 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T05:07:29.055 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T05:07:29.058 INFO:tasks.ceph.osd.3:Started 2024-09-16T05:07:29.058 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T05:07:29.058 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T05:07:29.060 INFO:tasks.ceph.osd.4:Started 2024-09-16T05:07:29.060 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T05:07:29.060 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T05:07:29.062 INFO:tasks.ceph.osd.5:Started 2024-09-16T05:07:29.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T05:07:29.073 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:29.073+0000 7fb43baf8180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T05:07:29.159 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:29.157+0000 7fb43baf8180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T05:07:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 10 monclient: get_monmap_and_config 2024-09-16T05:07:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 10 monclient: build_initial_monmap 2024-09-16T05:07:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 10 monclient: monmap: 2024-09-16T05:07:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:29.185620+0000 2024-09-16T05:07:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:29.185620+0000 2024-09-16T05:07:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:29.183 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:29.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 1 Processor -- start 2024-09-16T05:07:29.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 1 -- start start 2024-09-16T05:07:29.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 10 monclient: init 2024-09-16T05:07:29.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:29.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T05:07:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3640a7d60 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3640a9190 0x7fe3640a8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 10 monclient(hunting): picked mon.noname-b con 0x7fe3640a9190 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe3640a7600 0x7fe364137840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 10 monclient(hunting): picked mon.noname-c con 0x7fe3640a7600 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3640a8a70 con 0x7fe3640a7d60 2024-09-16T05:07:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fe3640a88a0 con 0x7fe3640a7600 2024-09-16T05:07:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe3640a8680 con 0x7fe3640a9190 2024-09-16T05:07:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe36ae06640 10 monclient(hunting): authenticate will time out at 933.814697s 2024-09-16T05:07:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3640a9190 0x7fe3640a8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe368ba2640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe3640a7d60 0x7fe364137130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:45044/0 (socket says 172.21.15.114:45044) 2024-09-16T05:07:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe368ba2640 1 -- 172.21.15.114:0/2393584427 learned_addr learned my addr 172.21.15.114:0/2393584427 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 10 monclient(hunting): get_auth_request con 0x7fe3640a9190 auth_method 0 2024-09-16T05:07:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe3693a3640 1 --2- 172.21.15.114:0/2393584427 >> v2:172.21.15.149:3300/0 conn(0x7fe3640a7600 0x7fe364137840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe3693a3640 10 monclient(hunting): get_auth_request con 0x7fe3640a7600 auth_method 0 2024-09-16T05:07:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe3693a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe3693a3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe3693a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 10 monclient(hunting): handle_auth_done global_id 4142 payload 274 2024-09-16T05:07:29.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe3637fe640 1 -- 172.21.15.114:0/2393584427 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 806665229 0 0) 0x7fe3640a8a70 con 0x7fe3640a7d60 2024-09-16T05:07:29.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 1 -- 172.21.15.114:0/2393584427 >> v2:172.21.15.149:3300/0 conn(0x7fe3640a7600 msgr2=0x7fe364137840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:29.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 1 --2- 172.21.15.114:0/2393584427 >> v2:172.21.15.149:3300/0 conn(0x7fe3640a7600 0x7fe364137840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:29.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 1 -- 172.21.15.114:0/2393584427 >> v1:172.21.15.114:6789/0 conn(0x7fe3640a7d60 legacy=0x7fe364137130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:29.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 10 monclient: _finish_hunting 0 2024-09-16T05:07:29.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 1 monclient: found mon.noname-b 2024-09-16T05:07:29.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:29.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:29.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe363fff640 1 -- 172.21.15.114:0/2393584427 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3640a3710 con 0x7fe3640a9190 2024-09-16T05:07:29.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe3637fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:07:29.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.182+0000 7fe3693a3640 1 --2- 172.21.15.114:0/2393584427 >> v2:172.21.15.149:3300/0 conn(0x7fe3640a7600 0x7fe364137840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:07:29.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe363fff640 1 --2- 172.21.15.114:0/2393584427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3640a9190 0x7fe3640a8140 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe3580097c0 tx=0x7fe35801c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=306bd8ab2f0cd765 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:29.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3637fe640 1 -- 172.21.15.114:0/2393584427 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3580b1070 con 0x7fe3640a9190 2024-09-16T05:07:29.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:29.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:29.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3637fe640 10 monclient: dump: 2024-09-16T05:07:29.194 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:29.194 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:29.194 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:29.194 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:29.194 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:29.194 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:29.195 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3637fe640 10 monclient: _finish_auth 0 2024-09-16T05:07:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3637fe640 10 monclient: _check_auth_tickets 2024-09-16T05:07:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3637fe640 1 -- 172.21.15.114:0/2393584427 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3580ac620 con 0x7fe3640a9190 2024-09-16T05:07:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 5 monclient: authenticate success, global_id 4142 2024-09-16T05:07:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3637fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3637fe640 1 -- 172.21.15.114:0/2393584427 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3580ac9f0 con 0x7fe3640a9190 2024-09-16T05:07:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 4 set_mon_vals no callback set 2024-09-16T05:07:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3637fe640 10 monclient: dump: 2024-09-16T05:07:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:29.197 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 10 monclient: shutdown 2024-09-16T05:07:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 -- 172.21.15.114:0/2393584427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3640a9190 msgr2=0x7fe3640a8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 --2- 172.21.15.114:0/2393584427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3640a9190 0x7fe3640a8140 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe3580097c0 tx=0x7fe35801c3c0 comp rx=0 tx=0).stop 2024-09-16T05:07:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 -- 172.21.15.114:0/2393584427 shutdown_connections 2024-09-16T05:07:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 --2- 172.21.15.114:0/2393584427 >> v2:172.21.15.149:3300/0 conn(0x7fe3640a7600 0x7fe364137840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 --2- 172.21.15.114:0/2393584427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3640a9190 0x7fe3640a8140 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 -- 172.21.15.114:0/2393584427 >> 172.21.15.114:0/2393584427 conn(0x7fe3640a9e00 msgr2=0x7fe3640a6300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 -- 172.21.15.114:0/2393584427 shutdown_connections 2024-09-16T05:07:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 -- 172.21.15.114:0/2393584427 wait complete. 2024-09-16T05:07:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 10 monclient: build_initial_monmap 2024-09-16T05:07:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 10 monclient: monmap: 2024-09-16T05:07:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:29.199 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 librados: starting msgr at 2024-09-16T05:07:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 librados: starting objecter 2024-09-16T05:07:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 Processor -- start 2024-09-16T05:07:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 -- start start 2024-09-16T05:07:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 librados: setting wanted keys 2024-09-16T05:07:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 librados: calling monclient init 2024-09-16T05:07:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 10 monclient: init 2024-09-16T05:07:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:07:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3640a7600 0x7fe3641365d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 10 monclient(hunting): picked mon.noname-c con 0x7fe3640a7600 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe3640a7d60 0x7fe364136b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 10 monclient(hunting): picked mon.noname-b con 0x7fe3640a7d60 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3640a9190 0x7fe36413cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 10 monclient(hunting): picked mon.noname-a con 0x7fe3640a9190 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe364137f00 con 0x7fe3640a9190 2024-09-16T05:07:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fe364137d80 con 0x7fe3640a7d60 2024-09-16T05:07:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe364141c90 con 0x7fe3640a7600 2024-09-16T05:07:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe36ae06640 10 monclient(hunting): authenticate will time out at 933.819784s 2024-09-16T05:07:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3693a3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3640a9190 0x7fe36413cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe368ba2640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3640a7600 0x7fe3641365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3693a3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3640a9190 0x7fe36413cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56910/0 (socket says 172.21.15.114:56910) 2024-09-16T05:07:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3693a3640 1 -- 172.21.15.114:0/1354454574 learned_addr learned my addr 172.21.15.114:0/1354454574 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3693a3640 10 monclient(hunting): get_auth_request con 0x7fe3640a9190 auth_method 0 2024-09-16T05:07:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3693a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3693a3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe3693a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe368ba2640 10 monclient(hunting): get_auth_request con 0x7fe3640a7600 auth_method 0 2024-09-16T05:07:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe368ba2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe368ba2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.186+0000 7fe368ba2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3693a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3693a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3693a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe368ba2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe368ba2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe368ba2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe363fff640 1 --2- 172.21.15.114:0/1354454574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe3640a7d60 0x7fe364136b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe363fff640 10 monclient(hunting): get_auth_request con 0x7fe3640a7d60 auth_method 0 2024-09-16T05:07:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe363fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe363fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe363fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3693a3640 10 monclient(hunting): handle_auth_done global_id 4152 payload 824 2024-09-16T05:07:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3693a3640 1 -- 172.21.15.114:0/1354454574 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3640a7600 msgr2=0x7fe3641365d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3693a3640 1 --2- 172.21.15.114:0/1354454574 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3640a7600 0x7fe3641365d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3693a3640 1 -- 172.21.15.114:0/1354454574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe3640a7d60 msgr2=0x7fe364136b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3693a3640 1 --2- 172.21.15.114:0/1354454574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe3640a7d60 0x7fe364136b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3693a3640 10 monclient: _finish_hunting 0 2024-09-16T05:07:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3693a3640 1 monclient: found mon.noname-a 2024-09-16T05:07:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3693a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3693a3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3693a3640 1 -- 172.21.15.114:0/1354454574 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe364121530 con 0x7fe3640a9190 2024-09-16T05:07:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3693a3640 1 --2- 172.21.15.114:0/1354454574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3640a9190 0x7fe36413cba0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009fa0 tx=0x7fe35c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 10 client.?.objecter ms_handle_connect 0x7fe3640a9190 2024-09-16T05:07:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 1 -- 172.21.15.114:0/1354454574 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c01a070 con 0x7fe3640a9190 2024-09-16T05:07:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 10 monclient: dump: 2024-09-16T05:07:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:29.204 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 10 monclient: _finish_auth 0 2024-09-16T05:07:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 10 monclient: _check_auth_tickets 2024-09-16T05:07:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 1 -- 172.21.15.114:0/1354454574 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe35c003bc0 con 0x7fe3640a9190 2024-09-16T05:07:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 1 -- 172.21.15.114:0/1354454574 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c004030 con 0x7fe3640a9190 2024-09-16T05:07:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 10 monclient: dump: 2024-09-16T05:07:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:29.206 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe362ffd640 4 set_mon_vals no callback set 2024-09-16T05:07:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe36ae06640 5 monclient: authenticate success, global_id 4152 2024-09-16T05:07:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe36ae06640 10 monclient: _renew_subs 2024-09-16T05:07:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe36ae06640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe36ae06640 1 -- 172.21.15.114:0/1354454574 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe36409d350 con 0x7fe3640a9190 2024-09-16T05:07:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe36ae06640 10 client.4152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe36ae06640 10 monclient: _renew_subs 2024-09-16T05:07:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe36ae06640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe36ae06640 1 -- 172.21.15.114:0/1354454574 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe364121250 con 0x7fe3640a9190 2024-09-16T05:07:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe36ae06640 1 librados: init done 2024-09-16T05:07:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 1 -- 172.21.15.114:0/1354454574 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe35c01d070 con 0x7fe3640a9190 2024-09-16T05:07:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 1 -- 172.21.15.114:0/1354454574 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fe35c0047c0 con 0x7fe3640a9190 2024-09-16T05:07:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 10 client.4152.objecter ms_dispatch 0x7fe364000ba0 osd_map(8..8 src has 1..8) 2024-09-16T05:07:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 3 client.4152.objecter handle_osd_map got epochs [8,8] > 0 2024-09-16T05:07:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 3 client.4152.objecter handle_osd_map decoding full epoch 8 2024-09-16T05:07:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe3617fa640 20 client.4152.objecter dump_active .. 0 homeless 2024-09-16T05:07:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe36ae06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe36ae06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe36ae06640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.190+0000 7fe36ae06640 1 -- 172.21.15.114:0/1354454574 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3640a8140 con 0x7fe3640a9190 2024-09-16T05:07:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.206+0000 7fe3617fa640 1 -- 172.21.15.114:0/1354454574 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fe35c024080 con 0x7fe3640a9190 2024-09-16T05:07:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.206+0000 7fe3617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.206+0000 7fe3617fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:29.211 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:29.209+0000 7fb43baf8180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T05:07:29.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.262+0000 7fe36ae06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:29.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.262+0000 7fe36ae06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:29.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.262+0000 7fe36ae06640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:29.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.262+0000 7fe36ae06640 1 -- 172.21.15.114:0/1354454574 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe364120cf0 con 0x7fe3640a9190 2024-09-16T05:07:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe3617fa640 1 -- 172.21.15.114:0/1354454574 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7fe35c010070 con 0x7fe3640a9190 2024-09-16T05:07:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe3617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe3617fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:29.266 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:07:29.266 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":8,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","created":"2024-09-16T05:07:27.305861+0000","modified":"2024-09-16T05:07:29.028596+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T05:07:29.028596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ffdab41-0191-4764-ad62-922ec1091569","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7de579ee-87a3-4463-87a5-6980b56cd113","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"258c8bf4-58c5-4c13-a0e5-ef9931efd698","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"acc1e03f-8a6c-4c85-9525-88707597d732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"92cca4cf-19e9-44a1-9588-7c301a2c2065","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac717f4e-83fc-46ea-9816-7e35ef75a674","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:07:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 10 librados: watch_flush enter 2024-09-16T05:07:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 10 librados: watch_flush exit 2024-09-16T05:07:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 20 client.4152.objecter shutdown clearing up homeless session... 2024-09-16T05:07:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 10 client.4152.objecter successfully canceled tick 2024-09-16T05:07:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 10 monclient: shutdown 2024-09-16T05:07:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 1 -- 172.21.15.114:0/1354454574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3640a9190 msgr2=0x7fe36413cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 1 --2- 172.21.15.114:0/1354454574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3640a9190 0x7fe36413cba0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009fa0 tx=0x7fe35c00d560 comp rx=0 tx=0).stop 2024-09-16T05:07:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 1 -- 172.21.15.114:0/1354454574 shutdown_connections 2024-09-16T05:07:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 1 --2- 172.21.15.114:0/1354454574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3640a9190 0x7fe36413cba0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 1 --2- 172.21.15.114:0/1354454574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe3640a7d60 0x7fe364136b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 1 --2- 172.21.15.114:0/1354454574 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3640a7600 0x7fe3641365d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 1 -- 172.21.15.114:0/1354454574 >> 172.21.15.114:0/1354454574 conn(0x7fe3640a9e00 msgr2=0x7fe3640a1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 1 -- 172.21.15.114:0/1354454574 shutdown_connections 2024-09-16T05:07:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 1 -- 172.21.15.114:0/1354454574 wait complete. 2024-09-16T05:07:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:29.266+0000 7fe36ae06640 1 librados: shutdown 2024-09-16T05:07:29.283 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T05:07:29.283 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T05:07:29.362 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:29.361+0000 7fb43baf8180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T05:07:29.423 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:29.421+0000 7fb43baf8180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T05:07:29.888 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T05:07:30.015 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.014+0000 7fad48bf5640 10 monclient: get_monmap_and_config 2024-09-16T05:07:30.016 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.014+0000 7fad48bf5640 10 monclient: build_initial_monmap 2024-09-16T05:07:30.016 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.014+0000 7fad48bf5640 10 monclient: monmap: 2024-09-16T05:07:30.016 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-09-16T05:07:30.016 INFO:teuthology.misc.health.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:30.016 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-09-16T05:07:30.018817+0000 2024-09-16T05:07:30.016 INFO:teuthology.misc.health.smithi114.stderr:created 2024-09-16T05:07:30.018817+0000 2024-09-16T05:07:30.016 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:30.016 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:30.016 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:30.016 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:30.016 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:30.016 INFO:teuthology.misc.health.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:30.016 INFO:teuthology.misc.health.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:30.017 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:30.018 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.014+0000 7fad48bf5640 1 Processor -- start 2024-09-16T05:07:30.018 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.014+0000 7fad48bf5640 1 -- start start 2024-09-16T05:07:30.018 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.014+0000 7fad48bf5640 10 monclient: init 2024-09-16T05:07:30.018 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad48bf5640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:30.018 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad48bf5640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-16T05:07:30.018 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad48bf5640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fad4409ca70 0x7fad4409ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:30.018 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad48bf5640 10 monclient(hunting): picked mon.noname-a con 0x7fad4409ca70 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:30.018 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad48bf5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad44073e70 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:30.018 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad48bf5640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad4409d390 0x7fad44098710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:30.018 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad48bf5640 10 monclient(hunting): picked mon.noname-b con 0x7fad4409d390 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:30.019 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad48bf5640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad4409e550 con 0x7fad44073e70 2024-09-16T05:07:30.019 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad48bf5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:30.019 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad48bf5640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fad4409a080 con 0x7fad4409ca70 2024-09-16T05:07:30.019 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad48bf5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:30.019 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad48bf5640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fad4409de90 con 0x7fad4409d390 2024-09-16T05:07:30.019 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad48bf5640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:30.019 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad48bf5640 10 monclient(hunting): authenticate will time out at 934.648551s 2024-09-16T05:07:30.019 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad42d9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad4409d390 0x7fad44098710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:30.019 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad42d9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad4409d390 0x7fad44098710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56744/0 (socket says 172.21.15.114:56744) 2024-09-16T05:07:30.019 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad42d9d640 1 -- 172.21.15.114:0/256748034 learned_addr learned my addr 172.21.15.114:0/256748034 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:30.019 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad42d9d640 10 monclient(hunting): get_auth_request con 0x7fad4409d390 auth_method 0 2024-09-16T05:07:30.019 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:30.019 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad42d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:30.019 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad42d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:30.019 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 1 --2- 172.21.15.114:0/256748034 >> v2:172.21.15.114:3300/0 conn(0x7fad4409ca70 0x7fad4409ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:30.020 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 10 monclient(hunting): get_auth_request con 0x7fad4409ca70 auth_method 0 2024-09-16T05:07:30.020 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:30.020 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:30.020 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:30.029 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad42d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:30.029 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad42d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:30.029 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad42d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:30.029 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:30.029 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:30.029 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:30.029 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 10 monclient(hunting): handle_auth_done global_id 4155 payload 274 2024-09-16T05:07:30.029 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 1 -- 172.21.15.114:0/256748034 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad4409d390 msgr2=0x7fad44098710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:30.029 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 1 --2- 172.21.15.114:0/256748034 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad4409d390 0x7fad44098710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:30.029 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 1 -- 172.21.15.114:0/256748034 >> v1:172.21.15.149:6789/0 conn(0x7fad44073e70 legacy=0x7fad44098000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:30.029 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 10 monclient: _finish_hunting 0 2024-09-16T05:07:30.030 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 1 monclient: found mon.noname-a 2024-09-16T05:07:30.030 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:30.030 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:30.030 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad4259c640 1 -- 172.21.15.114:0/256748034 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad4409e2d0 con 0x7fad4409ca70 2024-09-16T05:07:30.030 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.018+0000 7fad42d9d640 1 --2- 172.21.15.114:0/256748034 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad4409d390 0x7fad44098710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:30.030 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 1 --2- 172.21.15.114:0/256748034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad4409ca70 0x7fad4409ce50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fad340097c0 tx=0x7fad3401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a117744d917221d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:30.030 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad41d9b640 1 -- 172.21.15.114:0/256748034 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad340b0070 con 0x7fad4409ca70 2024-09-16T05:07:30.030 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad41d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:30.030 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad41d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:30.030 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad41d9b640 10 monclient: dump: 2024-09-16T05:07:30.031 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-09-16T05:07:30.031 INFO:teuthology.misc.health.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:30.031 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:30.031 INFO:teuthology.misc.health.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:30.031 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:30.031 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:30.031 INFO:teuthology.misc.health.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:30.031 INFO:teuthology.misc.health.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:30.031 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:30.031 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:30.031 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad41d9b640 10 monclient: _finish_auth 0 2024-09-16T05:07:30.031 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad41d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:07:30.031 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad41d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:30.031 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad41d9b640 1 -- 172.21.15.114:0/256748034 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad340ab600 con 0x7fad4409ca70 2024-09-16T05:07:30.031 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 5 monclient: authenticate success, global_id 4155 2024-09-16T05:07:30.032 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:30.032 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad41d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:30.032 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:30.032 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 4 set_mon_vals no callback set 2024-09-16T05:07:30.032 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad41d9b640 1 -- 172.21.15.114:0/256748034 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad340ab9d0 con 0x7fad4409ca70 2024-09-16T05:07:30.032 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad41d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:30.032 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad41d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:30.032 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad41d9b640 10 monclient: dump: 2024-09-16T05:07:30.032 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-09-16T05:07:30.032 INFO:teuthology.misc.health.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:30.032 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:30.032 INFO:teuthology.misc.health.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:30.032 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:30.033 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:30.033 INFO:teuthology.misc.health.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:30.033 INFO:teuthology.misc.health.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:30.033 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:30.033 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:30.033 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient: shutdown 2024-09-16T05:07:30.033 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:30.033 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 -- 172.21.15.114:0/256748034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad4409ca70 msgr2=0x7fad4409ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:30.033 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 --2- 172.21.15.114:0/256748034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad4409ca70 0x7fad4409ce50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fad340097c0 tx=0x7fad3401c360 comp rx=0 tx=0).stop 2024-09-16T05:07:30.033 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 -- 172.21.15.114:0/256748034 shutdown_connections 2024-09-16T05:07:30.033 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 --2- 172.21.15.114:0/256748034 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad4409d390 0x7fad44098710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:30.033 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 --2- 172.21.15.114:0/256748034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad4409ca70 0x7fad4409ce50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:30.033 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 -- 172.21.15.114:0/256748034 >> 172.21.15.114:0/256748034 conn(0x7fad440a2070 msgr2=0x7fad440a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:30.033 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 -- 172.21.15.114:0/256748034 shutdown_connections 2024-09-16T05:07:30.033 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 -- 172.21.15.114:0/256748034 wait complete. 2024-09-16T05:07:30.034 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient: build_initial_monmap 2024-09-16T05:07:30.034 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient: monmap: 2024-09-16T05:07:30.034 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-09-16T05:07:30.034 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:30.034 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:30.034 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-09-16T05:07:30.034 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:30.034 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:30.034 INFO:teuthology.misc.health.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:30.034 INFO:teuthology.misc.health.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:30.035 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:30.035 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:30.035 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 librados: starting msgr at 2024-09-16T05:07:30.035 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 librados: starting objecter 2024-09-16T05:07:30.035 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 Processor -- start 2024-09-16T05:07:30.035 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 -- start start 2024-09-16T05:07:30.036 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 librados: setting wanted keys 2024-09-16T05:07:30.036 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 librados: calling monclient init 2024-09-16T05:07:30.036 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient: init 2024-09-16T05:07:30.036 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:30.036 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:07:30.036 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad44073e70 0x7fad44139c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:30.036 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient(hunting): picked mon.noname-a con 0x7fad44073e70 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:30.036 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad4409ca70 0x7fad44137c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:30.036 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient(hunting): picked mon.noname-c con 0x7fad4409ca70 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:30.036 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fad4409d390 0x7fad441381b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:30.036 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient(hunting): picked mon.noname-b con 0x7fad4409d390 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:30.036 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:30.036 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fad44098dd0 con 0x7fad44073e70 2024-09-16T05:07:30.036 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:30.037 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fad44098c50 con 0x7fad4409d390 2024-09-16T05:07:30.037 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:30.037 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fad4413bb30 con 0x7fad4409ca70 2024-09-16T05:07:30.037 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:30.037 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient(hunting): authenticate will time out at 934.654709s 2024-09-16T05:07:30.037 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad44073e70 0x7fad44139c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:30.037 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad4409ca70 0x7fad44137c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:30.037 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad44073e70 0x7fad44139c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56934/0 (socket says 172.21.15.114:56934) 2024-09-16T05:07:30.037 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 1 -- 172.21.15.114:0/901684182 learned_addr learned my addr 172.21.15.114:0/901684182 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:30.037 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 10 monclient(hunting): get_auth_request con 0x7fad44073e70 auth_method 0 2024-09-16T05:07:30.037 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:30.037 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:30.038 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:30.038 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3bfff640 10 monclient(hunting): get_auth_request con 0x7fad4409ca70 auth_method 0 2024-09-16T05:07:30.038 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:30.038 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:30.038 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:30.038 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:30.038 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:30.038 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:30.038 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:30.038 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:30.038 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:30.038 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad42d9d640 1 --2- 172.21.15.114:0/901684182 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fad4409d390 0x7fad441381b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:30.039 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad42d9d640 10 monclient(hunting): get_auth_request con 0x7fad4409d390 auth_method 0 2024-09-16T05:07:30.039 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:30.039 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad42d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:30.039 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad42d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:30.039 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 10 monclient(hunting): handle_auth_done global_id 4158 payload 824 2024-09-16T05:07:30.039 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 1 -- 172.21.15.114:0/901684182 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad4409ca70 msgr2=0x7fad44137c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:30.039 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 1 --2- 172.21.15.114:0/901684182 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad4409ca70 0x7fad44137c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:30.039 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 1 -- 172.21.15.114:0/901684182 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fad4409d390 msgr2=0x7fad441381b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:30.039 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 1 --2- 172.21.15.114:0/901684182 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fad4409d390 0x7fad441381b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:30.039 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 10 monclient: _finish_hunting 0 2024-09-16T05:07:30.039 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 1 monclient: found mon.noname-a 2024-09-16T05:07:30.040 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:30.040 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:30.040 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 1 -- 172.21.15.114:0/901684182 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad44134450 con 0x7fad44073e70 2024-09-16T05:07:30.040 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3bfff640 1 --2- 172.21.15.114:0/901684182 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad4409ca70 0x7fad44137c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:30.040 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad4259c640 1 --2- 172.21.15.114:0/901684182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad44073e70 0x7fad44139c00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fad340ae040 tx=0x7fad34019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:30.040 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 10 client.?.objecter ms_handle_connect 0x7fad44073e70 2024-09-16T05:07:30.041 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:30.041 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 1 -- 172.21.15.114:0/901684182 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad340b0040 con 0x7fad44073e70 2024-09-16T05:07:30.041 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:30.041 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:30.041 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 10 monclient: dump: 2024-09-16T05:07:30.041 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-09-16T05:07:30.041 INFO:teuthology.misc.health.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:30.041 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:30.041 INFO:teuthology.misc.health.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:30.041 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:30.041 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:30.042 INFO:teuthology.misc.health.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:30.042 INFO:teuthology.misc.health.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:30.042 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:30.042 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:30.042 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 10 monclient: _finish_auth 0 2024-09-16T05:07:30.042 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 10 monclient: _check_auth_tickets 2024-09-16T05:07:30.042 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:30.042 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 5 monclient: authenticate success, global_id 4158 2024-09-16T05:07:30.042 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient: _renew_subs 2024-09-16T05:07:30.042 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 1 -- 172.21.15.114:0/901684182 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad340b43f0 con 0x7fad44073e70 2024-09-16T05:07:30.042 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:30.042 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 -- 172.21.15.114:0/901684182 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad4409c760 con 0x7fad44073e70 2024-09-16T05:07:30.042 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:30.043 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 1 -- 172.21.15.114:0/901684182 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad340b47c0 con 0x7fad44073e70 2024-09-16T05:07:30.043 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:30.043 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:30.043 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 client.4158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:30.043 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad40d99640 4 set_mon_vals no callback set 2024-09-16T05:07:30.043 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 10 monclient: dump: 2024-09-16T05:07:30.043 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-09-16T05:07:30.043 INFO:teuthology.misc.health.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:30.043 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:30.043 INFO:teuthology.misc.health.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:30.043 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:30.043 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:30.043 INFO:teuthology.misc.health.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:30.043 INFO:teuthology.misc.health.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:30.044 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:30.044 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:30.044 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient: _renew_subs 2024-09-16T05:07:30.044 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:30.044 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 -- 172.21.15.114:0/901684182 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad44134120 con 0x7fad44073e70 2024-09-16T05:07:30.044 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad48bf5640 1 librados: init done 2024-09-16T05:07:30.044 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 1 -- 172.21.15.114:0/901684182 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fad340b4960 con 0x7fad44073e70 2024-09-16T05:07:30.044 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 1 -- 172.21.15.114:0/901684182 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fad340aa3a0 con 0x7fad44073e70 2024-09-16T05:07:30.044 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 10 client.4158.objecter ms_dispatch 0x7fad44000ba0 osd_map(8..8 src has 1..8) 2024-09-16T05:07:30.045 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 3 client.4158.objecter handle_osd_map got epochs [8,8] > 0 2024-09-16T05:07:30.045 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 3 client.4158.objecter handle_osd_map decoding full epoch 8 2024-09-16T05:07:30.045 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.022+0000 7fad3affd640 20 client.4158.objecter dump_active .. 0 homeless 2024-09-16T05:07:30.045 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.026+0000 7fad48bf5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:30.045 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.026+0000 7fad48bf5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:30.045 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.026+0000 7fad48bf5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:30.045 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.026+0000 7fad48bf5640 1 -- 172.21.15.114:0/901684182 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad4409ce50 con 0x7fad44073e70 2024-09-16T05:07:30.045 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.026+0000 7fad3affd640 1 -- 172.21.15.114:0/901684182 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fad34016070 con 0x7fad44073e70 2024-09-16T05:07:30.045 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.026+0000 7fad3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:30.045 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.026+0000 7fad3affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:30.073 INFO:tasks.ceph.mgr.x.smithi149.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-16T05:07:30.073 INFO:tasks.ceph.mgr.x.smithi149.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-16T05:07:30.073 INFO:tasks.ceph.mgr.x.smithi149.stderr: from numpy import show_config as show_numpy_config 2024-09-16T05:07:30.085 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:30.086 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:30.086 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:30.086 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 1 -- 172.21.15.114:0/901684182 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad44068940 con 0x7fad44073e70 2024-09-16T05:07:30.086 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad3affd640 1 -- 172.21.15.114:0/901684182 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7fad340b4d90 con 0x7fad44073e70 2024-09-16T05:07:30.086 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:30.086 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad3affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:30.087 INFO:teuthology.misc.health.smithi114.stdout: 2024-09-16T05:07:30.087 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":8,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","created":"2024-09-16T05:07:27.305861+0000","modified":"2024-09-16T05:07:29.028596+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T05:07:29.028596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ffdab41-0191-4764-ad62-922ec1091569","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7de579ee-87a3-4463-87a5-6980b56cd113","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"258c8bf4-58c5-4c13-a0e5-ef9931efd698","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"acc1e03f-8a6c-4c85-9525-88707597d732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"92cca4cf-19e9-44a1-9588-7c301a2c2065","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac717f4e-83fc-46ea-9816-7e35ef75a674","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:07:30.088 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 10 librados: watch_flush enter 2024-09-16T05:07:30.088 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 10 librados: watch_flush exit 2024-09-16T05:07:30.088 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 20 client.4158.objecter shutdown clearing up homeless session... 2024-09-16T05:07:30.088 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 10 client.4158.objecter successfully canceled tick 2024-09-16T05:07:30.088 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 10 monclient: shutdown 2024-09-16T05:07:30.088 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:30.088 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 1 -- 172.21.15.114:0/901684182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad44073e70 msgr2=0x7fad44139c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:30.089 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 1 --2- 172.21.15.114:0/901684182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad44073e70 0x7fad44139c00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fad340ae040 tx=0x7fad34019100 comp rx=0 tx=0).stop 2024-09-16T05:07:30.089 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 1 -- 172.21.15.114:0/901684182 shutdown_connections 2024-09-16T05:07:30.089 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 1 --2- 172.21.15.114:0/901684182 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fad4409d390 0x7fad441381b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:30.089 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 1 --2- 172.21.15.114:0/901684182 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad4409ca70 0x7fad44137c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:30.089 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 1 --2- 172.21.15.114:0/901684182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad44073e70 0x7fad44139c00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:30.089 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 1 -- 172.21.15.114:0/901684182 >> 172.21.15.114:0/901684182 conn(0x7fad440a2070 msgr2=0x7fad4413c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:30.089 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 1 -- 172.21.15.114:0/901684182 shutdown_connections 2024-09-16T05:07:30.089 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 1 -- 172.21.15.114:0/901684182 wait complete. 2024-09-16T05:07:30.089 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:30.086+0000 7fad48bf5640 1 librados: shutdown 2024-09-16T05:07:30.102 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-16T05:07:30.155 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:30.153+0000 7fb43baf8180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T05:07:30.339 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:30.337+0000 7fb43baf8180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T05:07:30.509 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:30.505+0000 7fb43baf8180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T05:07:30.579 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:30.577+0000 7fb43baf8180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T05:07:30.679 INFO:tasks.ceph.osd.4.smithi149.stderr:2024-09-16T05:07:30.677+0000 7f59dfe547c0 -1 Falling back to public interface 2024-09-16T05:07:30.707 INFO:tasks.ceph.osd.3.smithi149.stderr:2024-09-16T05:07:30.705+0000 7f1a42e6e7c0 -1 Falling back to public interface 2024-09-16T05:07:30.707 INFO:tasks.ceph.osd.5.smithi149.stderr:2024-09-16T05:07:30.705+0000 7fdec4e337c0 -1 Falling back to public interface 2024-09-16T05:07:30.719 INFO:tasks.ceph.osd.1.smithi114.stderr:2024-09-16T05:07:30.718+0000 7f3f6850b7c0 -1 Falling back to public interface 2024-09-16T05:07:30.727 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:30.725+0000 7fb43baf8180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T05:07:30.730 INFO:tasks.ceph.osd.2.smithi114.stderr:2024-09-16T05:07:30.730+0000 7fadebf687c0 -1 Falling back to public interface 2024-09-16T05:07:30.926 INFO:tasks.ceph.osd.0.smithi114.stderr:2024-09-16T05:07:30.926+0000 7f4471e627c0 -1 Falling back to public interface 2024-09-16T05:07:31.356 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:31.353+0000 7fb43baf8180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T05:07:31.430 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:31.429+0000 7fb43baf8180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T05:07:31.612 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:31.609+0000 7fb43baf8180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T05:07:31.685 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:31.681+0000 7fb43baf8180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T05:07:31.973 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:31.969+0000 7fb43baf8180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T05:07:32.037 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:32.033+0000 7fb43baf8180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T05:07:33.074 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:33.073+0000 7fb3e2630640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T05:07:33.079 INFO:tasks.ceph.mgr.x.smithi149.stderr:2024-09-16T05:07:33.077+0000 7fb3e2630640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T05:07:34.734 INFO:tasks.ceph.osd.3.smithi149.stderr:2024-09-16T05:07:34.733+0000 7f1a42e6e7c0 -1 osd.3 0 log_to_monitors true 2024-09-16T05:07:35.022 INFO:tasks.ceph.osd.4.smithi149.stderr:2024-09-16T05:07:35.021+0000 7f59dfe547c0 -1 osd.4 0 log_to_monitors true 2024-09-16T05:07:35.438 INFO:tasks.ceph.osd.1.smithi114.stderr:2024-09-16T05:07:35.438+0000 7f3f6850b7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T05:07:35.442 INFO:tasks.ceph.osd.2.smithi114.stderr:2024-09-16T05:07:35.442+0000 7fadebf687c0 -1 osd.2 0 log_to_monitors true 2024-09-16T05:07:35.443 INFO:tasks.ceph.osd.5.smithi149.stderr:2024-09-16T05:07:35.441+0000 7fdec4e337c0 -1 osd.5 0 log_to_monitors true 2024-09-16T05:07:35.556 INFO:tasks.ceph.osd.0.smithi114.stderr:2024-09-16T05:07:35.554+0000 7f4471e627c0 -1 osd.0 0 log_to_monitors true 2024-09-16T05:07:36.144 INFO:tasks.ceph.osd.3.smithi149.stderr:2024-09-16T05:07:36.141+0000 7f1a3e5f4640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T05:07:36.154 INFO:tasks.ceph.osd.3.smithi149.stderr:2024-09-16T05:07:36.153+0000 7f1a3ac43640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:07:36.708 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T05:07:36.828 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.826+0000 7fccc2e55640 10 monclient: get_monmap_and_config 2024-09-16T05:07:36.828 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.826+0000 7fccc2e55640 10 monclient: build_initial_monmap 2024-09-16T05:07:36.828 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.826+0000 7fccc2e55640 10 monclient: monmap: 2024-09-16T05:07:36.828 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-09-16T05:07:36.828 INFO:teuthology.misc.health.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:36.828 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-09-16T05:07:36.831385+0000 2024-09-16T05:07:36.828 INFO:teuthology.misc.health.smithi114.stderr:created 2024-09-16T05:07:36.831385+0000 2024-09-16T05:07:36.828 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:36.829 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:36.829 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:36.829 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:36.829 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:36.829 INFO:teuthology.misc.health.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:36.829 INFO:teuthology.misc.health.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:36.829 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:36.831 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 1 Processor -- start 2024-09-16T05:07:36.831 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 1 -- start start 2024-09-16T05:07:36.832 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 10 monclient: init 2024-09-16T05:07:36.832 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:36.832 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-16T05:07:36.832 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fccbc1375e0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:36.832 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fccbc0a5e20 0x7fccbc09c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:36.832 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 10 monclient(hunting): picked mon.noname-a con 0x7fccbc0a5e20 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:36.832 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccbc13c4b0 0x7fccbc13e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:36.832 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 10 monclient(hunting): picked mon.noname-b con 0x7fccbc13c4b0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:36.832 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccbc09bb90 con 0x7fccbc1375e0 2024-09-16T05:07:36.832 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:36.832 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fccbc09be70 con 0x7fccbc0a5e20 2024-09-16T05:07:36.832 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:36.833 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fccbc09c150 con 0x7fccbc13c4b0 2024-09-16T05:07:36.833 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:36.833 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc2e55640 10 monclient(hunting): authenticate will time out at 941.461272s 2024-09-16T05:07:36.833 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc13f2640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccbc13c4b0 0x7fccbc13e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:36.833 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fccbc0a5e20 0x7fccbc09c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:36.833 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc13f2640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccbc13c4b0 0x7fccbc13e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55936/0 (socket says 172.21.15.114:55936) 2024-09-16T05:07:36.833 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc13f2640 1 -- 172.21.15.114:0/3226308104 learned_addr learned my addr 172.21.15.114:0/3226308104 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:36.833 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc13f2640 10 monclient(hunting): get_auth_request con 0x7fccbc13c4b0 auth_method 0 2024-09-16T05:07:36.833 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc13f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:36.833 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc13f2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:36.833 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc13f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:36.833 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 10 monclient(hunting): get_auth_request con 0x7fccbc0a5e20 auth_method 0 2024-09-16T05:07:36.833 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:36.834 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:36.834 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:36.834 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc13f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:36.834 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc13f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:36.834 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc13f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:36.834 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:36.834 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:36.834 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:36.834 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 10 monclient(hunting): handle_auth_done global_id 4179 payload 274 2024-09-16T05:07:36.834 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 1 -- 172.21.15.114:0/3226308104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccbc13c4b0 msgr2=0x7fccbc13e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:36.834 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 1 --2- 172.21.15.114:0/3226308104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccbc13c4b0 0x7fccbc13e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:36.834 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 1 -- 172.21.15.114:0/3226308104 >> v1:172.21.15.149:6789/0 conn(0x7fccbc1375e0 legacy=0x7fccbc1379c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:36.834 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 10 monclient: _finish_hunting 0 2024-09-16T05:07:36.834 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 1 monclient: found mon.noname-a 2024-09-16T05:07:36.835 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:36.835 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:36.835 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbbfff640 1 -- 172.21.15.114:0/3226308104 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccbc09c430 con 0x7fccbc0a5e20 2024-09-16T05:07:36.835 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbb7fe640 1 -- 172.21.15.114:0/3226308104 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1409861559 0 0) 0x7fccbc09bb90 con 0x7fccbc1375e0 2024-09-16T05:07:36.835 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccbb7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:07:36.835 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.830+0000 7fccc13f2640 1 --2- 172.21.15.114:0/3226308104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccbc13c4b0 0x7fccbc13e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:36.840 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbbfff640 1 --2- 172.21.15.114:0/3226308104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc0a5e20 0x7fccbc09c850 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fccb00097c0 tx=0x7fccb001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c37f59acb10406 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:36.840 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbb7fe640 1 -- 172.21.15.114:0/3226308104 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb00b0070 con 0x7fccbc0a5e20 2024-09-16T05:07:36.841 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:36.841 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:36.841 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbb7fe640 10 monclient: dump: 2024-09-16T05:07:36.841 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-09-16T05:07:36.841 INFO:teuthology.misc.health.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:36.841 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:36.841 INFO:teuthology.misc.health.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:36.841 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:36.841 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:36.841 INFO:teuthology.misc.health.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:36.841 INFO:teuthology.misc.health.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:36.841 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:36.841 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:36.841 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbb7fe640 10 monclient: _finish_auth 0 2024-09-16T05:07:36.842 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbb7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:07:36.842 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:36.842 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbb7fe640 1 -- 172.21.15.114:0/3226308104 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccb00ab600 con 0x7fccbc0a5e20 2024-09-16T05:07:36.842 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 5 monclient: authenticate success, global_id 4179 2024-09-16T05:07:36.842 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:36.842 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:36.842 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbb7fe640 1 -- 172.21.15.114:0/3226308104 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb00ab9d0 con 0x7fccbc0a5e20 2024-09-16T05:07:36.842 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:36.842 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:36.842 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbb7fe640 10 monclient: dump: 2024-09-16T05:07:36.842 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-09-16T05:07:36.842 INFO:teuthology.misc.health.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:36.842 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:36.842 INFO:teuthology.misc.health.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:36.842 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:36.843 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:36.843 INFO:teuthology.misc.health.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:36.843 INFO:teuthology.misc.health.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:36.843 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:36.843 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:36.843 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:36.843 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 4 set_mon_vals no callback set 2024-09-16T05:07:36.843 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 10 monclient: shutdown 2024-09-16T05:07:36.843 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:36.843 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 -- 172.21.15.114:0/3226308104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc0a5e20 msgr2=0x7fccbc09c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:36.843 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 --2- 172.21.15.114:0/3226308104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc0a5e20 0x7fccbc09c850 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fccb00097c0 tx=0x7fccb001c360 comp rx=0 tx=0).stop 2024-09-16T05:07:36.843 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 -- 172.21.15.114:0/3226308104 shutdown_connections 2024-09-16T05:07:36.843 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 --2- 172.21.15.114:0/3226308104 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccbc13c4b0 0x7fccbc13e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:36.843 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 --2- 172.21.15.114:0/3226308104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc0a5e20 0x7fccbc09c850 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:36.844 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 -- 172.21.15.114:0/3226308104 >> 172.21.15.114:0/3226308104 conn(0x7fccbc0a9dd0 msgr2=0x7fccbc0aa1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:36.844 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 -- 172.21.15.114:0/3226308104 shutdown_connections 2024-09-16T05:07:36.844 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 -- 172.21.15.114:0/3226308104 wait complete. 2024-09-16T05:07:36.844 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 10 monclient: build_initial_monmap 2024-09-16T05:07:36.844 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 10 monclient: monmap: 2024-09-16T05:07:36.844 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-09-16T05:07:36.844 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:36.844 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:36.844 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-09-16T05:07:36.844 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:36.844 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:36.844 INFO:teuthology.misc.health.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:36.844 INFO:teuthology.misc.health.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:36.845 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:36.845 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:36.845 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 librados: starting msgr at 2024-09-16T05:07:36.845 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 librados: starting objecter 2024-09-16T05:07:36.845 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 Processor -- start 2024-09-16T05:07:36.845 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 -- start start 2024-09-16T05:07:36.845 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 librados: setting wanted keys 2024-09-16T05:07:36.845 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 librados: calling monclient init 2024-09-16T05:07:36.845 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 10 monclient: init 2024-09-16T05:07:36.845 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:36.845 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:07:36.845 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccbc0a5e20 0x7fccbc136250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:36.845 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 10 monclient(hunting): picked mon.noname-c con 0x7fccbc0a5e20 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:36.845 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fccbc1375e0 0x7fccbc09fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:36.846 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 10 monclient(hunting): picked mon.noname-b con 0x7fccbc1375e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:36.846 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc13c4b0 0x7fccbc0a0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:36.846 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 10 monclient(hunting): picked mon.noname-a con 0x7fccbc13c4b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:36.846 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:36.846 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc141990 con 0x7fccbc13c4b0 2024-09-16T05:07:36.846 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:36.846 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc141810 con 0x7fccbc1375e0 2024-09-16T05:07:36.846 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:36.846 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fccbc141b10 con 0x7fccbc0a5e20 2024-09-16T05:07:36.846 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:36.846 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc2e55640 10 monclient(hunting): authenticate will time out at 941.467521s 2024-09-16T05:07:36.846 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc0bf1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccbc0a5e20 0x7fccbc136250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:36.846 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc0bf1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccbc0a5e20 0x7fccbc136250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55940/0 (socket says 172.21.15.114:55940) 2024-09-16T05:07:36.846 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc0bf1640 1 -- 172.21.15.114:0/3724517295 learned_addr learned my addr 172.21.15.114:0/3724517295 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:36.847 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc0bf1640 10 monclient(hunting): get_auth_request con 0x7fccbc0a5e20 auth_method 0 2024-09-16T05:07:36.847 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc0bf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:36.847 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc0bf1640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:36.847 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc0bf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:36.847 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 1 --2- 172.21.15.114:0/3724517295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc13c4b0 0x7fccbc0a0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:36.847 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 10 monclient(hunting): get_auth_request con 0x7fccbc13c4b0 auth_method 0 2024-09-16T05:07:36.847 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:36.847 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:36.847 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:36.847 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbbfff640 1 --2- 172.21.15.114:0/3724517295 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fccbc1375e0 0x7fccbc09fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:36.847 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbbfff640 10 monclient(hunting): get_auth_request con 0x7fccbc1375e0 auth_method 0 2024-09-16T05:07:36.847 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:36.847 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:36.847 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:36.848 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:36.848 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:36.848 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:36.848 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc0bf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:36.848 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc0bf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:36.848 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc0bf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:36.848 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:36.848 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:36.848 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:36.848 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 10 monclient(hunting): handle_auth_done global_id 4182 payload 824 2024-09-16T05:07:36.848 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 1 -- 172.21.15.114:0/3724517295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccbc0a5e20 msgr2=0x7fccbc136250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:36.848 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 1 --2- 172.21.15.114:0/3724517295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccbc0a5e20 0x7fccbc136250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:36.848 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 1 -- 172.21.15.114:0/3724517295 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fccbc1375e0 msgr2=0x7fccbc09fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:36.848 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 1 --2- 172.21.15.114:0/3724517295 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fccbc1375e0 0x7fccbc09fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:36.849 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 10 monclient: _finish_hunting 0 2024-09-16T05:07:36.849 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 1 monclient: found mon.noname-a 2024-09-16T05:07:36.849 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:36.849 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:36.849 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc13f2640 1 -- 172.21.15.114:0/3724517295 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccbc120950 con 0x7fccbc13c4b0 2024-09-16T05:07:36.849 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.834+0000 7fccc0bf1640 1 --2- 172.21.15.114:0/3724517295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccbc0a5e20 0x7fccbc136250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:36.849 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc13f2640 1 --2- 172.21.15.114:0/3724517295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc13c4b0 0x7fccbc0a0040 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fccb400efc0 tx=0x7fccb400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:36.849 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 10 client.?.objecter ms_handle_connect 0x7fccbc13c4b0 2024-09-16T05:07:36.849 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:36.849 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 1 -- 172.21.15.114:0/3724517295 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb400cb20 con 0x7fccbc13c4b0 2024-09-16T05:07:36.849 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:36.850 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:36.850 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 10 monclient: dump: 2024-09-16T05:07:36.850 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-09-16T05:07:36.850 INFO:teuthology.misc.health.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:36.850 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:36.850 INFO:teuthology.misc.health.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:36.850 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:36.851 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:36.852 INFO:teuthology.misc.health.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:36.852 INFO:teuthology.misc.health.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:36.852 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:36.852 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:36.852 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 10 monclient: _finish_auth 0 2024-09-16T05:07:36.852 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 10 monclient: _check_auth_tickets 2024-09-16T05:07:36.852 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:36.852 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 1 -- 172.21.15.114:0/3724517295 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccb400cce0 con 0x7fccbc13c4b0 2024-09-16T05:07:36.853 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc2e55640 5 monclient: authenticate success, global_id 4182 2024-09-16T05:07:36.853 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc2e55640 10 monclient: _renew_subs 2024-09-16T05:07:36.853 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc2e55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:36.853 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc2e55640 1 -- 172.21.15.114:0/3724517295 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccbc120c30 con 0x7fccbc13c4b0 2024-09-16T05:07:36.853 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:36.853 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 1 -- 172.21.15.114:0/3724517295 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb4003be0 con 0x7fccbc13c4b0 2024-09-16T05:07:36.853 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:36.853 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:36.853 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 10 monclient: dump: 2024-09-16T05:07:36.853 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-09-16T05:07:36.853 INFO:teuthology.misc.health.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:36.854 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:36.854 INFO:teuthology.misc.health.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:36.854 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:36.854 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:36.854 INFO:teuthology.misc.health.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:36.854 INFO:teuthology.misc.health.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:36.854 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:36.854 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:36.854 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc2e55640 10 client.4182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:36.854 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc2e55640 10 monclient: _renew_subs 2024-09-16T05:07:36.854 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccba7fc640 4 set_mon_vals no callback set 2024-09-16T05:07:36.854 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc2e55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:36.854 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc2e55640 1 -- 172.21.15.114:0/3724517295 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccbc0a87e0 con 0x7fccbc13c4b0 2024-09-16T05:07:36.855 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc2e55640 1 librados: init done 2024-09-16T05:07:36.855 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc2e55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:36.855 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc2e55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:36.855 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc2e55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:36.855 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc2e55640 1 -- 172.21.15.114:0/3724517295 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccbc09c850 con 0x7fccbc13c4b0 2024-09-16T05:07:36.855 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 1 -- 172.21.15.114:0/3724517295 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7fccb401b030 con 0x7fccbc13c4b0 2024-09-16T05:07:36.855 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 1 --2- 172.21.15.114:0/3724517295 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcca8037780 0x7fcca8039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:36.855 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 1 -- 172.21.15.114:0/3724517295 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1753+0+0 (secure 0 0 0) 0x7fccb40481c0 con 0x7fccbc13c4b0 2024-09-16T05:07:36.855 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 10 client.4182.objecter ms_dispatch 0x7fccbc000ba0 osd_map(10..10 src has 1..10) 2024-09-16T05:07:36.855 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 3 client.4182.objecter handle_osd_map got epochs [10,10] > 0 2024-09-16T05:07:36.856 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 3 client.4182.objecter handle_osd_map decoding full epoch 10 2024-09-16T05:07:36.856 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 20 client.4182.objecter dump_active .. 0 homeless 2024-09-16T05:07:36.856 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc0bf1640 1 --2- 172.21.15.114:0/3724517295 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcca8037780 0x7fcca8039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:36.856 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc0bf1640 10 monclient: get_auth_request con 0x7fcca8037780 auth_method 0 2024-09-16T05:07:36.856 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccc0bf1640 1 --2- 172.21.15.114:0/3724517295 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcca8037780 0x7fcca8039c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fccac007fd0 tx=0x7fccac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:36.856 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.838+0000 7fccb97fa640 10 client.4182.objecter ms_handle_connect 0x7fcca8037780 2024-09-16T05:07:36.857 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.842+0000 7fccb97fa640 1 -- 172.21.15.114:0/3724517295 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fccb4022020 con 0x7fccbc13c4b0 2024-09-16T05:07:36.857 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.842+0000 7fccb97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:36.857 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:36.842+0000 7fccb97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:37.006 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.006+0000 7fccc2e55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:37.006 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.006+0000 7fccc2e55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:37.006 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.006+0000 7fccc2e55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:37.006 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.006+0000 7fccc2e55640 1 -- 172.21.15.114:0/3724517295 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fccbc073f50 con 0x7fccbc13c4b0 2024-09-16T05:07:37.007 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.006+0000 7fccb97fa640 1 -- 172.21.15.114:0/3724517295 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+5606 (secure 0 0 0) 0x7fccb4014e20 con 0x7fccbc13c4b0 2024-09-16T05:07:37.007 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.006+0000 7fccb97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:37.007 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.006+0000 7fccb97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:37.007 INFO:teuthology.misc.health.smithi114.stdout: 2024-09-16T05:07:37.007 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","created":"2024-09-16T05:07:27.305861+0000","modified":"2024-09-16T05:07:36.128579+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T05:07:29.028596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ffdab41-0191-4764-ad62-922ec1091569","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7de579ee-87a3-4463-87a5-6980b56cd113","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"258c8bf4-58c5-4c13-a0e5-ef9931efd698","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"acc1e03f-8a6c-4c85-9525-88707597d732","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"92cca4cf-19e9-44a1-9588-7c301a2c2065","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac717f4e-83fc-46ea-9816-7e35ef75a674","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:07:37.010 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 10 librados: watch_flush enter 2024-09-16T05:07:37.010 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 10 librados: watch_flush exit 2024-09-16T05:07:37.010 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 20 client.4182.objecter shutdown clearing up homeless session... 2024-09-16T05:07:37.010 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 10 client.4182.objecter successfully canceled tick 2024-09-16T05:07:37.010 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 1 -- 172.21.15.114:0/3724517295 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcca8037780 msgr2=0x7fcca8039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:37.010 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 1 --2- 172.21.15.114:0/3724517295 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcca8037780 0x7fcca8039c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fccac007fd0 tx=0x7fccac009040 comp rx=0 tx=0).stop 2024-09-16T05:07:37.010 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 10 monclient: shutdown 2024-09-16T05:07:37.010 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:37.010 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 1 -- 172.21.15.114:0/3724517295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc13c4b0 msgr2=0x7fccbc0a0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:37.010 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 1 --2- 172.21.15.114:0/3724517295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc13c4b0 0x7fccbc0a0040 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fccb400efc0 tx=0x7fccb400c490 comp rx=0 tx=0).stop 2024-09-16T05:07:37.010 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 1 -- 172.21.15.114:0/3724517295 shutdown_connections 2024-09-16T05:07:37.010 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 1 --2- 172.21.15.114:0/3724517295 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcca8037780 0x7fcca8039c40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:37.011 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 1 --2- 172.21.15.114:0/3724517295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc13c4b0 0x7fccbc0a0040 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:37.011 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 1 --2- 172.21.15.114:0/3724517295 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fccbc1375e0 0x7fccbc09fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:37.011 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 1 --2- 172.21.15.114:0/3724517295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccbc0a5e20 0x7fccbc136250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:37.011 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 1 -- 172.21.15.114:0/3724517295 >> 172.21.15.114:0/3724517295 conn(0x7fccbc0a9dd0 msgr2=0x7fccbc13d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:37.011 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 1 -- 172.21.15.114:0/3724517295 shutdown_connections 2024-09-16T05:07:37.011 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 1 -- 172.21.15.114:0/3724517295 wait complete. 2024-09-16T05:07:37.011 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:37.010+0000 7fccc2e55640 1 librados: shutdown 2024-09-16T05:07:37.021 INFO:tasks.ceph.osd.4.smithi149.stderr:2024-09-16T05:07:37.017+0000 7f59db5da640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T05:07:37.026 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-16T05:07:37.029 INFO:tasks.ceph.osd.4.smithi149.stderr:2024-09-16T05:07:37.025+0000 7f59d3c21640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:07:37.143 INFO:tasks.ceph.osd.1.smithi114.stderr:2024-09-16T05:07:37.142+0000 7f3f63c91640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T05:07:37.179 INFO:tasks.ceph.osd.1.smithi114.stderr:2024-09-16T05:07:37.178+0000 7f3f602e0640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:07:38.138 INFO:tasks.ceph.osd.5.smithi149.stderr:2024-09-16T05:07:38.137+0000 7fdec05b9640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T05:07:38.144 INFO:tasks.ceph.osd.5.smithi149.stderr:2024-09-16T05:07:38.141+0000 7fdeb8c00640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:07:38.582 INFO:tasks.ceph.osd.0.smithi114.stderr:2024-09-16T05:07:38.582+0000 7f446d5e8640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T05:07:38.590 INFO:tasks.ceph.osd.0.smithi114.stderr:2024-09-16T05:07:38.590+0000 7f4465c2f640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:07:38.594 INFO:tasks.ceph.osd.2.smithi114.stderr:2024-09-16T05:07:38.594+0000 7fade8723640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T05:07:38.602 INFO:tasks.ceph.osd.2.smithi114.stderr:2024-09-16T05:07:38.602+0000 7faddfd35640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:07:43.632 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T05:07:43.752 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.750+0000 7f2481620640 10 monclient: get_monmap_and_config 2024-09-16T05:07:43.752 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.750+0000 7f2481620640 10 monclient: build_initial_monmap 2024-09-16T05:07:43.752 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.750+0000 7f2481620640 10 monclient: monmap: 2024-09-16T05:07:43.753 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-09-16T05:07:43.753 INFO:teuthology.misc.health.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:43.753 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-09-16T05:07:43.755814+0000 2024-09-16T05:07:43.753 INFO:teuthology.misc.health.smithi114.stderr:created 2024-09-16T05:07:43.755814+0000 2024-09-16T05:07:43.753 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:43.753 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:43.753 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:43.753 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:43.753 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:43.753 INFO:teuthology.misc.health.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:43.753 INFO:teuthology.misc.health.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:43.753 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:43.755 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 1 Processor -- start 2024-09-16T05:07:43.755 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 1 -- start start 2024-09-16T05:07:43.755 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 10 monclient: init 2024-09-16T05:07:43.756 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:43.756 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-16T05:07:43.756 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f247c09b780 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:43.756 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f247c09ea50 0x7f247c0a1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:43.756 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 10 monclient(hunting): picked mon.noname-a con 0x7f247c09ea50 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:43.756 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247c13a710 0x7f247c13cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:43.756 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 10 monclient(hunting): picked mon.noname-b con 0x7f247c13a710 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:43.756 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f247c09f640 con 0x7f247c09b780 2024-09-16T05:07:43.757 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:43.757 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f247c09bb60 con 0x7f247c09ea50 2024-09-16T05:07:43.757 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:43.757 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f247c09c320 con 0x7f247c13a710 2024-09-16T05:07:43.757 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:43.757 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 10 monclient(hunting): authenticate will time out at 948.385065s 2024-09-16T05:07:43.757 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247c13a710 0x7f247c13cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:43.757 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f24727fc640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f247c09ea50 0x7f247c0a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:43.757 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247c13a710 0x7f247c13cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55946/0 (socket says 172.21.15.114:55946) 2024-09-16T05:07:43.757 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 1 -- 172.21.15.114:0/10538281 learned_addr learned my addr 172.21.15.114:0/10538281 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:43.757 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 10 monclient(hunting): get_auth_request con 0x7f247c13a710 auth_method 0 2024-09-16T05:07:43.757 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:43.758 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:43.758 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:43.758 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f24727fc640 10 monclient(hunting): get_auth_request con 0x7f247c09ea50 auth_method 0 2024-09-16T05:07:43.758 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f24727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:43.758 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f24727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:43.758 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f24727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:43.758 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:43.758 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:43.758 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:43.758 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f24727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:43.758 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f24727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:43.758 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f24727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:43.758 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 10 monclient(hunting): handle_auth_done global_id 4187 payload 274 2024-09-16T05:07:43.758 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 1 -- 172.21.15.114:0/10538281 >> v2:172.21.15.114:3300/0 conn(0x7f247c09ea50 msgr2=0x7f247c0a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:43.759 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 1 --2- 172.21.15.114:0/10538281 >> v2:172.21.15.114:3300/0 conn(0x7f247c09ea50 0x7f247c0a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:43.759 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 1 -- 172.21.15.114:0/10538281 >> v1:172.21.15.149:6789/0 conn(0x7f247c09b780 legacy=0x7f247c0a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:43.759 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:07:43.759 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 1 monclient: found mon.noname-b 2024-09-16T05:07:43.759 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:43.759 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:43.759 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 1 -- 172.21.15.114:0/10538281 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247c09ccf0 con 0x7f247c13a710 2024-09-16T05:07:43.768 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247b7fe640 1 --2- 172.21.15.114:0/10538281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247c13a710 0x7f247c13cb00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2474009e50 tx=0x7f247401c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8b45cab6f91fad9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:43.768 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247a7fc640 1 -- 172.21.15.114:0/10538281 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24740a5700 con 0x7f247c13a710 2024-09-16T05:07:43.768 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:43.768 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:43.768 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247a7fc640 10 monclient: dump: 2024-09-16T05:07:43.768 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-09-16T05:07:43.768 INFO:teuthology.misc.health.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:43.768 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:43.768 INFO:teuthology.misc.health.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:43.769 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:43.769 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:43.769 INFO:teuthology.misc.health.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:43.769 INFO:teuthology.misc.health.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:43.769 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:43.769 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:43.769 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:07:43.769 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:07:43.769 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:43.769 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 5 monclient: authenticate success, global_id 4187 2024-09-16T05:07:43.769 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:43.769 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247a7fc640 1 -- 172.21.15.114:0/10538281 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24740a58a0 con 0x7f247c13a710 2024-09-16T05:07:43.769 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:43.769 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:43.770 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f2481620640 4 set_mon_vals no callback set 2024-09-16T05:07:43.770 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247a7fc640 1 -- 172.21.15.114:0/10538281 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24740a5c70 con 0x7f247c13a710 2024-09-16T05:07:43.770 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:43.770 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:43.770 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.754+0000 7f247a7fc640 10 monclient: dump: 2024-09-16T05:07:43.770 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-09-16T05:07:43.770 INFO:teuthology.misc.health.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:43.770 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:43.770 INFO:teuthology.misc.health.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:43.770 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:43.770 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:43.770 INFO:teuthology.misc.health.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:43.770 INFO:teuthology.misc.health.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:43.770 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:43.771 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:43.771 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient: shutdown 2024-09-16T05:07:43.771 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:43.771 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 -- 172.21.15.114:0/10538281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247c13a710 msgr2=0x7f247c13cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:43.771 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 --2- 172.21.15.114:0/10538281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247c13a710 0x7f247c13cb00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2474009e50 tx=0x7f247401c880 comp rx=0 tx=0).stop 2024-09-16T05:07:43.771 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 -- 172.21.15.114:0/10538281 shutdown_connections 2024-09-16T05:07:43.771 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 --2- 172.21.15.114:0/10538281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247c13a710 0x7f247c13cb00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:43.771 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 --2- 172.21.15.114:0/10538281 >> v2:172.21.15.114:3300/0 conn(0x7f247c09ea50 0x7f247c0a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:43.771 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 -- 172.21.15.114:0/10538281 >> 172.21.15.114:0/10538281 conn(0x7f247c0a2070 msgr2=0x7f247c0a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:43.771 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 -- 172.21.15.114:0/10538281 shutdown_connections 2024-09-16T05:07:43.771 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 -- 172.21.15.114:0/10538281 wait complete. 2024-09-16T05:07:43.771 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient: build_initial_monmap 2024-09-16T05:07:43.771 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient: monmap: 2024-09-16T05:07:43.771 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-09-16T05:07:43.772 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:43.772 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:43.772 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-09-16T05:07:43.772 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:43.772 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:43.772 INFO:teuthology.misc.health.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:43.772 INFO:teuthology.misc.health.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:43.772 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:43.772 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 librados: starting msgr at 2024-09-16T05:07:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 librados: starting objecter 2024-09-16T05:07:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 Processor -- start 2024-09-16T05:07:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 -- start start 2024-09-16T05:07:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 librados: setting wanted keys 2024-09-16T05:07:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 librados: calling monclient init 2024-09-16T05:07:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient: init 2024-09-16T05:07:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:07:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f247c09b780 0x7f247c098920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient(hunting): picked mon.noname-a con 0x7f247c09b780 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247c09ea50 0x7f247c08efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient(hunting): picked mon.noname-c con 0x7f247c09ea50 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f247c13a710 0x7f247c08f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient(hunting): picked mon.noname-b con 0x7f247c13a710 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f247c13f860 con 0x7f247c09b780 2024-09-16T05:07:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f247c09ef40 con 0x7f247c13a710 2024-09-16T05:07:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f247c13f9e0 con 0x7f247c09ea50 2024-09-16T05:07:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient(hunting): authenticate will time out at 948.390260s 2024-09-16T05:07:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f247affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f247c09b780 0x7f247c098920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247c09ea50 0x7f247c08efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f247affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f247c09b780 0x7f247c098920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48606/0 (socket says 172.21.15.114:48606) 2024-09-16T05:07:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f247affd640 1 -- 172.21.15.114:0/1453320005 learned_addr learned my addr 172.21.15.114:0/1453320005 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:43.775 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f247affd640 10 monclient(hunting): get_auth_request con 0x7f247c09b780 auth_method 0 2024-09-16T05:07:43.775 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f247affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:43.775 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f247affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:43.775 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f247affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:43.775 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 10 monclient(hunting): get_auth_request con 0x7f247c09ea50 auth_method 0 2024-09-16T05:07:43.775 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:43.775 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:43.775 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:43.775 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f247affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:43.777 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f247affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:43.777 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f247affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:43.777 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:43.777 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:43.777 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:43.777 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 10 monclient(hunting): handle_auth_done global_id 4190 payload 824 2024-09-16T05:07:43.777 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 1 -- 172.21.15.114:0/1453320005 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f247c13a710 msgr2=0x7f247c08f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:43.777 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f247b7fe640 1 --2- 172.21.15.114:0/1453320005 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f247c13a710 0x7f247c08f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:43.777 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 1 --2- 172.21.15.114:0/1453320005 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f247c13a710 0x7f247c08f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:43.777 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 1 -- 172.21.15.114:0/1453320005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f247c09b780 msgr2=0x7f247c098920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:43.777 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 1 --2- 172.21.15.114:0/1453320005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f247c09b780 0x7f247c098920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:43.777 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 10 monclient: _finish_hunting 0 2024-09-16T05:07:43.777 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 1 monclient: found mon.noname-c 2024-09-16T05:07:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 1 -- 172.21.15.114:0/1453320005 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247c098e60 con 0x7f247c09ea50 2024-09-16T05:07:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f247affd640 1 --2- 172.21.15.114:0/1453320005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f247c09b780 0x7f247c098920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f24727fc640 1 --2- 172.21.15.114:0/1453320005 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247c09ea50 0x7f247c08efd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2464009c50 tx=0x7f246400b510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2473fff640 10 client.?.objecter ms_handle_connect 0x7f247c09ea50 2024-09-16T05:07:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2473fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2473fff640 1 -- 172.21.15.114:0/1453320005 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246401e070 con 0x7f247c09ea50 2024-09-16T05:07:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2473fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2473fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2473fff640 10 monclient: dump: 2024-09-16T05:07:43.778 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-09-16T05:07:43.778 INFO:teuthology.misc.health.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:43.778 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:43.779 INFO:teuthology.misc.health.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:43.779 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:43.779 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:43.779 INFO:teuthology.misc.health.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:43.779 INFO:teuthology.misc.health.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:43.779 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:43.779 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:43.779 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2473fff640 10 monclient: _finish_auth 0 2024-09-16T05:07:43.779 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2473fff640 10 monclient: _check_auth_tickets 2024-09-16T05:07:43.779 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2473fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:43.779 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2473fff640 1 -- 172.21.15.114:0/1453320005 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f246400be30 con 0x7f247c09ea50 2024-09-16T05:07:43.780 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 5 monclient: authenticate success, global_id 4190 2024-09-16T05:07:43.780 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient: _renew_subs 2024-09-16T05:07:43.780 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:43.780 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 -- 172.21.15.114:0/1453320005 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f247c092f60 con 0x7f247c09ea50 2024-09-16T05:07:43.780 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2473fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:43.780 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2473fff640 1 -- 172.21.15.114:0/1453320005 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2464019410 con 0x7f247c09ea50 2024-09-16T05:07:43.780 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2473fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:43.780 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2473fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:43.781 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2473fff640 10 monclient: dump: 2024-09-16T05:07:43.781 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-09-16T05:07:43.781 INFO:teuthology.misc.health.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:43.781 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:43.781 INFO:teuthology.misc.health.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:43.781 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:43.781 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:43.781 INFO:teuthology.misc.health.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:43.781 INFO:teuthology.misc.health.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:43.781 INFO:teuthology.misc.health.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:43.781 INFO:teuthology.misc.health.smithi114.stderr: 2024-09-16T05:07:43.781 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 client.4190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:43.781 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient: _renew_subs 2024-09-16T05:07:43.781 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:43.782 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 -- 172.21.15.114:0/1453320005 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f247c09c760 con 0x7f247c09ea50 2024-09-16T05:07:43.782 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2481620640 1 librados: init done 2024-09-16T05:07:43.782 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.758+0000 7f2479ffb640 4 set_mon_vals no callback set 2024-09-16T05:07:43.782 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.762+0000 7f2473fff640 1 -- 172.21.15.114:0/1453320005 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2464021070 con 0x7f247c09ea50 2024-09-16T05:07:43.782 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.762+0000 7f2481620640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:43.782 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.762+0000 7f2481620640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:43.782 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.762+0000 7f2481620640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:43.782 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.762+0000 7f2481620640 1 -- 172.21.15.114:0/1453320005 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f247c0a0f80 con 0x7f247c09ea50 2024-09-16T05:07:43.782 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.762+0000 7f2473fff640 1 --2- 172.21.15.114:0/1453320005 >> v2:172.21.15.149:6812/1185531135 conn(0x7f245c037810 0x7f245c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:43.782 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.762+0000 7f2473fff640 1 -- 172.21.15.114:0/1453320005 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 3153+0+0 (secure 0 0 0) 0x7f246404be80 con 0x7f247c09ea50 2024-09-16T05:07:43.782 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.762+0000 7f2473fff640 10 client.4190.objecter ms_dispatch 0x7f247c000ba0 osd_map(14..14 src has 1..14) 2024-09-16T05:07:43.782 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.762+0000 7f2473fff640 3 client.4190.objecter handle_osd_map got epochs [14,14] > 0 2024-09-16T05:07:43.782 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.762+0000 7f2473fff640 3 client.4190.objecter handle_osd_map decoding full epoch 14 2024-09-16T05:07:43.782 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.762+0000 7f2473fff640 20 client.4190.objecter dump_active .. 0 homeless 2024-09-16T05:07:43.783 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.762+0000 7f247affd640 1 --2- 172.21.15.114:0/1453320005 >> v2:172.21.15.149:6812/1185531135 conn(0x7f245c037810 0x7f245c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:43.783 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.762+0000 7f247affd640 10 monclient: get_auth_request con 0x7f245c037810 auth_method 0 2024-09-16T05:07:43.783 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.762+0000 7f2473fff640 10 client.4190.objecter ms_handle_connect 0x7f245c037810 2024-09-16T05:07:43.783 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.762+0000 7f247affd640 1 --2- 172.21.15.114:0/1453320005 >> v2:172.21.15.149:6812/1185531135 conn(0x7f245c037810 0x7f245c039cd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f246c004470 tx=0x7f246c003700 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:43.783 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.766+0000 7f2473fff640 1 -- 172.21.15.114:0/1453320005 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f247c098e60 con 0x7f247c09ea50 2024-09-16T05:07:43.783 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.766+0000 7f2473fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:43.783 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.766+0000 7f2473fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:43.931 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.930+0000 7f2481620640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:43.931 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.930+0000 7f2481620640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:43.931 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.930+0000 7f2481620640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:43.931 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.930+0000 7f2481620640 1 -- 172.21.15.114:0/1453320005 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f247c068940 con 0x7f247c09ea50 2024-09-16T05:07:43.932 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.930+0000 7f2473fff640 1 -- 172.21.15.114:0/1453320005 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+9073 (secure 0 0 0) 0x7f24640174d0 con 0x7f247c09ea50 2024-09-16T05:07:43.932 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.930+0000 7f2473fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:43.932 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.930+0000 7f2473fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:43.932 INFO:teuthology.misc.health.smithi114.stdout: 2024-09-16T05:07:43.932 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":14,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","created":"2024-09-16T05:07:27.305861+0000","modified":"2024-09-16T05:07:40.189308+0000","last_up_change":"2024-09-16T05:07:39.180058+0000","last_in_change":"2024-09-16T05:07:29.028596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:07:38.142989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6ffdab41-0191-4764-ad62-922ec1091569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":269070361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":269070361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6811","nonce":269070361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":269070361}]},"public_addr":"172.21.15.114:6808/269070361","cluster_addr":"172.21.15.114:6809/269070361","heartbeat_back_addr":"172.21.15.114:6811/269070361","heartbeat_front_addr":"172.21.15.114:6810/269070361","state":["exists","up"]},{"osd":1,"uuid":"7de579ee-87a3-4463-87a5-6980b56cd113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2903720358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":2903720358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6803","nonce":2903720358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2903720358}]},"public_addr":"172.21.15.114:6800/2903720358","cluster_addr":"172.21.15.114:6801/2903720358","heartbeat_back_addr":"172.21.15.114:6803/2903720358","heartbeat_front_addr":"172.21.15.114:6802/2903720358","state":["exists","up"]},{"osd":2,"uuid":"258c8bf4-58c5-4c13-a0e5-ef9931efd698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1932657732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":1932657732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6807","nonce":1932657732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1932657732}]},"public_addr":"172.21.15.114:6804/1932657732","cluster_addr":"172.21.15.114:6805/1932657732","heartbeat_back_addr":"172.21.15.114:6807/1932657732","heartbeat_front_addr":"172.21.15.114:6806/1932657732","state":["exists","up"]},{"osd":3,"uuid":"acc1e03f-8a6c-4c85-9525-88707597d732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6805","nonce":1362772130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6807","nonce":1362772130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6811","nonce":1362772130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6809","nonce":1362772130}]},"public_addr":"172.21.15.149:6805/1362772130","cluster_addr":"172.21.15.149:6807/1362772130","heartbeat_back_addr":"172.21.15.149:6811/1362772130","heartbeat_front_addr":"172.21.15.149:6809/1362772130","state":["exists","up"]},{"osd":4,"uuid":"92cca4cf-19e9-44a1-9588-7c301a2c2065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":1092505261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6801","nonce":1092505261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6803","nonce":1092505261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":1092505261}]},"public_addr":"172.21.15.149:6800/1092505261","cluster_addr":"172.21.15.149:6801/1092505261","heartbeat_back_addr":"172.21.15.149:6803/1092505261","heartbeat_front_addr":"172.21.15.149:6802/1092505261","state":["exists","up"]},{"osd":5,"uuid":"ac717f4e-83fc-46ea-9816-7e35ef75a674","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":1244703827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":1244703827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":1244703827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":1244703827}]},"public_addr":"172.21.15.149:6804/1244703827","cluster_addr":"172.21.15.149:6806/1244703827","heartbeat_back_addr":"172.21.15.149:6810/1244703827","heartbeat_front_addr":"172.21.15.149:6808/1244703827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.554731+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.453254+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.446266+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:35.709335+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.069934+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.433872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:07:43.937 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 10 librados: watch_flush enter 2024-09-16T05:07:43.937 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 10 librados: watch_flush exit 2024-09-16T05:07:43.937 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 20 client.4190.objecter shutdown clearing up homeless session... 2024-09-16T05:07:43.937 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 10 client.4190.objecter successfully canceled tick 2024-09-16T05:07:43.937 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 1 -- 172.21.15.114:0/1453320005 >> v2:172.21.15.149:6812/1185531135 conn(0x7f245c037810 msgr2=0x7f245c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:43.937 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 1 --2- 172.21.15.114:0/1453320005 >> v2:172.21.15.149:6812/1185531135 conn(0x7f245c037810 0x7f245c039cd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f246c004470 tx=0x7f246c003700 comp rx=0 tx=0).stop 2024-09-16T05:07:43.937 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 10 monclient: shutdown 2024-09-16T05:07:43.937 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 1 -- 172.21.15.114:0/1453320005 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247c09ea50 msgr2=0x7f247c08efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 1 --2- 172.21.15.114:0/1453320005 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247c09ea50 0x7f247c08efd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2464009c50 tx=0x7f246400b510 comp rx=0 tx=0).stop 2024-09-16T05:07:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 1 -- 172.21.15.114:0/1453320005 shutdown_connections 2024-09-16T05:07:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 1 --2- 172.21.15.114:0/1453320005 >> v2:172.21.15.149:6812/1185531135 conn(0x7f245c037810 0x7f245c039cd0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 1 --2- 172.21.15.114:0/1453320005 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f247c13a710 0x7f247c08f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 1 --2- 172.21.15.114:0/1453320005 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247c09ea50 0x7f247c08efd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 1 --2- 172.21.15.114:0/1453320005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f247c09b780 0x7f247c098920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 1 -- 172.21.15.114:0/1453320005 >> 172.21.15.114:0/1453320005 conn(0x7f247c0a2070 msgr2=0x7f247c0a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 1 -- 172.21.15.114:0/1453320005 shutdown_connections 2024-09-16T05:07:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 1 -- 172.21.15.114:0/1453320005 wait complete. 2024-09-16T05:07:43.939 INFO:teuthology.misc.health.smithi114.stderr:2024-09-16T05:07:43.934+0000 7f2481620640 1 librados: shutdown 2024-09-16T05:07:43.949 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-16T05:07:43.950 INFO:tasks.ceph:Creating RBD pool 2024-09-16T05:07:43.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T05:07:44.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 10 monclient: get_monmap_and_config 2024-09-16T05:07:44.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 10 monclient: build_initial_monmap 2024-09-16T05:07:44.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 10 monclient: monmap: 2024-09-16T05:07:44.071 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:44.071 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:44.071 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:44.074073+0000 2024-09-16T05:07:44.071 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:44.074073+0000 2024-09-16T05:07:44.071 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:44.071 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:44.071 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:44.071 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:44.071 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:44.071 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:44.071 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:44.071 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:44.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 1 Processor -- start 2024-09-16T05:07:44.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 1 -- start start 2024-09-16T05:07:44.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 10 monclient: init 2024-09-16T05:07:44.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:44.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-16T05:07:44.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d1c0a0ec0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:44.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f7d1c09e1d0 0x7f7d1c09e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:44.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 10 monclient(hunting): picked mon.noname-c con 0x7f7d1c09e1d0 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:44.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d1c139640 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:44.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d1c09a760 con 0x7f7d1c139640 2024-09-16T05:07:44.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d1c0a1870 con 0x7f7d1c0a0ec0 2024-09-16T05:07:44.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:44.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f7d1c0994b0 con 0x7f7d1c09e1d0 2024-09-16T05:07:44.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:44.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d2468a640 10 monclient(hunting): authenticate will time out at 948.703514s 2024-09-16T05:07:44.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d22c27640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7d1c139640 0x7f7d1c13ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52526/0 (socket says 172.21.15.114:52526) 2024-09-16T05:07:44.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d22c27640 1 -- 172.21.15.114:0/2147731741 learned_addr learned my addr 172.21.15.114:0/2147731741 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:44.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d21424640 1 -- 172.21.15.114:0/2147731741 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2683751935 0 0) 0x7f7d1c09a760 con 0x7f7d1c139640 2024-09-16T05:07:44.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d21424640 10 monclient(hunting): _negotiate 2024-09-16T05:07:44.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d21424640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:44.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d21424640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:44.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d21424640 10 monclient(hunting): my global_id is 4200 2024-09-16T05:07:44.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d21424640 1 -- 172.21.15.114:0/2147731741 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d08003530 con 0x7f7d1c139640 2024-09-16T05:07:44.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.070+0000 7f7d21c25640 1 --2- 172.21.15.114:0/2147731741 >> v2:172.21.15.149:3300/0 conn(0x7f7d1c09e1d0 0x7f7d1c09e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:44.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient(hunting): get_auth_request con 0x7f7d1c09e1d0 auth_method 0 2024-09-16T05:07:44.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:44.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:44.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:44.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 -- 172.21.15.114:0/2147731741 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3788016718 0 0) 0x7f7d08003530 con 0x7f7d1c139640 2024-09-16T05:07:44.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 -- 172.21.15.114:0/2147731741 >> v2:172.21.15.149:3300/0 conn(0x7f7d1c09e1d0 msgr2=0x7f7d1c09e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:44.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 --2- 172.21.15.114:0/2147731741 >> v2:172.21.15.149:3300/0 conn(0x7f7d1c09e1d0 0x7f7d1c09e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:44.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 -- 172.21.15.114:0/2147731741 >> v1:172.21.15.149:6789/0 conn(0x7f7d1c0a0ec0 legacy=0x7f7d1c09dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:44.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: _finish_hunting 0 2024-09-16T05:07:44.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:07:44.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:44.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:07:44.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 -- 172.21.15.114:0/2147731741 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1c099790 con 0x7f7d1c139640 2024-09-16T05:07:44.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: _finish_auth 0 2024-09-16T05:07:44.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: _check_auth_tickets 2024-09-16T05:07:44.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:44.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 -- 172.21.15.114:0/2147731741 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f7d14001a20 con 0x7f7d1c139640 2024-09-16T05:07:44.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d2468a640 5 monclient: authenticate success, global_id 4200 2024-09-16T05:07:44.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d2468a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:44.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:44.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:07:44.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: dump: 2024-09-16T05:07:44.085 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:44.085 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:44.085 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:44.085 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:44.085 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:44.085 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:44.085 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:44.085 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:44.085 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:44.085 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:44.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:07:44.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:44.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 -- 172.21.15.114:0/2147731741 >> v1:172.21.15.114:6789/0 conn(0x7f7d1c139640 legacy=0x7f7d1c13ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:44.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:07:44.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 --2- 172.21.15.114:0/2147731741 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d080039c0 0x7f7d08003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:44.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient(hunting): picked mon.b con 0x7f7d080039c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:44.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 --2- 172.21.15.114:0/2147731741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d08004810 0x7f7d08004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:44.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient(hunting): picked mon.a con 0x7f7d08004810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:44.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 --2- 172.21.15.114:0/2147731741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d080052f0 0x7f7d0800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:44.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient(hunting): picked mon.c con 0x7f7d080052f0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:44.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:44.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 -- 172.21.15.114:0/2147731741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1c0994b0 con 0x7f7d08004810 2024-09-16T05:07:44.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:44.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 -- 172.21.15.114:0/2147731741 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0800c1c0 con 0x7f7d080039c0 2024-09-16T05:07:44.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:44.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 -- 172.21.15.114:0/2147731741 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7d0800c3c0 con 0x7f7d080052f0 2024-09-16T05:07:44.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:44.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:07:44.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d2468a640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:44.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 1 --2- 172.21.15.114:0/2147731741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d08004810 0x7f7d08004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:44.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient(hunting): get_auth_request con 0x7f7d08004810 auth_method 0 2024-09-16T05:07:44.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d22c27640 1 --2- 172.21.15.114:0/2147731741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d080052f0 0x7f7d0800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:44.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:44.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:44.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:44.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d22c27640 10 monclient(hunting): get_auth_request con 0x7f7d080052f0 auth_method 0 2024-09-16T05:07:44.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d22c27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:44.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d22c27640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:44.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d22c27640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:44.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:44.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:44.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:07:44.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d22c27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:44.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d22c27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:44.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d22c27640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:07:44.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d22426640 1 --2- 172.21.15.114:0/2147731741 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d080039c0 0x7f7d08003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:44.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d22426640 10 monclient(hunting): get_auth_request con 0x7f7d080039c0 auth_method 0 2024-09-16T05:07:44.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d22426640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:44.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d22426640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:44.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d22426640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:44.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient(hunting): handle_auth_done global_id 4200 payload 293 2024-09-16T05:07:44.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 1 -- 172.21.15.114:0/2147731741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d080052f0 msgr2=0x7f7d0800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:44.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 1 --2- 172.21.15.114:0/2147731741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d080052f0 0x7f7d0800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:44.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 1 -- 172.21.15.114:0/2147731741 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d080039c0 msgr2=0x7f7d08003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:44.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 1 --2- 172.21.15.114:0/2147731741 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d080039c0 0x7f7d08003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:44.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient: _finish_hunting 0 2024-09-16T05:07:44.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 1 monclient: found mon.a 2024-09-16T05:07:44.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:44.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:44.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 1 -- 172.21.15.114:0/2147731741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d08003530 con 0x7f7d08004810 2024-09-16T05:07:44.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient: _finish_auth 0 2024-09-16T05:07:44.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 10 monclient: _check_auth_tickets 2024-09-16T05:07:44.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:44.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d22c27640 1 --2- 172.21.15.114:0/2147731741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d080052f0 0x7f7d0800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:44.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d22c27640 1 -- 172.21.15.114:0/2147731741 reap_dead start 2024-09-16T05:07:44.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21c25640 1 --2- 172.21.15.114:0/2147731741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d08004810 0x7f7d08004c90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c00ba00 tx=0x7f7d0c01c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810c7aba306af7ae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:44.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 -- 172.21.15.114:0/2147731741 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c0b2070 con 0x7f7d08004810 2024-09-16T05:07:44.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:44.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:44.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: dump: 2024-09-16T05:07:44.090 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:44.090 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:44.090 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:44.090 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:44.090 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:44.090 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:44.090 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:44.090 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:44.090 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:44.090 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:44.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 -- 172.21.15.114:0/2147731741 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d0c0ad410 con 0x7f7d08004810 2024-09-16T05:07:44.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d2468a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:44.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:44.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 1 -- 172.21.15.114:0/2147731741 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c00b600 con 0x7f7d08004810 2024-09-16T05:07:44.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:44.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:44.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d21424640 10 monclient: dump: 2024-09-16T05:07:44.091 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:44.091 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:44.091 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:44.091 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:44.091 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:44.091 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:44.091 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:44.091 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:44.091 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:44.091 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:44.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d2468a640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:44.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d2468a640 4 set_mon_vals no callback set 2024-09-16T05:07:44.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d2468a640 10 monclient: shutdown 2024-09-16T05:07:44.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d2468a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:44.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d2468a640 1 -- 172.21.15.114:0/2147731741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d08004810 msgr2=0x7f7d08004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:44.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d2468a640 1 --2- 172.21.15.114:0/2147731741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d08004810 0x7f7d08004c90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c00ba00 tx=0x7f7d0c01c620 comp rx=0 tx=0).stop 2024-09-16T05:07:44.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d2468a640 1 -- 172.21.15.114:0/2147731741 shutdown_connections 2024-09-16T05:07:44.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d2468a640 1 --2- 172.21.15.114:0/2147731741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d08004810 0x7f7d08004c90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:44.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d2468a640 1 -- 172.21.15.114:0/2147731741 >> 172.21.15.114:0/2147731741 conn(0x7f7d1c0a1fb0 msgr2=0x7f7d1c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:44.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d2468a640 1 -- 172.21.15.114:0/2147731741 shutdown_connections 2024-09-16T05:07:44.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.074+0000 7f7d2468a640 1 -- 172.21.15.114:0/2147731741 wait complete. 2024-09-16T05:07:44.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient: build_initial_monmap 2024-09-16T05:07:44.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient: monmap: 2024-09-16T05:07:44.092 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:44.092 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:44.092 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:44.093 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:44.093 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:44.093 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:44.093 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:44.093 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:44.093 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:44.093 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:44.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 librados: starting msgr at 2024-09-16T05:07:44.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 librados: starting objecter 2024-09-16T05:07:44.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 Processor -- start 2024-09-16T05:07:44.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 -- start start 2024-09-16T05:07:44.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 librados: setting wanted keys 2024-09-16T05:07:44.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 librados: calling monclient init 2024-09-16T05:07:44.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient: init 2024-09-16T05:07:44.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:07:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d08004810 0x7f7d1c1cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient(hunting): picked mon.noname-c con 0x7f7d08004810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d1c1cff60 0x7f7d1c1d0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient(hunting): picked mon.noname-b con 0x7f7d1c1cff60 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d1c1cc810 0x7f7d1c1cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient(hunting): picked mon.noname-a con 0x7f7d1c1cc810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1c09c3b0 con 0x7f7d1c1cc810 2024-09-16T05:07:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1c09c230 con 0x7f7d1c1cff60 2024-09-16T05:07:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7d1c136e30 con 0x7f7d08004810 2024-09-16T05:07:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient(hunting): authenticate will time out at 948.709333s 2024-09-16T05:07:44.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d08004810 0x7f7d1c1cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:44.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22c27640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d1c1cc810 0x7f7d1c1cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:44.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d08004810 0x7f7d1c1cfa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55976/0 (socket says 172.21.15.114:55976) 2024-09-16T05:07:44.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 1 -- 172.21.15.114:0/1316778406 learned_addr learned my addr 172.21.15.114:0/1316778406 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:44.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 10 monclient(hunting): get_auth_request con 0x7f7d08004810 auth_method 0 2024-09-16T05:07:44.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:44.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:44.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:44.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22c27640 10 monclient(hunting): get_auth_request con 0x7f7d1c1cc810 auth_method 0 2024-09-16T05:07:44.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22c27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:44.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22c27640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:44.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22c27640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:44.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d21c25640 1 --2- 172.21.15.114:0/1316778406 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d1c1cff60 0x7f7d1c1d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:44.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:44.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:44.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:44.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22c27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:44.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22c27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:44.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22c27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:44.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d21c25640 10 monclient(hunting): get_auth_request con 0x7f7d1c1cff60 auth_method 0 2024-09-16T05:07:44.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d21c25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:44.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d21c25640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:44.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d21c25640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:44.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 10 monclient(hunting): handle_auth_done global_id 4193 payload 824 2024-09-16T05:07:44.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 1 -- 172.21.15.114:0/1316778406 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d1c1cff60 msgr2=0x7f7d1c1d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:44.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 1 --2- 172.21.15.114:0/1316778406 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d1c1cff60 0x7f7d1c1d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:44.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 1 -- 172.21.15.114:0/1316778406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d1c1cc810 msgr2=0x7f7d1c1cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:44.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 1 --2- 172.21.15.114:0/1316778406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d1c1cc810 0x7f7d1c1cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:44.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 10 monclient: _finish_hunting 0 2024-09-16T05:07:44.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 1 monclient: found mon.noname-c 2024-09-16T05:07:44.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:44.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:44.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 1 -- 172.21.15.114:0/1316778406 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1c1317f0 con 0x7f7d08004810 2024-09-16T05:07:44.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d21c25640 1 --2- 172.21.15.114:0/1316778406 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d1c1cff60 0x7f7d1c1d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:07:44.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22c27640 1 --2- 172.21.15.114:0/1316778406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d1c1cc810 0x7f7d1c1cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:44.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d22426640 1 --2- 172.21.15.114:0/1316778406 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d08004810 0x7f7d1c1cfa20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800e440 tx=0x7f7d1800e840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:44.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 10 client.?.objecter ms_handle_connect 0x7f7d08004810 2024-09-16T05:07:44.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:44.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 1 -- 172.21.15.114:0/1316778406 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d18019070 con 0x7f7d08004810 2024-09-16T05:07:44.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:44.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:44.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 10 monclient: dump: 2024-09-16T05:07:44.097 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:44.098 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:44.098 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:44.098 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:44.098 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:44.098 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:44.098 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:44.098 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:44.098 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:44.098 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:44.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 10 monclient: _finish_auth 0 2024-09-16T05:07:44.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:07:44.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:44.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 1 -- 172.21.15.114:0/1316778406 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d180052e0 con 0x7f7d08004810 2024-09-16T05:07:44.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 5 monclient: authenticate success, global_id 4193 2024-09-16T05:07:44.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient: _renew_subs 2024-09-16T05:07:44.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:44.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 -- 172.21.15.114:0/1316778406 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d1c072dc0 con 0x7f7d08004810 2024-09-16T05:07:44.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:44.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 1 -- 172.21.15.114:0/1316778406 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d18007250 con 0x7f7d08004810 2024-09-16T05:07:44.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:44.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:44.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 10 monclient: dump: 2024-09-16T05:07:44.099 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:44.099 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:44.099 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:44.099 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:44.099 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:44.099 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:44.099 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:44.099 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:44.100 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:44.100 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:44.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 client.4193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:44.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient: _renew_subs 2024-09-16T05:07:44.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:44.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 -- 172.21.15.114:0/1316778406 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d1c0a3e60 con 0x7f7d08004810 2024-09-16T05:07:44.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 librados: init done 2024-09-16T05:07:44.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d13fff640 4 set_mon_vals no callback set 2024-09-16T05:07:44.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:44.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:44.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:44.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 1 -- 172.21.15.114:0/1316778406 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7d1801c070 con 0x7f7d08004810 2024-09-16T05:07:44.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d2468a640 1 -- 172.21.15.114:0/1316778406 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d1c1371c0 con 0x7f7d08004810 2024-09-16T05:07:44.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 1 --2- 172.21.15.114:0/1316778406 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7d00037840 0x7f7d00039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:44.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 1 -- 172.21.15.114:0/1316778406 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 3153+0+0 (secure 0 0 0) 0x7f7d18047610 con 0x7f7d08004810 2024-09-16T05:07:44.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 10 client.4193.objecter ms_dispatch 0x7f7d1c000ba0 osd_map(14..14 src has 1..14) 2024-09-16T05:07:44.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 3 client.4193.objecter handle_osd_map got epochs [14,14] > 0 2024-09-16T05:07:44.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 3 client.4193.objecter handle_osd_map decoding full epoch 14 2024-09-16T05:07:44.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d12ffd640 20 client.4193.objecter dump_active .. 0 homeless 2024-09-16T05:07:44.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d21c25640 1 --2- 172.21.15.114:0/1316778406 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7d00037840 0x7f7d00039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:44.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.078+0000 7f7d21c25640 10 monclient: get_auth_request con 0x7f7d00037840 auth_method 0 2024-09-16T05:07:44.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.082+0000 7f7d12ffd640 10 client.4193.objecter ms_handle_connect 0x7f7d00037840 2024-09-16T05:07:44.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.082+0000 7f7d21c25640 1 --2- 172.21.15.114:0/1316778406 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7d00037840 0x7f7d00039d00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0ac000 tx=0x7f7d0c0af040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:44.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.082+0000 7f7d12ffd640 1 -- 172.21.15.114:0/1316778406 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7d1c072dc0 con 0x7f7d08004810 2024-09-16T05:07:44.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.082+0000 7f7d12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:44.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.082+0000 7f7d12ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:44.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.238+0000 7f7d2468a640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T05:07:44.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.238+0000 7f7d2468a640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T05:07:44.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.238+0000 7f7d2468a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:44.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:44.238+0000 7f7d2468a640 1 -- 172.21.15.114:0/1316778406 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f7d1c0686f0 con 0x7f7d08004810 2024-09-16T05:07:45.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.230+0000 7f7d12ffd640 1 -- 172.21.15.114:0/1316778406 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v15) ==== 109+0+0 (secure 0 0 0) 0x7f7d180073f0 con 0x7f7d08004810 2024-09-16T05:07:45.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.230+0000 7f7d12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T05:07:45.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.230+0000 7f7d12ffd640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-16T05:07:45.233 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'rbd' created 2024-09-16T05:07:45.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 10 librados: watch_flush enter 2024-09-16T05:07:45.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 10 librados: watch_flush exit 2024-09-16T05:07:45.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 20 client.4193.objecter shutdown clearing up homeless session... 2024-09-16T05:07:45.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 10 client.4193.objecter successfully canceled tick 2024-09-16T05:07:45.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 1 -- 172.21.15.114:0/1316778406 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7d00037840 msgr2=0x7f7d00039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:45.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 1 --2- 172.21.15.114:0/1316778406 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7d00037840 0x7f7d00039d00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0ac000 tx=0x7f7d0c0af040 comp rx=0 tx=0).stop 2024-09-16T05:07:45.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 10 monclient: shutdown 2024-09-16T05:07:45.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:45.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 1 -- 172.21.15.114:0/1316778406 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d08004810 msgr2=0x7f7d1c1cfa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:45.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 1 --2- 172.21.15.114:0/1316778406 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d08004810 0x7f7d1c1cfa20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800e440 tx=0x7f7d1800e840 comp rx=0 tx=0).stop 2024-09-16T05:07:45.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 1 -- 172.21.15.114:0/1316778406 shutdown_connections 2024-09-16T05:07:45.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 1 --2- 172.21.15.114:0/1316778406 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7d00037840 0x7f7d00039d00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:45.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 1 --2- 172.21.15.114:0/1316778406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d1c1cc810 0x7f7d1c1cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:45.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 1 --2- 172.21.15.114:0/1316778406 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d1c1cff60 0x7f7d1c1d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:45.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 1 --2- 172.21.15.114:0/1316778406 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d08004810 0x7f7d1c1cfa20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:45.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.234+0000 7f7d2468a640 1 -- 172.21.15.114:0/1316778406 >> 172.21.15.114:0/1316778406 conn(0x7f7d1c0a1fb0 msgr2=0x7f7d1c06f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:45.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.238+0000 7f7d2468a640 1 -- 172.21.15.114:0/1316778406 shutdown_connections 2024-09-16T05:07:45.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.238+0000 7f7d2468a640 1 -- 172.21.15.114:0/1316778406 wait complete. 2024-09-16T05:07:45.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.238+0000 7f7d2468a640 1 librados: shutdown 2024-09-16T05:07:45.258 DEBUG:teuthology.orchestra.run.smithi114:> rbd --cluster ceph pool init rbd 2024-09-16T05:07:45.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 10 monclient: get_monmap_and_config 2024-09-16T05:07:45.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 10 monclient: build_initial_monmap 2024-09-16T05:07:45.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 10 monclient: monmap: 2024-09-16T05:07:45.305 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:45.305 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:45.305 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:45.308457+0000 2024-09-16T05:07:45.305 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:45.308457+0000 2024-09-16T05:07:45.305 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:45.306 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:45.306 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:45.306 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:45.306 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:45.306 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:45.306 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:45.306 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:45.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 1 Processor -- start 2024-09-16T05:07:45.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 1 -- start start 2024-09-16T05:07:45.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 10 monclient: init 2024-09-16T05:07:45.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 10 monclient: _reopen_session rank -1 2024-09-16T05:07:45.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T05:07:45.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 1 --2- >> v2:172.21.15.149:3300/0 conn(0x56506a553df0 0x56506a4b90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 10 monclient(hunting): picked mon.noname-c con 0x56506a553df0 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 1 --2- >> v2:172.21.15.114:3300/0 conn(0x56506a4b7a90 0x56506a5541d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 10 monclient(hunting): picked mon.noname-a con 0x56506a4b7a90 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56506a4c1b10 0x56506a4c1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 10 monclient(hunting): picked mon.noname-b con 0x56506a4c1b10 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x56506a332e20 con 0x56506a4b7a90 2024-09-16T05:07:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x56506a38d840 con 0x56506a553df0 2024-09-16T05:07:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x56506a389800 con 0x56506a4c1b10 2024-09-16T05:07:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 10 monclient(hunting): _renew_subs 2024-09-16T05:07:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8de84f80 10 monclient(hunting): authenticate will time out at 949.937777s 2024-09-16T05:07:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56506a4c1b10 0x56506a4c1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56506a4c1b10 0x56506a4c1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55980/0 (socket says 172.21.15.114:55980) 2024-09-16T05:07:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 1 -- 172.21.15.114:0/2833133389 learned_addr learned my addr 172.21.15.114:0/2833133389 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 10 monclient(hunting): get_auth_request con 0x56506a4c1b10 auth_method 0 2024-09-16T05:07:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:45.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:45.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa87fff640 1 --2- 172.21.15.114:0/2833133389 >> v2:172.21.15.114:3300/0 conn(0x56506a4b7a90 0x56506a5541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:45.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:45.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:45.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:45.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa87fff640 10 monclient(hunting): get_auth_request con 0x56506a4b7a90 auth_method 0 2024-09-16T05:07:45.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:45.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa87fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:45.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:45.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:45.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:45.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8c998640 1 --2- 172.21.15.114:0/2833133389 >> v2:172.21.15.149:3300/0 conn(0x56506a553df0 0x56506a4b90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:45.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:45.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 10 monclient(hunting): handle_auth_done global_id 4196 payload 274 2024-09-16T05:07:45.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 1 -- 172.21.15.114:0/2833133389 >> v2:172.21.15.149:3300/0 conn(0x56506a553df0 msgr2=0x56506a4b90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:45.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 1 --2- 172.21.15.114:0/2833133389 >> v2:172.21.15.149:3300/0 conn(0x56506a553df0 0x56506a4b90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:45.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 1 -- 172.21.15.114:0/2833133389 >> v2:172.21.15.114:3300/0 conn(0x56506a4b7a90 msgr2=0x56506a5541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:45.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 1 --2- 172.21.15.114:0/2833133389 >> v2:172.21.15.114:3300/0 conn(0x56506a4b7a90 0x56506a5541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:45.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 10 monclient: _finish_hunting 0 2024-09-16T05:07:45.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 1 monclient: found mon.noname-b 2024-09-16T05:07:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8dbfa640 1 -- 172.21.15.114:0/2833133389 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56506a33dea0 con 0x56506a4c1b10 2024-09-16T05:07:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8c998640 10 monclient: get_auth_request con 0x56506a553df0 auth_method 0 2024-09-16T05:07:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.306+0000 7ffa8c998640 1 --2- 172.21.15.114:0/2833133389 >> v2:172.21.15.149:3300/0 conn(0x56506a553df0 0x56506a4b90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:07:45.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8dbfa640 1 --2- 172.21.15.114:0/2833133389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56506a4c1b10 0x56506a4c1fa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x56506a4a1870 tx=0x7ffa8001b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d51e1587ffd2576 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:45.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa877fe640 1 -- 172.21.15.114:0/2833133389 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa800ab410 con 0x56506a4c1b10 2024-09-16T05:07:45.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa877fe640 10 monclient: dump: 2024-09-16T05:07:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:45.319 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa877fe640 10 monclient: _finish_auth 0 2024-09-16T05:07:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa877fe640 10 monclient: _check_auth_tickets 2024-09-16T05:07:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa877fe640 1 -- 172.21.15.114:0/2833133389 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa800a7070 con 0x56506a4c1b10 2024-09-16T05:07:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 5 monclient: authenticate success, global_id 4196 2024-09-16T05:07:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa877fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa877fe640 1 -- 172.21.15.114:0/2833133389 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa800ab830 con 0x56506a4c1b10 2024-09-16T05:07:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient: get_monmap_and_config success 2024-09-16T05:07:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 4 set_mon_vals no callback set 2024-09-16T05:07:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa877fe640 10 monclient: dump: 2024-09-16T05:07:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:45.321 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient: shutdown 2024-09-16T05:07:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2833133389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56506a4c1b10 msgr2=0x56506a4c1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 --2- 172.21.15.114:0/2833133389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56506a4c1b10 0x56506a4c1fa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x56506a4a1870 tx=0x7ffa8001b040 comp rx=0 tx=0).stop 2024-09-16T05:07:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2833133389 shutdown_connections 2024-09-16T05:07:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 --2- 172.21.15.114:0/2833133389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56506a4c1b10 0x56506a4c1fa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 --2- 172.21.15.114:0/2833133389 >> v2:172.21.15.114:3300/0 conn(0x56506a4b7a90 0x56506a5541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 --2- 172.21.15.114:0/2833133389 >> v2:172.21.15.149:3300/0 conn(0x56506a553df0 0x56506a4b90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2833133389 >> 172.21.15.114:0/2833133389 conn(0x56506a4c32c0 msgr2=0x56506a4a0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2833133389 shutdown_connections 2024-09-16T05:07:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2833133389 wait complete. 2024-09-16T05:07:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient: build_initial_monmap 2024-09-16T05:07:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient: monmap: 2024-09-16T05:07:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:45.323 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:45.323 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:45.323 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:45.323 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:45.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 librados: starting msgr at 2024-09-16T05:07:45.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 librados: starting objecter 2024-09-16T05:07:45.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 Processor -- start 2024-09-16T05:07:45.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 -- start start 2024-09-16T05:07:45.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 librados: setting wanted keys 2024-09-16T05:07:45.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 librados: calling monclient init 2024-09-16T05:07:45.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient: init 2024-09-16T05:07:45.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient: _reopen_session rank -1 2024-09-16T05:07:45.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:07:45.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56506a4b7a90 0x56506a553440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:45.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient(hunting): picked mon.noname-c con 0x56506a4b7a90 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:45.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56506a4c1b10 0x56506a4bcc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:45.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient(hunting): picked mon.noname-b con 0x56506a4c1b10 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:45.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56506a553df0 0x56506a4bd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:45.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient(hunting): picked mon.noname-a con 0x56506a553df0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:45.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:45.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56506a4a15d0 con 0x56506a553df0 2024-09-16T05:07:45.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:45.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x56506a4a1450 con 0x56506a4c1b10 2024-09-16T05:07:45.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:45.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x56506a554710 con 0x56506a4b7a90 2024-09-16T05:07:45.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient(hunting): _renew_subs 2024-09-16T05:07:45.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient(hunting): authenticate will time out at 949.943052s 2024-09-16T05:07:45.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56506a4b7a90 0x56506a553440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8dbfa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56506a553df0 0x56506a4bd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56506a4b7a90 0x56506a553440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55990/0 (socket says 172.21.15.114:55990) 2024-09-16T05:07:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 1 -- 172.21.15.114:0/2040185577 learned_addr learned my addr 172.21.15.114:0/2040185577 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 10 monclient(hunting): get_auth_request con 0x56506a4b7a90 auth_method 0 2024-09-16T05:07:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8dbfa640 10 monclient(hunting): get_auth_request con 0x56506a553df0 auth_method 0 2024-09-16T05:07:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8dbfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8dbfa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8dbfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8dbfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8dbfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8dbfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa87fff640 1 --2- 172.21.15.114:0/2040185577 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56506a4c1b10 0x56506a4bcc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa87fff640 10 monclient(hunting): get_auth_request con 0x56506a4c1b10 auth_method 0 2024-09-16T05:07:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa87fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 10 monclient(hunting): handle_auth_done global_id 4199 payload 824 2024-09-16T05:07:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 1 -- 172.21.15.114:0/2040185577 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56506a4c1b10 msgr2=0x56506a4bcc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 1 --2- 172.21.15.114:0/2040185577 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56506a4c1b10 0x56506a4bcc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 1 -- 172.21.15.114:0/2040185577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56506a553df0 msgr2=0x56506a4bd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 1 --2- 172.21.15.114:0/2040185577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56506a553df0 0x56506a4bd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:45.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 10 monclient: _finish_hunting 0 2024-09-16T05:07:45.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 1 monclient: found mon.noname-c 2024-09-16T05:07:45.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:45.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:45.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 1 -- 172.21.15.114:0/2040185577 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56506a324180 con 0x56506a4b7a90 2024-09-16T05:07:45.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8c998640 1 --2- 172.21.15.114:0/2040185577 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56506a4b7a90 0x56506a553440 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c006640 tx=0x7ffa7c006b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:45.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa84ff9640 10 client.?.objecter ms_handle_connect 0x56506a4b7a90 2024-09-16T05:07:45.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa84ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:45.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa84ff9640 1 -- 172.21.15.114:0/2040185577 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c007400 con 0x56506a4b7a90 2024-09-16T05:07:45.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:45.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa84ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:45.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa84ff9640 10 monclient: dump: 2024-09-16T05:07:45.327 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:45.327 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:45.328 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:45.328 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:45.328 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:45.328 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:45.328 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:45.328 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:45.328 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:45.328 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:45.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa84ff9640 10 monclient: _finish_auth 0 2024-09-16T05:07:45.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa84ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:07:45.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:45.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa84ff9640 1 -- 172.21.15.114:0/2040185577 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa7c0075a0 con 0x56506a4b7a90 2024-09-16T05:07:45.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 5 monclient: authenticate success, global_id 4199 2024-09-16T05:07:45.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient: _renew_subs 2024-09-16T05:07:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2040185577 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x56506a30e990 con 0x56506a4b7a90 2024-09-16T05:07:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa84ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa84ff9640 1 -- 172.21.15.114:0/2040185577 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c007950 con 0x56506a4b7a90 2024-09-16T05:07:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa84ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa84ff9640 10 monclient: dump: 2024-09-16T05:07:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:45.330 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:45.330 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:45.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.310+0000 7ffa867fc640 4 set_mon_vals no callback set 2024-09-16T05:07:45.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa8de84f80 10 client.4199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:45.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa8de84f80 10 monclient: _renew_subs 2024-09-16T05:07:45.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa8de84f80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:45.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2040185577 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x56506a4bf3c0 con 0x56506a4b7a90 2024-09-16T05:07:45.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa8de84f80 1 librados: init done 2024-09-16T05:07:45.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa8de84f80 10 librados: wait_for_osdmap waiting 2024-09-16T05:07:45.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa84ff9640 1 -- 172.21.15.114:0/2040185577 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ffa7c007af0 con 0x56506a4b7a90 2024-09-16T05:07:45.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa84ff9640 1 --2- 172.21.15.114:0/2040185577 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffa6c035940 0x7ffa6c037e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:45.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa84ff9640 1 -- 172.21.15.114:0/2040185577 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3495+0+0 (secure 0 0 0) 0x7ffa7c047170 con 0x56506a4b7a90 2024-09-16T05:07:45.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa84ff9640 10 client.4199.objecter ms_dispatch 0x56506a3a47b0 osd_map(15..15 src has 1..15) 2024-09-16T05:07:45.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa84ff9640 3 client.4199.objecter handle_osd_map got epochs [15,15] > 0 2024-09-16T05:07:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa84ff9640 3 client.4199.objecter handle_osd_map decoding full epoch 15 2024-09-16T05:07:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa84ff9640 20 client.4199.objecter dump_active .. 0 homeless 2024-09-16T05:07:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa8de84f80 10 librados: wait_for_osdmap done waiting 2024-09-16T05:07:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa8de84f80 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T05:07:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa8de84f80 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T05:07:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa8de84f80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2040185577 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x56506a4bf140 con 0x56506a4b7a90 2024-09-16T05:07:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa87fff640 1 --2- 172.21.15.114:0/2040185577 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffa6c035940 0x7ffa6c037e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa87fff640 10 monclient: get_auth_request con 0x7ffa6c035940 auth_method 0 2024-09-16T05:07:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa84ff9640 10 client.4199.objecter ms_handle_connect 0x7ffa6c035940 2024-09-16T05:07:45.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:45.314+0000 7ffa87fff640 1 --2- 172.21.15.114:0/2040185577 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffa6c035940 0x7ffa6c037e00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x56506a4be460 tx=0x7ffa7400c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:46.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa84ff9640 1 -- 172.21.15.114:0/2040185577 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v16) ==== 141+0+0 (secure 0 0 0) 0x7ffa7c01b400 con 0x56506a4b7a90 2024-09-16T05:07:46.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa84ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T05:07:46.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa84ff9640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-16T05:07:46.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:46.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2040185577 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffa800a7070 con 0x56506a4b7a90 2024-09-16T05:07:46.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa84ff9640 1 -- 172.21.15.114:0/2040185577 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7ffa7c007de0 con 0x56506a4b7a90 2024-09-16T05:07:46.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa84ff9640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-16T05:07:46.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa867fc640 10 client.4199.objecter _get_latest_version latest 16, waiting 2024-09-16T05:07:46.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa867fc640 10 client.4199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:46.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa867fc640 10 monclient: _renew_subs 2024-09-16T05:07:46.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa867fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:46.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa867fc640 1 -- 172.21.15.114:0/2040185577 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=16}) -- 0x7ffa64003ca0 con 0x56506a4b7a90 2024-09-16T05:07:46.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa84ff9640 1 -- 172.21.15.114:0/2040185577 <== mon.2 v2:172.21.15.114:3301/0 8 ==== osd_map(16..16 src has 1..16) ==== 674+0+0 (secure 0 0 0) 0x7ffa7c011da0 con 0x56506a4b7a90 2024-09-16T05:07:46.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa84ff9640 10 client.4199.objecter ms_dispatch 0x56506a3a47b0 osd_map(16..16 src has 1..16) 2024-09-16T05:07:46.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa84ff9640 3 client.4199.objecter handle_osd_map got epochs [16,16] > 15 2024-09-16T05:07:46.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa84ff9640 3 client.4199.objecter handle_osd_map decoding incremental epoch 16 2024-09-16T05:07:46.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa84ff9640 20 client.4199.objecter dump_active .. 0 homeless 2024-09-16T05:07:46.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 10 librados: create oid=rbd_trash nspace= 2024-09-16T05:07:46.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 10 client.4199.objecter _op_submit op 0x56506a56c640 2024-09-16T05:07:46.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 20 client.4199.objecter _calc_target epoch 16 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T05:07:46.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 20 client.4199.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-09-16T05:07:46.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 10 client.4199.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [1,2] primary 1 2024-09-16T05:07:46.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 1 --2- 172.21.15.114:0/2040185577 >> v2:172.21.15.114:6800/2903720358 conn(0x56506a56d000 0x56506a56f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:46.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 20 client.4199.objecter _get_session s=0x56506a552ad0 osd=1 3 2024-09-16T05:07:46.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 10 client.4199.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.1 2024-09-16T05:07:46.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 20 client.4199.objecter get_session s=0x56506a552ad0 osd=1 3 2024-09-16T05:07:46.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 15 client.4199.objecter _session_op_assign 1 1 2024-09-16T05:07:46.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 15 client.4199.objecter _send_op 1 to 2.3 on osd.1 2024-09-16T05:07:46.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2040185577 --> v2:172.21.15.114:6800/2903720358 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) -- 0x56506a56f930 con 0x56506a56d000 2024-09-16T05:07:46.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 20 client.4199.objecter put_session s=0x56506a552ad0 osd=1 4 2024-09-16T05:07:46.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8de84f80 5 client.4199.objecter 1 in flight 2024-09-16T05:07:46.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8dbfa640 1 --2- 172.21.15.114:0/2040185577 >> v2:172.21.15.114:6800/2903720358 conn(0x56506a56d000 0x56506a56f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:46.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.242+0000 7ffa8dbfa640 10 monclient: get_auth_request con 0x56506a56d000 auth_method 0 2024-09-16T05:07:46.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.246+0000 7ffa84ff9640 10 client.4199.objecter ms_handle_connect 0x56506a56d000 2024-09-16T05:07:46.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.246+0000 7ffa8dbfa640 1 --2- 172.21.15.114:0/2040185577 >> v2:172.21.15.114:6800/2903720358 conn(0x56506a56d000 0x56506a56f3f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.250+0000 7ffa8dbfa640 1 -- 172.21.15.114:0/2040185577 <== osd.1 v2:172.21.15.114:6800/2903720358 1 ==== osd_op_reply(1 rbd_trash [create] v16'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7ffa800b6420 con 0x56506a56d000 2024-09-16T05:07:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.250+0000 7ffa8dbfa640 10 client.4199.objecter ms_dispatch 0x56506a3a47b0 osd_op_reply(1 rbd_trash [create] v16'1 uv1 ondisk = 0) 2024-09-16T05:07:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.250+0000 7ffa8dbfa640 10 client.4199.objecter in handle_osd_op_reply 2024-09-16T05:07:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.250+0000 7ffa8dbfa640 7 client.4199.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-09-16T05:07:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.250+0000 7ffa8dbfa640 10 client.4199.objecter op 0 rval 0 len 0 2024-09-16T05:07:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.250+0000 7ffa8dbfa640 15 client.4199.objecter handle_osd_op_reply completed tid 1 2024-09-16T05:07:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.250+0000 7ffa8dbfa640 15 client.4199.objecter _finish_op 1 2024-09-16T05:07:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.250+0000 7ffa8dbfa640 20 client.4199.objecter put_session s=0x56506a552ad0 osd=1 4 2024-09-16T05:07:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.250+0000 7ffa8dbfa640 15 client.4199.objecter _session_op_remove 1 1 2024-09-16T05:07:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.250+0000 7ffa8dbfa640 5 client.4199.objecter 0 in flight 2024-09-16T05:07:46.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.250+0000 7ffa8de84f80 10 librados: Objecter returned from create r=0 2024-09-16T05:07:46.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 10 client.4199.objecter _op_submit op 0x56506a5c15c0 2024-09-16T05:07:46.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 20 client.4199.objecter _calc_target epoch 16 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T05:07:46.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 20 client.4199.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T05:07:46.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 10 client.4199.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [0,4] primary 0 2024-09-16T05:07:46.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 1 --2- 172.21.15.114:0/2040185577 >> v2:172.21.15.114:6808/269070361 conn(0x56506a5c2170 0x56506a5c4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:46.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 20 client.4199.objecter _get_session s=0x56506a5c1af0 osd=0 3 2024-09-16T05:07:46.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 10 client.4199.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.0 2024-09-16T05:07:46.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 20 client.4199.objecter get_session s=0x56506a5c1af0 osd=0 3 2024-09-16T05:07:46.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 15 client.4199.objecter _session_op_assign 0 2 2024-09-16T05:07:46.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 15 client.4199.objecter _send_op 2 to 2.2 on osd.0 2024-09-16T05:07:46.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2040185577 --> v2:172.21.15.114:6808/269070361 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16) -- 0x56506a5c4bb0 con 0x56506a5c2170 2024-09-16T05:07:46.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 20 client.4199.objecter put_session s=0x56506a5c1af0 osd=0 4 2024-09-16T05:07:46.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 5 client.4199.objecter 1 in flight 2024-09-16T05:07:46.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8c998640 1 --2- 172.21.15.114:0/2040185577 >> v2:172.21.15.114:6808/269070361 conn(0x56506a5c2170 0x56506a5c4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:46.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8c998640 10 monclient: get_auth_request con 0x56506a5c2170 auth_method 0 2024-09-16T05:07:46.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa84ff9640 10 client.4199.objecter ms_handle_connect 0x56506a5c2170 2024-09-16T05:07:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8c998640 1 --2- 172.21.15.114:0/2040185577 >> v2:172.21.15.114:6808/269070361 conn(0x56506a5c2170 0x56506a5c4600 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8c998640 1 -- 172.21.15.114:0/2040185577 <== osd.0 v2:172.21.15.114:6808/269070361 1 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7ffa7c01bce0 con 0x56506a5c2170 2024-09-16T05:07:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8c998640 10 client.4199.objecter ms_dispatch 0x56506a3a47b0 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T05:07:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8c998640 10 client.4199.objecter in handle_osd_op_reply 2024-09-16T05:07:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8c998640 7 client.4199.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-09-16T05:07:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8c998640 10 client.4199.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T05:07:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8c998640 10 client.4199.objecter op 0 rval 0 len 0 2024-09-16T05:07:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8c998640 15 client.4199.objecter handle_osd_op_reply completed tid 2 2024-09-16T05:07:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8c998640 15 client.4199.objecter _finish_op 2 2024-09-16T05:07:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8c998640 20 client.4199.objecter put_session s=0x56506a5c1af0 osd=0 4 2024-09-16T05:07:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8c998640 15 client.4199.objecter _session_op_remove 0 2 2024-09-16T05:07:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8c998640 5 client.4199.objecter 0 in flight 2024-09-16T05:07:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 10 client.4199.objecter _op_submit op 0x56506a5c0750 2024-09-16T05:07:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 20 client.4199.objecter _calc_target epoch 16 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T05:07:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 20 client.4199.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T05:07:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 10 client.4199.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [0,4] primary 0 2024-09-16T05:07:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 20 client.4199.objecter _get_session s=0x56506a5c1af0 osd=0 3 2024-09-16T05:07:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 10 client.4199.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.0 2024-09-16T05:07:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 20 client.4199.objecter get_session s=0x56506a5c1af0 osd=0 3 2024-09-16T05:07:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 15 client.4199.objecter _session_op_assign 0 3 2024-09-16T05:07:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 15 client.4199.objecter _send_op 3 to 2.2 on osd.0 2024-09-16T05:07:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2040185577 --> v2:172.21.15.114:6808/269070361 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16) -- 0x56506a5c0c00 con 0x56506a5c2170 2024-09-16T05:07:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 20 client.4199.objecter put_session s=0x56506a5c1af0 osd=0 4 2024-09-16T05:07:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.254+0000 7ffa8de84f80 5 client.4199.objecter 1 in flight 2024-09-16T05:07:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.258+0000 7ffa8c998640 1 -- 172.21.15.114:0/2040185577 <== osd.0 v2:172.21.15.114:6808/269070361 2 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7ffa7c01bce0 con 0x56506a5c2170 2024-09-16T05:07:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.258+0000 7ffa8c998640 10 client.4199.objecter ms_dispatch 0x56506a3a47b0 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T05:07:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.258+0000 7ffa8c998640 10 client.4199.objecter in handle_osd_op_reply 2024-09-16T05:07:46.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.258+0000 7ffa8c998640 7 client.4199.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-09-16T05:07:46.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.258+0000 7ffa8c998640 10 client.4199.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T05:07:46.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.258+0000 7ffa8c998640 10 client.4199.objecter op 0 rval 0 len 0 2024-09-16T05:07:46.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.258+0000 7ffa8c998640 15 client.4199.objecter handle_osd_op_reply completed tid 3 2024-09-16T05:07:46.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.258+0000 7ffa8c998640 15 client.4199.objecter _finish_op 3 2024-09-16T05:07:46.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.258+0000 7ffa8c998640 20 client.4199.objecter put_session s=0x56506a5c1af0 osd=0 4 2024-09-16T05:07:46.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.258+0000 7ffa8c998640 15 client.4199.objecter _session_op_remove 0 3 2024-09-16T05:07:46.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.258+0000 7ffa8c998640 5 client.4199.objecter 0 in flight 2024-09-16T05:07:46.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.258+0000 7ffa86ffd640 10 client.4199.objecter allocate_selfmanaged_snap; pool: 2 2024-09-16T05:07:46.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.258+0000 7ffa86ffd640 10 client.4199.objecter pool_op_submit 4 2024-09-16T05:07:46.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.258+0000 7ffa86ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:46.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:46.258+0000 7ffa86ffd640 1 -- 172.21.15.114:0/2040185577 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7ffa680030e0 con 0x56506a4b7a90 2024-09-16T05:07:47.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa84ff9640 1 -- 172.21.15.114:0/2040185577 <== mon.2 v2:172.21.15.114:3301/0 9 ==== pool_op_reply(tid 4 (0) Success v17) ==== 55+0+0 (secure 0 0 0) 0x7ffa7c00b6d0 con 0x56506a4b7a90 2024-09-16T05:07:47.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa84ff9640 10 client.4199.objecter ms_dispatch 0x56506a3a47b0 pool_op_reply(tid 4 (0) Success v17) 2024-09-16T05:07:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa84ff9640 10 client.4199.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v17) 2024-09-16T05:07:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa84ff9640 10 client.4199.objecter have request 4 at 0x7ffa68003070 Op: create unmanaged snap 2024-09-16T05:07:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa84ff9640 20 client.4199.objecter waiting for client to reach epoch 17 before calling back 2024-09-16T05:07:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa84ff9640 10 client.4199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa84ff9640 10 monclient: _renew_subs 2024-09-16T05:07:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa84ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa84ff9640 1 -- 172.21.15.114:0/2040185577 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=17}) -- 0x7ffa6c03b860 con 0x56506a4b7a90 2024-09-16T05:07:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa84ff9640 10 client.4199.objecter done 2024-09-16T05:07:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa84ff9640 1 -- 172.21.15.114:0/2040185577 <== mon.2 v2:172.21.15.114:3301/0 10 ==== osd_map(17..17 src has 1..17) ==== 626+0+0 (secure 0 0 0) 0x7ffa7c00b8b0 con 0x56506a4b7a90 2024-09-16T05:07:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa84ff9640 10 client.4199.objecter ms_dispatch 0x56506a3a47b0 osd_map(17..17 src has 1..17) 2024-09-16T05:07:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa84ff9640 3 client.4199.objecter handle_osd_map got epochs [17,17] > 16 2024-09-16T05:07:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa84ff9640 3 client.4199.objecter handle_osd_map decoding incremental epoch 17 2024-09-16T05:07:47.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa84ff9640 20 client.4199.objecter dump_active .. 0 homeless 2024-09-16T05:07:47.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa867fc640 20 librados: queue_aio_write 0x56506a5bcb50 completion 0x56506a34da30 write_seq 1 2024-09-16T05:07:47.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa867fc640 10 client.4199.objecter _op_submit op 0x7ffa64006390 2024-09-16T05:07:47.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa867fc640 20 client.4199.objecter _calc_target epoch 17 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T05:07:47.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa867fc640 20 client.4199.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T05:07:47.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa867fc640 10 client.4199.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [0,4] primary 0 2024-09-16T05:07:47.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa867fc640 20 client.4199.objecter _get_session s=0x56506a5c1af0 osd=0 3 2024-09-16T05:07:47.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa867fc640 10 client.4199.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.0 2024-09-16T05:07:47.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa867fc640 20 client.4199.objecter get_session s=0x56506a5c1af0 osd=0 3 2024-09-16T05:07:47.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa867fc640 15 client.4199.objecter _session_op_assign 0 5 2024-09-16T05:07:47.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa867fc640 15 client.4199.objecter _send_op 5 to 2.2 on osd.0 2024-09-16T05:07:47.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa867fc640 1 -- 172.21.15.114:0/2040185577 --> v2:172.21.15.114:6808/269070361 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x7ffa640068c0 con 0x56506a5c2170 2024-09-16T05:07:47.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa867fc640 20 client.4199.objecter put_session s=0x56506a5c1af0 osd=0 4 2024-09-16T05:07:47.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.242+0000 7ffa867fc640 5 client.4199.objecter 1 in flight 2024-09-16T05:07:47.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa8c998640 1 -- 172.21.15.114:0/2040185577 <== osd.0 v2:172.21.15.114:6808/269070361 3 ==== osd_op_reply(5 rbd_info [create,write 0~8] v17'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7ffa7c01bce0 con 0x56506a5c2170 2024-09-16T05:07:47.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa8c998640 10 client.4199.objecter ms_dispatch 0x56506a3a47b0 osd_op_reply(5 rbd_info [create,write 0~8] v17'1 uv1 ondisk = 0) 2024-09-16T05:07:47.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa8c998640 10 client.4199.objecter in handle_osd_op_reply 2024-09-16T05:07:47.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa8c998640 7 client.4199.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-09-16T05:07:47.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa8c998640 10 client.4199.objecter op 0 rval 0 len 0 2024-09-16T05:07:47.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa8c998640 10 client.4199.objecter op 1 rval 0 len 0 2024-09-16T05:07:47.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa8c998640 15 client.4199.objecter handle_osd_op_reply completed tid 5 2024-09-16T05:07:47.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa8c998640 15 client.4199.objecter _finish_op 5 2024-09-16T05:07:47.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa8c998640 20 client.4199.objecter put_session s=0x56506a5c1af0 osd=0 4 2024-09-16T05:07:47.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa8c998640 15 client.4199.objecter _session_op_remove 0 5 2024-09-16T05:07:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa8c998640 5 client.4199.objecter 0 in flight 2024-09-16T05:07:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa8c998640 20 librados: complete_aio_write 0x56506a34da30 2024-09-16T05:07:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa86ffd640 10 client.4199.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-09-16T05:07:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa86ffd640 10 client.4199.objecter pool_op_submit 6 2024-09-16T05:07:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa86ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:47.254+0000 7ffa86ffd640 1 -- 172.21.15.114:0/2040185577 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v17) -- 0x7ffa68004530 con 0x56506a4b7a90 2024-09-16T05:07:48.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.262+0000 7ffa84ff9640 1 -- 172.21.15.114:0/2040185577 <== mon.2 v2:172.21.15.114:3301/0 11 ==== pool_op_reply(tid 6 (0) Success v18) ==== 43+0+0 (secure 0 0 0) 0x7ffa7c036940 con 0x56506a4b7a90 2024-09-16T05:07:48.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.262+0000 7ffa84ff9640 10 client.4199.objecter ms_dispatch 0x56506a3a47b0 pool_op_reply(tid 6 (0) Success v18) 2024-09-16T05:07:48.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.262+0000 7ffa84ff9640 10 client.4199.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v18) 2024-09-16T05:07:48.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.262+0000 7ffa84ff9640 10 client.4199.objecter have request 6 at 0x7ffa68003070 Op: delete unmanaged snap 2024-09-16T05:07:48.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.262+0000 7ffa84ff9640 20 client.4199.objecter waiting for client to reach epoch 18 before calling back 2024-09-16T05:07:48.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.262+0000 7ffa84ff9640 10 client.4199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:48.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.262+0000 7ffa84ff9640 10 monclient: _renew_subs 2024-09-16T05:07:48.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.262+0000 7ffa84ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:48.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.262+0000 7ffa84ff9640 1 -- 172.21.15.114:0/2040185577 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=18}) -- 0x7ffa6c03bae0 con 0x56506a4b7a90 2024-09-16T05:07:48.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.262+0000 7ffa84ff9640 10 client.4199.objecter done 2024-09-16T05:07:48.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa84ff9640 1 -- 172.21.15.114:0/2040185577 <== mon.2 v2:172.21.15.114:3301/0 12 ==== osd_map(18..18 src has 1..18) ==== 654+0+0 (secure 0 0 0) 0x7ffa7c035030 con 0x56506a4b7a90 2024-09-16T05:07:48.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa84ff9640 10 client.4199.objecter ms_dispatch 0x56506a3a47b0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:48.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa84ff9640 3 client.4199.objecter handle_osd_map got epochs [18,18] > 17 2024-09-16T05:07:48.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa84ff9640 3 client.4199.objecter handle_osd_map decoding incremental epoch 18 2024-09-16T05:07:48.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa84ff9640 20 client.4199.objecter dump_active .. 0 homeless 2024-09-16T05:07:48.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa867fc640 20 librados: queue_aio_write 0x56506a5bcb50 completion 0x7ffa68001d70 write_seq 2 2024-09-16T05:07:48.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa867fc640 10 client.4199.objecter _op_submit op 0x7ffa64006390 2024-09-16T05:07:48.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa867fc640 20 client.4199.objecter _calc_target epoch 18 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T05:07:48.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa867fc640 20 client.4199.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T05:07:48.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa867fc640 10 client.4199.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [0,4] primary 0 2024-09-16T05:07:48.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa867fc640 20 client.4199.objecter _get_session s=0x56506a5c1af0 osd=0 3 2024-09-16T05:07:48.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa867fc640 10 client.4199.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.0 2024-09-16T05:07:48.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa867fc640 20 client.4199.objecter get_session s=0x56506a5c1af0 osd=0 3 2024-09-16T05:07:48.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa867fc640 15 client.4199.objecter _session_op_assign 0 7 2024-09-16T05:07:48.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa867fc640 15 client.4199.objecter _send_op 7 to 2.2 on osd.0 2024-09-16T05:07:48.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa867fc640 1 -- 172.21.15.114:0/2040185577 --> v2:172.21.15.114:6808/269070361 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7ffa64006c60 con 0x56506a5c2170 2024-09-16T05:07:48.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa867fc640 20 client.4199.objecter put_session s=0x56506a5c1af0 osd=0 4 2024-09-16T05:07:48.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.266+0000 7ffa867fc640 5 client.4199.objecter 1 in flight 2024-09-16T05:07:48.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.270+0000 7ffa8c998640 1 -- 172.21.15.114:0/2040185577 <== osd.0 v2:172.21.15.114:6808/269070361 4 ==== osd_op_reply(7 rbd_info [write 0~19] v18'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffa7c01bce0 con 0x56506a5c2170 2024-09-16T05:07:48.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.270+0000 7ffa8c998640 10 client.4199.objecter ms_dispatch 0x56506a3a47b0 osd_op_reply(7 rbd_info [write 0~19] v18'2 uv2 ondisk = 0) 2024-09-16T05:07:48.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.270+0000 7ffa8c998640 10 client.4199.objecter in handle_osd_op_reply 2024-09-16T05:07:48.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.270+0000 7ffa8c998640 7 client.4199.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-09-16T05:07:48.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.270+0000 7ffa8c998640 10 client.4199.objecter op 0 rval 0 len 0 2024-09-16T05:07:48.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.270+0000 7ffa8c998640 15 client.4199.objecter handle_osd_op_reply completed tid 7 2024-09-16T05:07:48.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.270+0000 7ffa8c998640 15 client.4199.objecter _finish_op 7 2024-09-16T05:07:48.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.270+0000 7ffa8c998640 20 client.4199.objecter put_session s=0x56506a5c1af0 osd=0 4 2024-09-16T05:07:48.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.270+0000 7ffa8c998640 15 client.4199.objecter _session_op_remove 0 7 2024-09-16T05:07:48.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.270+0000 7ffa8c998640 5 client.4199.objecter 0 in flight 2024-09-16T05:07:48.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.270+0000 7ffa8c998640 20 librados: complete_aio_write 0x7ffa68001d70 2024-09-16T05:07:48.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 10 librados: watch_flush enter 2024-09-16T05:07:48.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 10 librados: watch_flush exit 2024-09-16T05:07:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 10 client.4199.objecter close_session for osd.0 2024-09-16T05:07:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2040185577 >> v2:172.21.15.114:6808/269070361 conn(0x56506a5c2170 msgr2=0x56506a5c4600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 1 --2- 172.21.15.114:0/2040185577 >> v2:172.21.15.114:6808/269070361 conn(0x56506a5c2170 0x56506a5c4600 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 20 client.4199.objecter put_session s=0x56506a5c1af0 osd=0 1 2024-09-16T05:07:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 10 client.4199.objecter close_session for osd.1 2024-09-16T05:07:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2040185577 >> v2:172.21.15.114:6800/2903720358 conn(0x56506a56d000 msgr2=0x56506a56f3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 1 --2- 172.21.15.114:0/2040185577 >> v2:172.21.15.114:6800/2903720358 conn(0x56506a56d000 0x56506a56f3f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 20 client.4199.objecter put_session s=0x56506a552ad0 osd=1 1 2024-09-16T05:07:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 20 client.4199.objecter shutdown clearing up homeless session... 2024-09-16T05:07:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 10 client.4199.objecter successfully canceled tick 2024-09-16T05:07:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2040185577 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffa6c035940 msgr2=0x7ffa6c037e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 1 --2- 172.21.15.114:0/2040185577 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffa6c035940 0x7ffa6c037e00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x56506a4be460 tx=0x7ffa7400c040 comp rx=0 tx=0).stop 2024-09-16T05:07:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8dbfa640 1 -- 172.21.15.114:0/2040185577 reap_dead start 2024-09-16T05:07:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 10 monclient: shutdown 2024-09-16T05:07:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2040185577 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56506a4b7a90 msgr2=0x56506a553440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 1 --2- 172.21.15.114:0/2040185577 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56506a4b7a90 0x56506a553440 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c006640 tx=0x7ffa7c006b00 comp rx=0 tx=0).stop 2024-09-16T05:07:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2040185577 shutdown_connections 2024-09-16T05:07:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 1 --2- 172.21.15.114:0/2040185577 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56506a4b7a90 0x56506a553440 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2040185577 >> 172.21.15.114:0/2040185577 conn(0x56506a4c32c0 msgr2=0x56506a554f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:48.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2040185577 shutdown_connections 2024-09-16T05:07:48.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 1 -- 172.21.15.114:0/2040185577 wait complete. 2024-09-16T05:07:48.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.274+0000 7ffa8de84f80 1 librados: shutdown 2024-09-16T05:07:48.281 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T05:07:48.282 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T05:07:48.282 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T05:07:48.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T05:07:48.283 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T05:07:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.410+0000 7fe061759640 10 monclient: get_monmap_and_config 2024-09-16T05:07:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.410+0000 7fe061759640 10 monclient: build_initial_monmap 2024-09-16T05:07:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.410+0000 7fe061759640 10 monclient: monmap: 2024-09-16T05:07:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:48.414885+0000 2024-09-16T05:07:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:48.414885+0000 2024-09-16T05:07:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:48.412 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:48.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.410+0000 7fe061759640 1 Processor -- start 2024-09-16T05:07:48.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.410+0000 7fe061759640 1 -- start start 2024-09-16T05:07:48.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.410+0000 7fe061759640 10 monclient: init 2024-09-16T05:07:48.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:48.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-16T05:07:48.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe05c0a0ec0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:48.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe05c09e1d0 0x7fe05c09e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:48.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 10 monclient(hunting): picked mon.noname-c con 0x7fe05c09e1d0 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:48.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c139640 0x7fe05c13ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:48.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 10 monclient(hunting): picked mon.noname-b con 0x7fe05c139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:48.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe05c09a760 con 0x7fe05c0a0ec0 2024-09-16T05:07:48.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:48.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fe05c0a12a0 con 0x7fe05c09e1d0 2024-09-16T05:07:48.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:48.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe05c0994b0 con 0x7fe05c139640 2024-09-16T05:07:48.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:48.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 10 monclient(hunting): authenticate will time out at 953.044610s 2024-09-16T05:07:48.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c139640 0x7fe05c13ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:48.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05affd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe05c0a0ec0 0x7fe05c09dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38516/0 (socket says 172.21.15.114:38516) 2024-09-16T05:07:48.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05affd640 1 -- 172.21.15.114:0/3951941348 learned_addr learned my addr 172.21.15.114:0/3951941348 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:48.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 10 monclient(hunting): get_auth_request con 0x7fe05c139640 auth_method 0 2024-09-16T05:07:48.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:48.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:48.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:48.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:48.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:48.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:48.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05a7fc640 1 --2- 172.21.15.114:0/3951941348 >> v2:172.21.15.149:3300/0 conn(0x7fe05c09e1d0 0x7fe05c09e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:48.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 1 -- 172.21.15.114:0/3951941348 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2322471839 0 0) 0x7fe05c09a760 con 0x7fe05c0a0ec0 2024-09-16T05:07:48.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:07:48.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:48.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:48.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 10 monclient(hunting): my global_id is 4212 2024-09-16T05:07:48.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 1 -- 172.21.15.114:0/3951941348 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe044003530 con 0x7fe05c0a0ec0 2024-09-16T05:07:48.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05a7fc640 10 monclient(hunting): get_auth_request con 0x7fe05c09e1d0 auth_method 0 2024-09-16T05:07:48.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:48.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:48.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:48.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 10 monclient(hunting): handle_auth_done global_id 4202 payload 274 2024-09-16T05:07:48.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 1 -- 172.21.15.114:0/3951941348 >> v2:172.21.15.149:3300/0 conn(0x7fe05c09e1d0 msgr2=0x7fe05c09e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:48.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 1 --2- 172.21.15.114:0/3951941348 >> v2:172.21.15.149:3300/0 conn(0x7fe05c09e1d0 0x7fe05c09e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 1 -- 172.21.15.114:0/3951941348 >> v1:172.21.15.114:6789/0 conn(0x7fe05c0a0ec0 legacy=0x7fe05c09dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:07:48.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 1 monclient: found mon.noname-b 2024-09-16T05:07:48.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:48.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:48.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 1 -- 172.21.15.114:0/3951941348 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe05c0a1870 con 0x7fe05c139640 2024-09-16T05:07:48.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe05b7fe640 1 --2- 172.21.15.114:0/3951941348 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c139640 0x7fe05c13ba30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe054009d90 tx=0x7fe05401c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bf619a18444c10 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:48.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 1 -- 172.21.15.114:0/3951941348 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0540b2070 con 0x7fe05c139640 2024-09-16T05:07:48.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:48.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:48.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 10 monclient: dump: 2024-09-16T05:07:48.425 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:48.425 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:48.425 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.426 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:48.429 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 10 monclient: _finish_auth 0 2024-09-16T05:07:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:07:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 5 monclient: authenticate success, global_id 4202 2024-09-16T05:07:48.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 1 -- 172.21.15.114:0/3951941348 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0540a5890 con 0x7fe05c139640 2024-09-16T05:07:48.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:48.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:48.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 1 -- 172.21.15.114:0/3951941348 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0540a5c60 con 0x7fe05c139640 2024-09-16T05:07:48.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:48.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe061759640 4 set_mon_vals no callback set 2024-09-16T05:07:48.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:48.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.414+0000 7fe059ffb640 10 monclient: dump: 2024-09-16T05:07:48.430 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:48.430 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:48.430 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.431 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.431 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:48.431 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:48.431 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:48.431 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:48.431 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:48.431 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:48.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient: shutdown 2024-09-16T05:07:48.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:48.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 -- 172.21.15.114:0/3951941348 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c139640 msgr2=0x7fe05c13ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 --2- 172.21.15.114:0/3951941348 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c139640 0x7fe05c13ba30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe054009d90 tx=0x7fe05401c8c0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 -- 172.21.15.114:0/3951941348 shutdown_connections 2024-09-16T05:07:48.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 --2- 172.21.15.114:0/3951941348 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c139640 0x7fe05c13ba30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 --2- 172.21.15.114:0/3951941348 >> v2:172.21.15.149:3300/0 conn(0x7fe05c09e1d0 0x7fe05c09e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 -- 172.21.15.114:0/3951941348 >> 172.21.15.114:0/3951941348 conn(0x7fe05c0a1fb0 msgr2=0x7fe05c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:48.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 -- 172.21.15.114:0/3951941348 shutdown_connections 2024-09-16T05:07:48.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 -- 172.21.15.114:0/3951941348 wait complete. 2024-09-16T05:07:48.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient: build_initial_monmap 2024-09-16T05:07:48.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient: monmap: 2024-09-16T05:07:48.432 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:48.432 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:48.432 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:48.433 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:48.433 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:48.433 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:48.433 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:48.433 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:48.433 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:48.433 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:48.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 librados: starting msgr at 2024-09-16T05:07:48.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 librados: starting objecter 2024-09-16T05:07:48.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 Processor -- start 2024-09-16T05:07:48.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 -- start start 2024-09-16T05:07:48.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 librados: setting wanted keys 2024-09-16T05:07:48.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 librados: calling monclient init 2024-09-16T05:07:48.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient: init 2024-09-16T05:07:48.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:48.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:07:48.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe05c09e1d0 0x7fe05c098a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:48.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient(hunting): picked mon.noname-a con 0x7fe05c09e1d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:48.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe05c0a0ec0 0x7fe05c13cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:48.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient(hunting): picked mon.noname-b con 0x7fe05c0a0ec0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:48.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c139640 0x7fe05c13d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:48.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient(hunting): picked mon.noname-c con 0x7fe05c139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:48.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:48.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe05c13c900 con 0x7fe05c09e1d0 2024-09-16T05:07:48.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:48.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fe05c13c780 con 0x7fe05c0a0ec0 2024-09-16T05:07:48.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:48.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe05c13ca80 con 0x7fe05c139640 2024-09-16T05:07:48.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:48.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient(hunting): authenticate will time out at 953.049697s 2024-09-16T05:07:48.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe05c09e1d0 0x7fe05c098a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:48.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c139640 0x7fe05c13d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:48.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c139640 0x7fe05c13d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55406/0 (socket says 172.21.15.114:55406) 2024-09-16T05:07:48.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 1 -- 172.21.15.114:0/853357398 learned_addr learned my addr 172.21.15.114:0/853357398 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:48.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05affd640 10 monclient(hunting): get_auth_request con 0x7fe05c09e1d0 auth_method 0 2024-09-16T05:07:48.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:48.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:48.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:48.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 10 monclient(hunting): get_auth_request con 0x7fe05c139640 auth_method 0 2024-09-16T05:07:48.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:48.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:48.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:48.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:48.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:48.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:48.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:48.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:48.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:48.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 10 monclient(hunting): handle_auth_done global_id 4205 payload 824 2024-09-16T05:07:48.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05a7fc640 1 --2- 172.21.15.114:0/853357398 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe05c0a0ec0 0x7fe05c13cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:48.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 1 -- 172.21.15.114:0/853357398 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe05c0a0ec0 msgr2=0x7fe05c13cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 1 --2- 172.21.15.114:0/853357398 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe05c0a0ec0 0x7fe05c13cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 1 -- 172.21.15.114:0/853357398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe05c09e1d0 msgr2=0x7fe05c098a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 1 --2- 172.21.15.114:0/853357398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe05c09e1d0 0x7fe05c098a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:07:48.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 1 monclient: found mon.noname-c 2024-09-16T05:07:48.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:48.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:48.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 1 -- 172.21.15.114:0/853357398 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe05c093000 con 0x7fe05c139640 2024-09-16T05:07:48.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05affd640 1 --2- 172.21.15.114:0/853357398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe05c09e1d0 0x7fe05c098a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:48.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe05b7fe640 1 --2- 172.21.15.114:0/853357398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c139640 0x7fe05c13d4c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe0540b0040 tx=0x7fe054019660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:48.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 10 client.?.objecter ms_handle_connect 0x7fe05c139640 2024-09-16T05:07:48.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:48.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 1 -- 172.21.15.114:0/853357398 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0540b2050 con 0x7fe05c139640 2024-09-16T05:07:48.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:48.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:48.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 10 monclient: dump: 2024-09-16T05:07:48.438 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:48.438 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:48.438 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.439 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.439 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:48.439 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:48.439 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:48.439 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:48.439 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:48.439 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:48.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 10 monclient: _finish_auth 0 2024-09-16T05:07:48.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 10 monclient: _check_auth_tickets 2024-09-16T05:07:48.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:48.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 1 -- 172.21.15.114:0/853357398 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0540be070 con 0x7fe05c139640 2024-09-16T05:07:48.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 5 monclient: authenticate success, global_id 4205 2024-09-16T05:07:48.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient: _renew_subs 2024-09-16T05:07:48.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 -- 172.21.15.114:0/853357398 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe05c13ea60 con 0x7fe05c139640 2024-09-16T05:07:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 1 -- 172.21.15.114:0/853357398 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0540ada70 con 0x7fe05c139640 2024-09-16T05:07:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 10 monclient: dump: 2024-09-16T05:07:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:48.441 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0597fa640 4 set_mon_vals no callback set 2024-09-16T05:07:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 client.4205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient: _renew_subs 2024-09-16T05:07:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 -- 172.21.15.114:0/853357398 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe05c091870 con 0x7fe05c139640 2024-09-16T05:07:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 librados: init done 2024-09-16T05:07:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 1 -- 172.21.15.114:0/853357398 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe0540a5710 con 0x7fe05c139640 2024-09-16T05:07:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 1 --2- 172.21.15.114:0/853357398 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe048037810 0x7fe048039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe061759640 1 -- 172.21.15.114:0/853357398 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe05c09e5e0 con 0x7fe05c139640 2024-09-16T05:07:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 1 -- 172.21.15.114:0/853357398 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7fe0540e3910 con 0x7fe05c139640 2024-09-16T05:07:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 10 client.4205.objecter ms_dispatch 0x7fe05c000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 3 client.4205.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 3 client.4205.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.418+0000 7fe0437fe640 20 client.4205.objecter dump_active .. 0 homeless 2024-09-16T05:07:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.422+0000 7fe05affd640 1 --2- 172.21.15.114:0/853357398 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe048037810 0x7fe048039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.422+0000 7fe05affd640 10 monclient: get_auth_request con 0x7fe048037810 auth_method 0 2024-09-16T05:07:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.422+0000 7fe0437fe640 10 client.4205.objecter ms_handle_connect 0x7fe048037810 2024-09-16T05:07:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.422+0000 7fe05affd640 1 --2- 172.21.15.114:0/853357398 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe048037810 0x7fe048039cd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe04c005890 tx=0x7fe04c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:48.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.426+0000 7fe0437fe640 1 -- 172.21.15.114:0/853357398 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fe054016540 con 0x7fe05c139640 2024-09-16T05:07:48.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.426+0000 7fe0437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:48.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.426+0000 7fe0437fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:48.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.634+0000 7fe061759640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-16T05:07:48.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.634+0000 7fe061759640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T05:07:48.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.634+0000 7fe061759640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:48.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.634+0000 7fe061759640 1 -- 172.21.15.114:0/853357398 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fe05c092410 con 0x7fe05c139640 2024-09-16T05:07:48.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe0437fe640 1 -- 172.21.15.114:0/853357398 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+84120 (secure 0 0 0) 0x7fe0540c23c0 con 0x7fe05c139640 2024-09-16T05:07:48.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe0437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T05:07:48.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe0437fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:07:48.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 10 librados: watch_flush enter 2024-09-16T05:07:48.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 10 librados: watch_flush exit 2024-09-16T05:07:48.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 20 client.4205.objecter shutdown clearing up homeless session... 2024-09-16T05:07:48.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 10 client.4205.objecter successfully canceled tick 2024-09-16T05:07:48.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 1 -- 172.21.15.114:0/853357398 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe048037810 msgr2=0x7fe048039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 1 --2- 172.21.15.114:0/853357398 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe048037810 0x7fe048039cd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe04c005890 tx=0x7fe04c009040 comp rx=0 tx=0).stop 2024-09-16T05:07:48.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 10 monclient: shutdown 2024-09-16T05:07:48.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:48.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 1 -- 172.21.15.114:0/853357398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c139640 msgr2=0x7fe05c13d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 1 --2- 172.21.15.114:0/853357398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c139640 0x7fe05c13d4c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe0540b0040 tx=0x7fe054019660 comp rx=0 tx=0).stop 2024-09-16T05:07:48.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 1 -- 172.21.15.114:0/853357398 shutdown_connections 2024-09-16T05:07:48.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 1 --2- 172.21.15.114:0/853357398 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe048037810 0x7fe048039cd0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 1 --2- 172.21.15.114:0/853357398 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe05c139640 0x7fe05c13d4c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 1 --2- 172.21.15.114:0/853357398 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe05c0a0ec0 0x7fe05c13cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 1 --2- 172.21.15.114:0/853357398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe05c09e1d0 0x7fe05c098a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 1 -- 172.21.15.114:0/853357398 >> 172.21.15.114:0/853357398 conn(0x7fe05c0a1fb0 msgr2=0x7fe05c139ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:48.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 1 -- 172.21.15.114:0/853357398 shutdown_connections 2024-09-16T05:07:48.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 1 -- 172.21.15.114:0/853357398 wait complete. 2024-09-16T05:07:48.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.638+0000 7fe061759640 1 librados: shutdown 2024-09-16T05:07:48.656 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":5,"flags":0,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":1185531135}]},"active_addr":"172.21.15.149:6812/1185531135","active_change":"2024-09-16T05:07:32.060336+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.149:0","nonce":3734548556}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.149:0","nonce":3857408436}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.149:0","nonce":4185353869}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.149:0","nonce":3812917563}]}]} 2024-09-16T05:07:48.659 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T05:07:48.659 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T05:07:48.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T05:07:48.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.770+0000 7f6615ab7640 10 monclient: get_monmap_and_config 2024-09-16T05:07:48.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.770+0000 7f6615ab7640 10 monclient: build_initial_monmap 2024-09-16T05:07:48.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.770+0000 7f6615ab7640 10 monclient: monmap: 2024-09-16T05:07:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:48.774988+0000 2024-09-16T05:07:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:48.774988+0000 2024-09-16T05:07:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:48.772 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:48.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.770+0000 7f6615ab7640 1 Processor -- start 2024-09-16T05:07:48.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.770+0000 7f6615ab7640 1 -- start start 2024-09-16T05:07:48.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.770+0000 7f6615ab7640 10 monclient: init 2024-09-16T05:07:48.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:48.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-16T05:07:48.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66100a0ec0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:48.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f661009e1d0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:48.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f661009e5e0 0x7f661013cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:48.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 10 monclient(hunting): picked mon.noname-c con 0x7f661009e5e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:48.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f661009a760 con 0x7f661009e1d0 2024-09-16T05:07:48.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66100a1870 con 0x7f66100a0ec0 2024-09-16T05:07:48.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:48.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f66100a12a0 con 0x7f661009e5e0 2024-09-16T05:07:48.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:48.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 10 monclient(hunting): authenticate will time out at 953.404138s 2024-09-16T05:07:48.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f661009e1d0 0x7f6610139640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38532/0 (socket says 172.21.15.114:38532) 2024-09-16T05:07:48.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 1 -- 172.21.15.114:0/1691401990 learned_addr learned my addr 172.21.15.114:0/1691401990 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:48.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 1 --2- 172.21.15.114:0/1691401990 >> v2:172.21.15.149:3300/0 conn(0x7f661009e5e0 0x7f661013cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:48.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 10 monclient(hunting): get_auth_request con 0x7f661009e5e0 auth_method 0 2024-09-16T05:07:48.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:48.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:48.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:48.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 -- 172.21.15.114:0/1691401990 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2886910787 0 0) 0x7f661009a760 con 0x7f661009e1d0 2024-09-16T05:07:48.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:07:48.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:48.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:48.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient(hunting): my global_id is 4218 2024-09-16T05:07:48.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 -- 172.21.15.114:0/1691401990 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65fc003530 con 0x7f661009e1d0 2024-09-16T05:07:48.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:48.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:48.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:48.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 -- 172.21.15.114:0/1691401990 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3401402018 0 0) 0x7f65fc003530 con 0x7f661009e1d0 2024-09-16T05:07:48.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 -- 172.21.15.114:0/1691401990 >> v2:172.21.15.149:3300/0 conn(0x7f661009e5e0 msgr2=0x7f661013cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:48.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 --2- 172.21.15.114:0/1691401990 >> v2:172.21.15.149:3300/0 conn(0x7f661009e5e0 0x7f661013cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 -- 172.21.15.114:0/1691401990 >> v1:172.21.15.149:6789/0 conn(0x7f66100a0ec0 legacy=0x7f661009dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:07:48.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:07:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:07:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 -- 172.21.15.114:0/1691401990 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6610099790 con 0x7f661009e1d0 2024-09-16T05:07:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:07:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:07:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 -- 172.21.15.114:0/1691401990 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f65f8001800 con 0x7f661009e1d0 2024-09-16T05:07:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:07:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: dump: 2024-09-16T05:07:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:48.789 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:07:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 -- 172.21.15.114:0/1691401990 >> v1:172.21.15.114:6789/0 conn(0x7f661009e1d0 legacy=0x7f6610139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:07:48.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 --2- 172.21.15.114:0/1691401990 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65fc0039c0 0x7f65fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:48.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient(hunting): picked mon.b con 0x7f65fc0039c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:48.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 --2- 172.21.15.114:0/1691401990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65fc004810 0x7f65fc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:48.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient(hunting): picked mon.a con 0x7f65fc004810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:48.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 --2- 172.21.15.114:0/1691401990 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65fc0051d0 0x7f65fc00dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:48.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient(hunting): picked mon.c con 0x7f65fc0051d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:48.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:48.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 -- 172.21.15.114:0/1691401990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f66100a12a0 con 0x7f65fc004810 2024-09-16T05:07:48.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:48.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 -- 172.21.15.114:0/1691401990 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc00e380 con 0x7f65fc0039c0 2024-09-16T05:07:48.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:48.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 -- 172.21.15.114:0/1691401990 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f65fc00e580 con 0x7f65fc0051d0 2024-09-16T05:07:48.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:48.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:07:48.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 1 --2- 172.21.15.114:0/1691401990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65fc004810 0x7f65fc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:48.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 10 monclient(hunting): get_auth_request con 0x7f65fc004810 auth_method 0 2024-09-16T05:07:48.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:48.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:48.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:48.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 1 --2- 172.21.15.114:0/1691401990 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65fc0051d0 0x7f65fc00dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:48.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 10 monclient(hunting): get_auth_request con 0x7f65fc0051d0 auth_method 0 2024-09-16T05:07:48.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:48.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:48.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:48.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:48.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:48.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:07:48.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660f7fe640 1 --2- 172.21.15.114:0/1691401990 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65fc0039c0 0x7f65fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:48.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:48.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:48.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:07:48.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660f7fe640 10 monclient(hunting): get_auth_request con 0x7f65fc0039c0 auth_method 0 2024-09-16T05:07:48.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:48.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:48.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:48.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 10 monclient(hunting): handle_auth_done global_id 4218 payload 293 2024-09-16T05:07:48.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 1 -- 172.21.15.114:0/1691401990 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65fc0051d0 msgr2=0x7f65fc00dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:48.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 1 --2- 172.21.15.114:0/1691401990 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65fc0051d0 0x7f65fc00dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 1 -- 172.21.15.114:0/1691401990 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65fc0039c0 msgr2=0x7f65fc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:48.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 1 --2- 172.21.15.114:0/1691401990 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65fc0039c0 0x7f65fc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 10 monclient: _finish_hunting 0 2024-09-16T05:07:48.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 1 monclient: found mon.a 2024-09-16T05:07:48.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:48.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:48.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 1 -- 172.21.15.114:0/1691401990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65fc003530 con 0x7f65fc004810 2024-09-16T05:07:48.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 10 monclient: _finish_auth 0 2024-09-16T05:07:48.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 10 monclient: _check_auth_tickets 2024-09-16T05:07:48.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:48.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660ffff640 1 -- 172.21.15.114:0/1691401990 reap_dead start 2024-09-16T05:07:48.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 5 monclient: authenticate success, global_id 4218 2024-09-16T05:07:48.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:48.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660effd640 1 --2- 172.21.15.114:0/1691401990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65fc004810 0x7f65fc004c90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f65f800c240 tx=0x7f65f801cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c49c1a32b6964e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:48.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 -- 172.21.15.114:0/1691401990 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8007d10 con 0x7f65fc004810 2024-09-16T05:07:48.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:48.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:48.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: dump: 2024-09-16T05:07:48.794 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:48.794 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:48.794 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.794 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.794 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:48.794 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:48.795 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:48.795 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:48.795 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:48.795 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:48.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 -- 172.21.15.114:0/1691401990 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65f80a9070 con 0x7f65fc004810 2024-09-16T05:07:48.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:48.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:48.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:48.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 1 -- 172.21.15.114:0/1691401990 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8007510 con 0x7f65fc004810 2024-09-16T05:07:48.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:48.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f6615ab7640 4 set_mon_vals no callback set 2024-09-16T05:07:48.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:48.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.774+0000 7f660e7fc640 10 monclient: dump: 2024-09-16T05:07:48.795 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:48.796 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:48.796 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.796 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.796 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:48.796 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:48.796 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:48.796 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:48.796 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:48.796 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:48.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient: shutdown 2024-09-16T05:07:48.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:48.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 -- 172.21.15.114:0/1691401990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65fc004810 msgr2=0x7f65fc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 --2- 172.21.15.114:0/1691401990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65fc004810 0x7f65fc004c90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f65f800c240 tx=0x7f65f801cbe0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 -- 172.21.15.114:0/1691401990 shutdown_connections 2024-09-16T05:07:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 --2- 172.21.15.114:0/1691401990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65fc004810 0x7f65fc004c90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 -- 172.21.15.114:0/1691401990 >> 172.21.15.114:0/1691401990 conn(0x7f66100a1fb0 msgr2=0x7f66100a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 -- 172.21.15.114:0/1691401990 shutdown_connections 2024-09-16T05:07:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 -- 172.21.15.114:0/1691401990 wait complete. 2024-09-16T05:07:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient: build_initial_monmap 2024-09-16T05:07:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient: monmap: 2024-09-16T05:07:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:48.798 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:48.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 librados: starting msgr at 2024-09-16T05:07:48.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 librados: starting objecter 2024-09-16T05:07:48.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 Processor -- start 2024-09-16T05:07:48.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 -- start start 2024-09-16T05:07:48.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 librados: setting wanted keys 2024-09-16T05:07:48.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 librados: calling monclient init 2024-09-16T05:07:48.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient: init 2024-09-16T05:07:48.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:48.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:07:48.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65fc004810 0x7f66101c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:48.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient(hunting): picked mon.noname-c con 0x7f65fc004810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:48.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6610133a50 0x7f6610133e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:48.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient(hunting): picked mon.noname-a con 0x7f6610133a50 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:48.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f661012a570 0x7f661012a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:48.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient(hunting): picked mon.noname-b con 0x7f661012a570 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:48.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:48.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f661009d340 con 0x7f6610133a50 2024-09-16T05:07:48.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:48.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f661009d1c0 con 0x7f661012a570 2024-09-16T05:07:48.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:48.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f661009d4c0 con 0x7f65fc004810 2024-09-16T05:07:48.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:48.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient(hunting): authenticate will time out at 953.409871s 2024-09-16T05:07:48.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6610133a50 0x7f6610133e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:48.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660f7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65fc004810 0x7f66101c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:48.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6610133a50 0x7f6610133e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54736/0 (socket says 172.21.15.114:54736) 2024-09-16T05:07:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 1 -- 172.21.15.114:0/2274651508 learned_addr learned my addr 172.21.15.114:0/2274651508 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 10 monclient(hunting): get_auth_request con 0x7f6610133a50 auth_method 0 2024-09-16T05:07:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660f7fe640 10 monclient(hunting): get_auth_request con 0x7f65fc004810 auth_method 0 2024-09-16T05:07:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:48.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:48.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 10 monclient(hunting): handle_auth_done global_id 4221 payload 824 2024-09-16T05:07:48.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 1 -- 172.21.15.114:0/2274651508 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65fc004810 msgr2=0x7f66101c18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 1 --2- 172.21.15.114:0/2274651508 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65fc004810 0x7f66101c18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 1 -- 172.21.15.114:0/2274651508 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f661012a570 msgr2=0x7f661012a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 1 --2- 172.21.15.114:0/2274651508 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f661012a570 0x7f661012a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 10 monclient: _finish_hunting 0 2024-09-16T05:07:48.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 1 monclient: found mon.noname-a 2024-09-16T05:07:48.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:48.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:48.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 1 -- 172.21.15.114:0/2274651508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66101cf3d0 con 0x7f6610133a50 2024-09-16T05:07:48.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660f7fe640 1 --2- 172.21.15.114:0/2274651508 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65fc004810 0x7f66101c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:48.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660effd640 1 --2- 172.21.15.114:0/2274651508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6610133a50 0x7f6610133e50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f65f8000f00 tx=0x7f65fc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:48.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 10 client.?.objecter ms_handle_connect 0x7f6610133a50 2024-09-16T05:07:48.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:48.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 1 -- 172.21.15.114:0/2274651508 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f80bc070 con 0x7f6610133a50 2024-09-16T05:07:48.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:48.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:48.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 10 monclient: dump: 2024-09-16T05:07:48.802 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:48.802 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:48.802 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.802 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.802 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:48.802 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:48.802 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:48.802 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:48.802 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:48.803 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:48.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 10 monclient: _finish_auth 0 2024-09-16T05:07:48.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 10 monclient: _check_auth_tickets 2024-09-16T05:07:48.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:48.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 5 monclient: authenticate success, global_id 4221 2024-09-16T05:07:48.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 1 -- 172.21.15.114:0/2274651508 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65f80a5dd0 con 0x7f6610133a50 2024-09-16T05:07:48.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient: _renew_subs 2024-09-16T05:07:48.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:48.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 -- 172.21.15.114:0/2274651508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f661012cf10 con 0x7f6610133a50 2024-09-16T05:07:48.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:48.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 1 -- 172.21.15.114:0/2274651508 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8007d10 con 0x7f6610133a50 2024-09-16T05:07:48.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:48.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:48.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 10 monclient: dump: 2024-09-16T05:07:48.804 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:48.804 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:48.804 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.804 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:48.804 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:48.804 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:48.804 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:48.804 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:48.804 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:48.804 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:48.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f660d7fa640 4 set_mon_vals no callback set 2024-09-16T05:07:48.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:48.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient: _renew_subs 2024-09-16T05:07:48.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:48.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 -- 172.21.15.114:0/2274651508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f661012c200 con 0x7f6610133a50 2024-09-16T05:07:48.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 librados: init done 2024-09-16T05:07:48.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:48.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:48.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:48.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6615ab7640 1 -- 172.21.15.114:0/2274651508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66100a4940 con 0x7f6610133a50 2024-09-16T05:07:48.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 1 -- 172.21.15.114:0/2274651508 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f65f80bf070 con 0x7f6610133a50 2024-09-16T05:07:48.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.778+0000 7f6607fff640 1 --2- 172.21.15.114:0/2274651508 >> v2:172.21.15.149:6812/1185531135 conn(0x7f65f0037840 0x7f65f0039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:48.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.782+0000 7f6607fff640 1 -- 172.21.15.114:0/2274651508 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f65f80e19f0 con 0x7f6610133a50 2024-09-16T05:07:48.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.782+0000 7f6607fff640 10 client.4221.objecter ms_dispatch 0x7f6610000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:48.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.782+0000 7f6607fff640 3 client.4221.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:48.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.782+0000 7f6607fff640 3 client.4221.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:48.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.782+0000 7f6607fff640 20 client.4221.objecter dump_active .. 0 homeless 2024-09-16T05:07:48.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.782+0000 7f660f7fe640 1 --2- 172.21.15.114:0/2274651508 >> v2:172.21.15.149:6812/1185531135 conn(0x7f65f0037840 0x7f65f0039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:48.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.782+0000 7f660f7fe640 10 monclient: get_auth_request con 0x7f65f0037840 auth_method 0 2024-09-16T05:07:48.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.782+0000 7f660f7fe640 1 --2- 172.21.15.114:0/2274651508 >> v2:172.21.15.149:6812/1185531135 conn(0x7f65f0037840 0x7f65f0039d00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6600000c10 tx=0x7f6600009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:48.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.782+0000 7f6607fff640 10 client.4221.objecter ms_handle_connect 0x7f65f0037840 2024-09-16T05:07:48.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.786+0000 7f6607fff640 1 -- 172.21.15.114:0/2274651508 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f65f80e3e20 con 0x7f6610133a50 2024-09-16T05:07:48.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.786+0000 7f6607fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:48.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.786+0000 7f6607fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:48.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.946+0000 7f6615ab7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:48.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.946+0000 7f6615ab7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:48.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.946+0000 7f6615ab7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:48.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.946+0000 7f6615ab7640 1 -- 172.21.15.114:0/2274651508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f661009d7c0 con 0x7f6610133a50 2024-09-16T05:07:48.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6607fff640 1 -- 172.21.15.114:0/2274651508 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+10979 (secure 0 0 0) 0x7f65f80c8b00 con 0x7f6610133a50 2024-09-16T05:07:48.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6607fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:48.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6607fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:48.949 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:07:48.950 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":18,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","created":"2024-09-16T05:07:27.305861+0000","modified":"2024-09-16T05:07:48.248102+0000","last_up_change":"2024-09-16T05:07:39.180058+0000","last_in_change":"2024-09-16T05:07:29.028596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:07:38.142989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T05:07:44.273942+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6ffdab41-0191-4764-ad62-922ec1091569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":269070361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":269070361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6811","nonce":269070361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":269070361}]},"public_addr":"172.21.15.114:6808/269070361","cluster_addr":"172.21.15.114:6809/269070361","heartbeat_back_addr":"172.21.15.114:6811/269070361","heartbeat_front_addr":"172.21.15.114:6810/269070361","state":["exists","up"]},{"osd":1,"uuid":"7de579ee-87a3-4463-87a5-6980b56cd113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2903720358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":2903720358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6803","nonce":2903720358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2903720358}]},"public_addr":"172.21.15.114:6800/2903720358","cluster_addr":"172.21.15.114:6801/2903720358","heartbeat_back_addr":"172.21.15.114:6803/2903720358","heartbeat_front_addr":"172.21.15.114:6802/2903720358","state":["exists","up"]},{"osd":2,"uuid":"258c8bf4-58c5-4c13-a0e5-ef9931efd698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1932657732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":1932657732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6807","nonce":1932657732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1932657732}]},"public_addr":"172.21.15.114:6804/1932657732","cluster_addr":"172.21.15.114:6805/1932657732","heartbeat_back_addr":"172.21.15.114:6807/1932657732","heartbeat_front_addr":"172.21.15.114:6806/1932657732","state":["exists","up"]},{"osd":3,"uuid":"acc1e03f-8a6c-4c85-9525-88707597d732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6805","nonce":1362772130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6807","nonce":1362772130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6811","nonce":1362772130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6809","nonce":1362772130}]},"public_addr":"172.21.15.149:6805/1362772130","cluster_addr":"172.21.15.149:6807/1362772130","heartbeat_back_addr":"172.21.15.149:6811/1362772130","heartbeat_front_addr":"172.21.15.149:6809/1362772130","state":["exists","up"]},{"osd":4,"uuid":"92cca4cf-19e9-44a1-9588-7c301a2c2065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":1092505261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6801","nonce":1092505261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6803","nonce":1092505261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":1092505261}]},"public_addr":"172.21.15.149:6800/1092505261","cluster_addr":"172.21.15.149:6801/1092505261","heartbeat_back_addr":"172.21.15.149:6803/1092505261","heartbeat_front_addr":"172.21.15.149:6802/1092505261","state":["exists","up"]},{"osd":5,"uuid":"ac717f4e-83fc-46ea-9816-7e35ef75a674","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":1244703827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":1244703827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":1244703827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":1244703827}]},"public_addr":"172.21.15.149:6804/1244703827","cluster_addr":"172.21.15.149:6806/1244703827","heartbeat_back_addr":"172.21.15.149:6810/1244703827","heartbeat_front_addr":"172.21.15.149:6808/1244703827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.554731+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.453254+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.446266+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:35.709335+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.069934+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.433872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:07:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 10 librados: watch_flush enter 2024-09-16T05:07:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 10 librados: watch_flush exit 2024-09-16T05:07:48.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 20 client.4221.objecter shutdown clearing up homeless session... 2024-09-16T05:07:48.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 10 client.4221.objecter successfully canceled tick 2024-09-16T05:07:48.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 1 -- 172.21.15.114:0/2274651508 >> v2:172.21.15.149:6812/1185531135 conn(0x7f65f0037840 msgr2=0x7f65f0039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 1 --2- 172.21.15.114:0/2274651508 >> v2:172.21.15.149:6812/1185531135 conn(0x7f65f0037840 0x7f65f0039d00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6600000c10 tx=0x7f6600009040 comp rx=0 tx=0).stop 2024-09-16T05:07:48.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 10 monclient: shutdown 2024-09-16T05:07:48.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:48.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 1 -- 172.21.15.114:0/2274651508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6610133a50 msgr2=0x7f6610133e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:48.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 1 --2- 172.21.15.114:0/2274651508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6610133a50 0x7f6610133e50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f65f8000f00 tx=0x7f65fc005840 comp rx=0 tx=0).stop 2024-09-16T05:07:48.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 1 -- 172.21.15.114:0/2274651508 shutdown_connections 2024-09-16T05:07:48.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 1 --2- 172.21.15.114:0/2274651508 >> v2:172.21.15.149:6812/1185531135 conn(0x7f65f0037840 0x7f65f0039d00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 1 --2- 172.21.15.114:0/2274651508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6610133a50 0x7f6610133e50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 1 --2- 172.21.15.114:0/2274651508 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f661012a570 0x7f661012a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 1 --2- 172.21.15.114:0/2274651508 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65fc004810 0x7f66101c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:48.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 1 -- 172.21.15.114:0/2274651508 >> 172.21.15.114:0/2274651508 conn(0x7f66100a1fb0 msgr2=0x7f661009a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:48.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 1 -- 172.21.15.114:0/2274651508 shutdown_connections 2024-09-16T05:07:48.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 1 -- 172.21.15.114:0/2274651508 wait complete. 2024-09-16T05:07:48.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:48.950+0000 7f6615ab7640 1 librados: shutdown 2024-09-16T05:07:48.967 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T05:07:48.967 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T05:07:49.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.086+0000 7f172f9d3640 10 monclient: get_monmap_and_config 2024-09-16T05:07:49.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.086+0000 7f172f9d3640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.086+0000 7f172f9d3640 10 monclient: monmap: 2024-09-16T05:07:49.087 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.088 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.088 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:49.090862+0000 2024-09-16T05:07:49.088 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:49.090862+0000 2024-09-16T05:07:49.088 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.088 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.088 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:49.088 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:49.088 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:49.088 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:49.088 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:49.088 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.086+0000 7f172f9d3640 1 Processor -- start 2024-09-16T05:07:49.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 1 -- start start 2024-09-16T05:07:49.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 10 monclient: init 2024-09-16T05:07:49.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T05:07:49.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f17280a0ba0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:49.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172809f650 0x7f172809fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 10 monclient(hunting): picked mon.noname-b con 0x7f172809f650 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17281398f0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:49.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17280a1ac0 con 0x7f17280a0ba0 2024-09-16T05:07:49.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f172809d880 con 0x7f17281398f0 2024-09-16T05:07:49.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1728068960 con 0x7f172809f650 2024-09-16T05:07:49.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 10 monclient(hunting): authenticate will time out at 953.720868s 2024-09-16T05:07:49.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172809f650 0x7f172809fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172d76f640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f17280a0ba0 0x7f1728068250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38544/0 (socket says 172.21.15.114:38544) 2024-09-16T05:07:49.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172d76f640 1 -- 172.21.15.114:0/2680376646 learned_addr learned my addr 172.21.15.114:0/2680376646 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:49.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 10 monclient(hunting): get_auth_request con 0x7f172809f650 auth_method 0 2024-09-16T05:07:49.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 10 monclient(hunting): handle_auth_done global_id 4211 payload 274 2024-09-16T05:07:49.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 1 -- 172.21.15.114:0/2680376646 >> v1:172.21.15.149:6789/0 conn(0x7f17281398f0 legacy=0x7f172813bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:49.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 1 -- 172.21.15.114:0/2680376646 >> v1:172.21.15.114:6789/0 conn(0x7f17280a0ba0 legacy=0x7f1728068250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 1 monclient: found mon.noname-b 2024-09-16T05:07:49.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f171ffff640 1 -- 172.21.15.114:0/2680376646 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 192034024 0 0) 0x7f17280a1ac0 con 0x7f17280a0ba0 2024-09-16T05:07:49.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 1 -- 172.21.15.114:0/2680376646 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f172809db60 con 0x7f172809f650 2024-09-16T05:07:49.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f171ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:07:49.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172cf6e640 1 --2- 172.21.15.114:0/2680376646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172809f650 0x7f172809fa70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f17180097f0 tx=0x7f171801c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56229d239c6bf5b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f171ffff640 1 -- 172.21.15.114:0/2680376646 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17180b0070 con 0x7f172809f650 2024-09-16T05:07:49.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f171ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f171ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:49.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f171ffff640 10 monclient: dump: 2024-09-16T05:07:49.101 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.101 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.101 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.101 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.102 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.102 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.102 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.102 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.102 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.102 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f171ffff640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f171ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f171ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f171ffff640 1 -- 172.21.15.114:0/2680376646 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17180ab600 con 0x7f172809f650 2024-09-16T05:07:49.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 5 monclient: authenticate success, global_id 4211 2024-09-16T05:07:49.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:49.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f171ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:49.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f172f9d3640 4 set_mon_vals no callback set 2024-09-16T05:07:49.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f171ffff640 1 -- 172.21.15.114:0/2680376646 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17180a4e60 con 0x7f172809f650 2024-09-16T05:07:49.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f171ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f171ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:49.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.090+0000 7f171ffff640 10 monclient: dump: 2024-09-16T05:07:49.103 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.104 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.104 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.104 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.104 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.104 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.104 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.104 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.104 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.104 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient: shutdown 2024-09-16T05:07:49.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 -- 172.21.15.114:0/2680376646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172809f650 msgr2=0x7f172809fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 --2- 172.21.15.114:0/2680376646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172809f650 0x7f172809fa70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f17180097f0 tx=0x7f171801c330 comp rx=0 tx=0).stop 2024-09-16T05:07:49.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 -- 172.21.15.114:0/2680376646 shutdown_connections 2024-09-16T05:07:49.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 --2- 172.21.15.114:0/2680376646 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f172809f650 0x7f172809fa70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 -- 172.21.15.114:0/2680376646 >> 172.21.15.114:0/2680376646 conn(0x7f17280a6310 msgr2=0x7f17280a2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 -- 172.21.15.114:0/2680376646 shutdown_connections 2024-09-16T05:07:49.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 -- 172.21.15.114:0/2680376646 wait complete. 2024-09-16T05:07:49.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient: monmap: 2024-09-16T05:07:49.105 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.105 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:49.105 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:49.105 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:49.105 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.105 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.106 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:49.106 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:49.106 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:49.106 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 librados: starting msgr at 2024-09-16T05:07:49.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 librados: starting objecter 2024-09-16T05:07:49.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 Processor -- start 2024-09-16T05:07:49.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 -- start start 2024-09-16T05:07:49.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 librados: setting wanted keys 2024-09-16T05:07:49.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 librados: calling monclient init 2024-09-16T05:07:49.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient: init 2024-09-16T05:07:49.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:07:49.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f172809f650 0x7f17281cf430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient(hunting): picked mon.noname-b con 0x7f172809f650 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:49.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17280a0ba0 0x7f17281cf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient(hunting): picked mon.noname-c con 0x7f17280a0ba0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17281398f0 0x7f172811cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient(hunting): picked mon.noname-a con 0x7f17281398f0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:49.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f172809ffb0 con 0x7f17281398f0 2024-09-16T05:07:49.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f17280a3250 con 0x7f172809f650 2024-09-16T05:07:49.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f17280a0130 con 0x7f17280a0ba0 2024-09-16T05:07:49.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient(hunting): authenticate will time out at 953.726296s 2024-09-16T05:07:49.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17281398f0 0x7f172811cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17281398f0 0x7f172811cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54740/0 (socket says 172.21.15.114:54740) 2024-09-16T05:07:49.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 1 -- 172.21.15.114:0/3588961836 learned_addr learned my addr 172.21.15.114:0/3588961836 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 10 monclient(hunting): get_auth_request con 0x7f17281398f0 auth_method 0 2024-09-16T05:07:49.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172cf6e640 1 --2- 172.21.15.114:0/3588961836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17280a0ba0 0x7f17281cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172cf6e640 10 monclient(hunting): get_auth_request con 0x7f17280a0ba0 auth_method 0 2024-09-16T05:07:49.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172cf6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172cf6e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172cf6e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172d76f640 1 --2- 172.21.15.114:0/3588961836 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f172809f650 0x7f17281cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172cf6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172cf6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172cf6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172d76f640 10 monclient(hunting): get_auth_request con 0x7f172809f650 auth_method 0 2024-09-16T05:07:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172d76f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172d76f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172d76f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 10 monclient(hunting): handle_auth_done global_id 4227 payload 824 2024-09-16T05:07:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 1 -- 172.21.15.114:0/3588961836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17280a0ba0 msgr2=0x7f17281cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 1 --2- 172.21.15.114:0/3588961836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17280a0ba0 0x7f17281cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 1 -- 172.21.15.114:0/3588961836 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f172809f650 msgr2=0x7f17281cf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 1 --2- 172.21.15.114:0/3588961836 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f172809f650 0x7f17281cf430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 1 monclient: found mon.noname-a 2024-09-16T05:07:49.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 1 -- 172.21.15.114:0/3588961836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17281204b0 con 0x7f17281398f0 2024-09-16T05:07:49.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172df70640 1 --2- 172.21.15.114:0/3588961836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17281398f0 0x7f172811cb30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f172000c780 tx=0x7f172000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171dffb640 10 client.?.objecter ms_handle_connect 0x7f17281398f0 2024-09-16T05:07:49.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:49.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171dffb640 1 -- 172.21.15.114:0/3588961836 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1720016020 con 0x7f17281398f0 2024-09-16T05:07:49.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:49.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171dffb640 10 monclient: dump: 2024-09-16T05:07:49.110 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.110 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.110 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.110 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.111 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.111 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.111 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.111 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.111 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.111 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171dffb640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171dffb640 1 -- 172.21.15.114:0/3588961836 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f172000a9f0 con 0x7f17281398f0 2024-09-16T05:07:49.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 5 monclient: authenticate success, global_id 4227 2024-09-16T05:07:49.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171dffb640 1 -- 172.21.15.114:0/3588961836 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172000ae20 con 0x7f17281398f0 2024-09-16T05:07:49.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171dffb640 10 monclient: dump: 2024-09-16T05:07:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.112 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f171f7fe640 4 set_mon_vals no callback set 2024-09-16T05:07:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient: _renew_subs 2024-09-16T05:07:49.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 -- 172.21.15.114:0/3588961836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f172811f2e0 con 0x7f17281398f0 2024-09-16T05:07:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 client.4227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient: _renew_subs 2024-09-16T05:07:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 -- 172.21.15.114:0/3588961836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17281201d0 con 0x7f17281398f0 2024-09-16T05:07:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.094+0000 7f172f9d3640 1 librados: init done 2024-09-16T05:07:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.098+0000 7f172f9d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.098+0000 7f172f9d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.098+0000 7f172f9d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.098+0000 7f171dffb640 1 -- 172.21.15.114:0/3588961836 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1720019020 con 0x7f17281398f0 2024-09-16T05:07:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.098+0000 7f172f9d3640 1 -- 172.21.15.114:0/3588961836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f172809fa70 con 0x7f17281398f0 2024-09-16T05:07:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.098+0000 7f171dffb640 1 --2- 172.21.15.114:0/3588961836 >> v2:172.21.15.149:6812/1185531135 conn(0x7f170c037840 0x7f170c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.098+0000 7f171dffb640 1 -- 172.21.15.114:0/3588961836 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f17200482a0 con 0x7f17281398f0 2024-09-16T05:07:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.098+0000 7f171dffb640 10 client.4227.objecter ms_dispatch 0x7f1728000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:49.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.098+0000 7f171dffb640 3 client.4227.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:49.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.098+0000 7f171dffb640 3 client.4227.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:49.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.098+0000 7f171dffb640 20 client.4227.objecter dump_active .. 0 homeless 2024-09-16T05:07:49.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.098+0000 7f172d76f640 1 --2- 172.21.15.114:0/3588961836 >> v2:172.21.15.149:6812/1185531135 conn(0x7f170c037840 0x7f170c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.098+0000 7f172d76f640 10 monclient: get_auth_request con 0x7f170c037840 auth_method 0 2024-09-16T05:07:49.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.098+0000 7f171dffb640 10 client.4227.objecter ms_handle_connect 0x7f170c037840 2024-09-16T05:07:49.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.098+0000 7f172d76f640 1 --2- 172.21.15.114:0/3588961836 >> v2:172.21.15.149:6812/1185531135 conn(0x7f170c037840 0x7f170c039d00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1724000f70 tx=0x7f1724007b60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.102+0000 7f171dffb640 1 -- 172.21.15.114:0/3588961836 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f1720020070 con 0x7f17281398f0 2024-09-16T05:07:49.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.102+0000 7f171dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.102+0000 7f171dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:49.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.254+0000 7f172f9d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:49.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.254+0000 7f172f9d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:49.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.254+0000 7f172f9d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.254+0000 7f172f9d3640 1 -- 172.21.15.114:0/3588961836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f172811e780 con 0x7f17281398f0 2024-09-16T05:07:49.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.254+0000 7f171dffb640 1 -- 172.21.15.114:0/3588961836 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+10979 (secure 0 0 0) 0x7f1720012dc0 con 0x7f17281398f0 2024-09-16T05:07:49.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.254+0000 7f171dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:49.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.254+0000 7f171dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:49.256 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:07:49.256 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":18,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","created":"2024-09-16T05:07:27.305861+0000","modified":"2024-09-16T05:07:48.248102+0000","last_up_change":"2024-09-16T05:07:39.180058+0000","last_in_change":"2024-09-16T05:07:29.028596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:07:38.142989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T05:07:44.273942+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6ffdab41-0191-4764-ad62-922ec1091569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":269070361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":269070361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6811","nonce":269070361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":269070361}]},"public_addr":"172.21.15.114:6808/269070361","cluster_addr":"172.21.15.114:6809/269070361","heartbeat_back_addr":"172.21.15.114:6811/269070361","heartbeat_front_addr":"172.21.15.114:6810/269070361","state":["exists","up"]},{"osd":1,"uuid":"7de579ee-87a3-4463-87a5-6980b56cd113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2903720358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":2903720358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6803","nonce":2903720358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2903720358}]},"public_addr":"172.21.15.114:6800/2903720358","cluster_addr":"172.21.15.114:6801/2903720358","heartbeat_back_addr":"172.21.15.114:6803/2903720358","heartbeat_front_addr":"172.21.15.114:6802/2903720358","state":["exists","up"]},{"osd":2,"uuid":"258c8bf4-58c5-4c13-a0e5-ef9931efd698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1932657732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":1932657732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6807","nonce":1932657732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1932657732}]},"public_addr":"172.21.15.114:6804/1932657732","cluster_addr":"172.21.15.114:6805/1932657732","heartbeat_back_addr":"172.21.15.114:6807/1932657732","heartbeat_front_addr":"172.21.15.114:6806/1932657732","state":["exists","up"]},{"osd":3,"uuid":"acc1e03f-8a6c-4c85-9525-88707597d732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6805","nonce":1362772130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6807","nonce":1362772130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6811","nonce":1362772130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6809","nonce":1362772130}]},"public_addr":"172.21.15.149:6805/1362772130","cluster_addr":"172.21.15.149:6807/1362772130","heartbeat_back_addr":"172.21.15.149:6811/1362772130","heartbeat_front_addr":"172.21.15.149:6809/1362772130","state":["exists","up"]},{"osd":4,"uuid":"92cca4cf-19e9-44a1-9588-7c301a2c2065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":1092505261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6801","nonce":1092505261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6803","nonce":1092505261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":1092505261}]},"public_addr":"172.21.15.149:6800/1092505261","cluster_addr":"172.21.15.149:6801/1092505261","heartbeat_back_addr":"172.21.15.149:6803/1092505261","heartbeat_front_addr":"172.21.15.149:6802/1092505261","state":["exists","up"]},{"osd":5,"uuid":"ac717f4e-83fc-46ea-9816-7e35ef75a674","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":1244703827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":1244703827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":1244703827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":1244703827}]},"public_addr":"172.21.15.149:6804/1244703827","cluster_addr":"172.21.15.149:6806/1244703827","heartbeat_back_addr":"172.21.15.149:6810/1244703827","heartbeat_front_addr":"172.21.15.149:6808/1244703827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.554731+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.453254+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.446266+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:35.709335+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.069934+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.433872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:07:49.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 10 librados: watch_flush enter 2024-09-16T05:07:49.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 10 librados: watch_flush exit 2024-09-16T05:07:49.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 20 client.4227.objecter shutdown clearing up homeless session... 2024-09-16T05:07:49.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 10 client.4227.objecter successfully canceled tick 2024-09-16T05:07:49.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 1 -- 172.21.15.114:0/3588961836 >> v2:172.21.15.149:6812/1185531135 conn(0x7f170c037840 msgr2=0x7f170c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 1 --2- 172.21.15.114:0/3588961836 >> v2:172.21.15.149:6812/1185531135 conn(0x7f170c037840 0x7f170c039d00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1724000f70 tx=0x7f1724007b60 comp rx=0 tx=0).stop 2024-09-16T05:07:49.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 10 monclient: shutdown 2024-09-16T05:07:49.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 1 -- 172.21.15.114:0/3588961836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17281398f0 msgr2=0x7f172811cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 1 --2- 172.21.15.114:0/3588961836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17281398f0 0x7f172811cb30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f172000c780 tx=0x7f172000cc40 comp rx=0 tx=0).stop 2024-09-16T05:07:49.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 1 -- 172.21.15.114:0/3588961836 shutdown_connections 2024-09-16T05:07:49.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 1 --2- 172.21.15.114:0/3588961836 >> v2:172.21.15.149:6812/1185531135 conn(0x7f170c037840 0x7f170c039d00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 1 --2- 172.21.15.114:0/3588961836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17281398f0 0x7f172811cb30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 1 --2- 172.21.15.114:0/3588961836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f17280a0ba0 0x7f17281cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 1 --2- 172.21.15.114:0/3588961836 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f172809f650 0x7f17281cf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 1 -- 172.21.15.114:0/3588961836 >> 172.21.15.114:0/3588961836 conn(0x7f17280a6310 msgr2=0x7f1728138fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 1 -- 172.21.15.114:0/3588961836 shutdown_connections 2024-09-16T05:07:49.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 1 -- 172.21.15.114:0/3588961836 wait complete. 2024-09-16T05:07:49.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.258+0000 7f172f9d3640 1 librados: shutdown 2024-09-16T05:07:49.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T05:07:49.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T05:07:49.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T05:07:49.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T05:07:49.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T05:07:49.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T05:07:49.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.386+0000 7f025baae640 10 monclient: get_monmap_and_config 2024-09-16T05:07:49.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.386+0000 7f025baae640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.386+0000 7f025baae640 10 monclient: monmap: 2024-09-16T05:07:49.391 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.391 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.391 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:49.390953+0000 2024-09-16T05:07:49.391 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:49.390953+0000 2024-09-16T05:07:49.391 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.391 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.392 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:49.392 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:49.392 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:49.392 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:49.392 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:49.392 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.386+0000 7f025baae640 1 Processor -- start 2024-09-16T05:07:49.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.386+0000 7f025baae640 1 -- start start 2024-09-16T05:07:49.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.386+0000 7f025baae640 10 monclient: init 2024-09-16T05:07:49.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.386+0000 7f025baae640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.386+0000 7f025baae640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-16T05:07:49.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.386+0000 7f025baae640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02541376e0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:49.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.386+0000 7f025baae640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f02541370a0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:49.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.386+0000 7f025baae640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f02540a5d00 0x7f0254137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025baae640 10 monclient(hunting): picked mon.noname-a con 0x7f02540a5d00 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:49.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025baae640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02540a5090 con 0x7f02541370a0 2024-09-16T05:07:49.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025baae640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f025409bd20 con 0x7f02541376e0 2024-09-16T05:07:49.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025baae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025baae640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f0254138000 con 0x7f02540a5d00 2024-09-16T05:07:49.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025baae640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025baae640 10 monclient(hunting): authenticate will time out at 954.020156s 2024-09-16T05:07:49.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f02540a5d00 0x7f0254137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f0259049640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f02541370a0 0x7f02540a4980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38552/0 (socket says 172.21.15.114:38552) 2024-09-16T05:07:49.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f0259049640 1 -- 172.21.15.114:0/2757611894 learned_addr learned my addr 172.21.15.114:0/2757611894 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:49.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 10 monclient(hunting): get_auth_request con 0x7f02540a5d00 auth_method 0 2024-09-16T05:07:49.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 10 monclient(hunting): handle_auth_done global_id 4230 payload 274 2024-09-16T05:07:49.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 1 -- 172.21.15.114:0/2757611894 >> v1:172.21.15.149:6789/0 conn(0x7f02541376e0 legacy=0x7f02540a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 1 -- 172.21.15.114:0/2757611894 >> v1:172.21.15.114:6789/0 conn(0x7f02541370a0 legacy=0x7f02540a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 1 monclient: found mon.noname-a 2024-09-16T05:07:49.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 1 -- 172.21.15.114:0/2757611894 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f025409c000 con 0x7f02540a5d00 2024-09-16T05:07:49.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025a04b640 1 --2- 172.21.15.114:0/2757611894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02540a5d00 0x7f0254137ac0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f024c009d90 tx=0x7f024c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc1d2785ebcdc42 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f0258848640 1 -- 172.21.15.114:0/2757611894 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c0b1070 con 0x7f02540a5d00 2024-09-16T05:07:49.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f0258848640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f0258848640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:49.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f0258848640 10 monclient: dump: 2024-09-16T05:07:49.401 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.401 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.401 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.401 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.402 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.402 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.402 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.402 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.402 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.402 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f0258848640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f0258848640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f0258848640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f0258848640 1 -- 172.21.15.114:0/2757611894 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f024c0a49d0 con 0x7f02540a5d00 2024-09-16T05:07:49.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f0258848640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f0258848640 1 -- 172.21.15.114:0/2757611894 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c0a4da0 con 0x7f02540a5d00 2024-09-16T05:07:49.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f0258848640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f0258848640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:49.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f0258848640 10 monclient: dump: 2024-09-16T05:07:49.404 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.404 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.406 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025baae640 5 monclient: authenticate success, global_id 4230 2024-09-16T05:07:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025baae640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:49.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f025baae640 4 set_mon_vals no callback set 2024-09-16T05:07:49.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f16ea668640 10 monclient: get_monmap_and_config 2024-09-16T05:07:49.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f16ea668640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 10 monclient: shutdown 2024-09-16T05:07:49.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.390+0000 7f16ea668640 10 monclient: monmap: 2024-09-16T05:07:49.408 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.408 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.409 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:49.396254+0000 2024-09-16T05:07:49.409 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:49.396254+0000 2024-09-16T05:07:49.409 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.409 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.409 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:49.409 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:49.409 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:49.409 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:49.410 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:49.410 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 -- 172.21.15.114:0/2757611894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02540a5d00 msgr2=0x7f0254137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 --2- 172.21.15.114:0/2757611894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02540a5d00 0x7f0254137ac0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f024c009d90 tx=0x7f024c01c430 comp rx=0 tx=0).stop 2024-09-16T05:07:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 -- 172.21.15.114:0/2757611894 shutdown_connections 2024-09-16T05:07:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 --2- 172.21.15.114:0/2757611894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02540a5d00 0x7f0254137ac0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 -- 172.21.15.114:0/2757611894 >> 172.21.15.114:0/2757611894 conn(0x7f02540a9a40 msgr2=0x7f02540a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 -- 172.21.15.114:0/2757611894 shutdown_connections 2024-09-16T05:07:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 -- 172.21.15.114:0/2757611894 wait complete. 2024-09-16T05:07:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16ea668640 1 Processor -- start 2024-09-16T05:07:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16ea668640 1 -- start start 2024-09-16T05:07:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16ea668640 10 monclient: init 2024-09-16T05:07:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16ea668640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16ea668640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T05:07:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16ea668640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f16e406bcc0 0x7f16e409f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16ea668640 10 monclient(hunting): picked mon.noname-a con 0x7f16e406bcc0 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16ea668640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16e406c0a0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 10 monclient: monmap: 2024-09-16T05:07:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:49.415 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16ea668640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16e413c7e0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f16e406bcc0 0x7f16e409f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e37fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f16e406c0a0 0x7f16e406c4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38560/0 (socket says 172.21.15.114:38560) 2024-09-16T05:07:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e37fe640 1 -- 172.21.15.114:0/1414000961 learned_addr learned my addr 172.21.15.114:0/1414000961 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 librados: starting msgr at 2024-09-16T05:07:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 librados: starting objecter 2024-09-16T05:07:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16ea668640 1 -- 172.21.15.114:0/1414000961 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16e406d120 con 0x7f16e406c0a0 2024-09-16T05:07:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16ea668640 1 -- 172.21.15.114:0/1414000961 --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16e406d740 con 0x7f16e413c7e0 2024-09-16T05:07:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16ea668640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 Processor -- start 2024-09-16T05:07:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16ea668640 1 -- 172.21.15.114:0/1414000961 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f16e409f5b0 con 0x7f16e406bcc0 2024-09-16T05:07:49.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16ea668640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e2ffd640 1 -- 172.21.15.114:0/1414000961 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1492798531 0 0) 0x7f16e406d120 con 0x7f16e406c0a0 2024-09-16T05:07:49.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16ea668640 10 monclient(hunting): authenticate will time out at 954.025986s 2024-09-16T05:07:49.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 -- start start 2024-09-16T05:07:49.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 librados: setting wanted keys 2024-09-16T05:07:49.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 librados: calling monclient init 2024-09-16T05:07:49.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 10 monclient: init 2024-09-16T05:07:49.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 10 monclient(hunting): get_auth_request con 0x7f16e406bcc0 auth_method 0 2024-09-16T05:07:49.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:07:49.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e2ffd640 10 monclient(hunting): my global_id is 4236 2024-09-16T05:07:49.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e2ffd640 1 -- 172.21.15.114:0/1414000961 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16cc003530 con 0x7f16e406c0a0 2024-09-16T05:07:49.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e2ffd640 1 -- 172.21.15.114:0/1414000961 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3485878899 0 0) 0x7f16e406d740 con 0x7f16e413c7e0 2024-09-16T05:07:49.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:07:49.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e2ffd640 10 monclient(hunting): my global_id is 4255 2024-09-16T05:07:49.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e2ffd640 1 -- 172.21.15.114:0/1414000961 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16e406d120 con 0x7f16e413c7e0 2024-09-16T05:07:49.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 10 monclient(hunting): handle_auth_done global_id 4239 payload 274 2024-09-16T05:07:49.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 1 -- 172.21.15.114:0/1414000961 >> v1:172.21.15.149:6789/0 conn(0x7f16e413c7e0 legacy=0x7f16e413ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 1 -- 172.21.15.114:0/1414000961 >> v1:172.21.15.114:6789/0 conn(0x7f16e406c0a0 legacy=0x7f16e406c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 1 monclient: found mon.noname-a 2024-09-16T05:07:49.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f16e3fff640 1 -- 172.21.15.114:0/1414000961 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e40a36a0 con 0x7f16e406bcc0 2024-09-16T05:07:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:07:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02540a5d00 0x7f02541d0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 10 monclient(hunting): picked mon.noname-c con 0x7f02540a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02541370a0 0x7f02541d0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 10 monclient(hunting): picked mon.noname-a con 0x7f02541370a0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f02541376e0 0x7f025413c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 10 monclient(hunting): picked mon.noname-b con 0x7f02541376e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0254143e80 con 0x7f02541370a0 2024-09-16T05:07:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02540a5d00 0x7f02541d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f0254143d00 con 0x7f02541376e0 2024-09-16T05:07:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f0254144000 con 0x7f02540a5d00 2024-09-16T05:07:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02540a5d00 0x7f02541d0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55460/0 (socket says 172.21.15.114:55460) 2024-09-16T05:07:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025baae640 10 monclient(hunting): authenticate will time out at 954.027209s 2024-09-16T05:07:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 1 -- 172.21.15.114:0/2455680133 learned_addr learned my addr 172.21.15.114:0/2455680133 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 10 monclient(hunting): get_auth_request con 0x7f02540a5d00 auth_method 0 2024-09-16T05:07:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025a04b640 1 --2- 172.21.15.114:0/2455680133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f02541376e0 0x7f025413c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025a04b640 10 monclient(hunting): get_auth_request con 0x7f02541376e0 auth_method 0 2024-09-16T05:07:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025a04b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025a04b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025a04b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 10 monclient(hunting): handle_auth_done global_id 4217 payload 824 2024-09-16T05:07:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 1 -- 172.21.15.114:0/2455680133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f02541376e0 msgr2=0x7f025413c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 1 --2- 172.21.15.114:0/2455680133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f02541376e0 0x7f025413c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 1 -- 172.21.15.114:0/2455680133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02541370a0 msgr2=0x7f02541d0960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:07:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 1 --2- 172.21.15.114:0/2455680133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02541370a0 0x7f02541d0960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 1 monclient: found mon.noname-c 2024-09-16T05:07:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.394+0000 7f025984a640 1 -- 172.21.15.114:0/2455680133 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f025413fac0 con 0x7f02540a5d00 2024-09-16T05:07:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 client.?.objecter ms_handle_connect 0x7f02540a5d00 2024-09-16T05:07:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025984a640 1 --2- 172.21.15.114:0/2455680133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02540a5d00 0x7f02541d0190 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f025000c410 tx=0x7f025000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 1 -- 172.21.15.114:0/2455680133 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0250017070 con 0x7f02540a5d00 2024-09-16T05:07:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 monclient: dump: 2024-09-16T05:07:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.431 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 1 -- 172.21.15.114:0/2455680133 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02500052d0 con 0x7f02540a5d00 2024-09-16T05:07:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 1 -- 172.21.15.114:0/2455680133 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02500054c0 con 0x7f02540a5d00 2024-09-16T05:07:49.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:49.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 monclient: dump: 2024-09-16T05:07:49.432 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.432 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.432 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.432 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.432 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.432 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.432 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.433 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.433 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.433 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025baae640 5 monclient: authenticate success, global_id 4217 2024-09-16T05:07:49.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025baae640 10 monclient: _renew_subs 2024-09-16T05:07:49.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025baae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025baae640 1 -- 172.21.15.114:0/2455680133 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f025413e8f0 con 0x7f02540a5d00 2024-09-16T05:07:49.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025baae640 10 client.4217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025baae640 10 monclient: _renew_subs 2024-09-16T05:07:49.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025baae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025baae640 1 -- 172.21.15.114:0/2455680133 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f025413eb10 con 0x7f02540a5d00 2024-09-16T05:07:49.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025baae640 1 librados: init done 2024-09-16T05:07:49.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025baae640 10 client.4217.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025baae640 15 client.4217.objecter _session_command_op_assign -1 1 2024-09-16T05:07:49.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025baae640 20 client.4217.objecter _get_session osd=-1 returning homeless 2024-09-16T05:07:49.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025baae640 10 client.4217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025baae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025baae640 1 -- 172.21.15.114:0/2455680133 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f02541367f0 con 0x7f02540a5d00 2024-09-16T05:07:49.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 1 -- 172.21.15.114:0/2455680133 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f025001a070 con 0x7f02540a5d00 2024-09-16T05:07:49.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16e3fff640 1 --2- 172.21.15.114:0/1414000961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16e406bcc0 0x7f16e409f070 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f16d40097c0 tx=0x7f16d401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd3aaf19baa009f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 1 --2- 172.21.15.114:0/2455680133 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0238037810 0x7f0238039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 1 -- 172.21.15.114:0/2455680133 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f0250047be0 con 0x7f02540a5d00 2024-09-16T05:07:49.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 client.4217.objecter ms_dispatch 0x7f0254000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:49.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 3 client.4217.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:49.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 3 client.4217.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:49.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16e2ffd640 1 -- 172.21.15.114:0/1414000961 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d40b0070 con 0x7f16e406bcc0 2024-09-16T05:07:49.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:49.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 client.4217.objecter checking command 1 2024-09-16T05:07:49.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16e2ffd640 10 monclient: dump: 2024-09-16T05:07:49.436 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.436 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.436 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.436 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.436 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.437 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.437 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.437 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.437 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.437 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16e2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16e2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 5 monclient: authenticate success, global_id 4239 2024-09-16T05:07:49.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:49.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16e2ffd640 1 -- 172.21.15.114:0/1414000961 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16d40ab600 con 0x7f16e406bcc0 2024-09-16T05:07:49.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16e2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16e2ffd640 1 -- 172.21.15.114:0/1414000961 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d40ab9b0 con 0x7f16e406bcc0 2024-09-16T05:07:49.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:49.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 4 set_mon_vals no callback set 2024-09-16T05:07:49.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:49.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16e2ffd640 10 monclient: dump: 2024-09-16T05:07:49.438 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.438 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.438 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.438 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.438 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.438 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.439 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.439 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.439 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.439 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 1 --2- 172.21.15.114:0/2455680133 >> v2:172.21.15.114:6808/269070361 conn(0x7f023803d2b0 0x7f023803f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 20 client.4217.objecter _get_session s=0x7f023803cc00 osd=0 3 2024-09-16T05:07:49.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 20 client.4217.objecter put_session s=0x7f023803cc00 osd=0 3 2024-09-16T05:07:49.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 15 client.4217.objecter _session_command_op_remove -1 1 2024-09-16T05:07:49.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 20 client.4217.objecter _get_session s=0x7f023803cc00 osd=0 3 2024-09-16T05:07:49.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 20 client.4217.objecter get_session s=0x7f023803cc00 osd=0 3 2024-09-16T05:07:49.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 15 client.4217.objecter _session_command_op_assign 0 1 2024-09-16T05:07:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 20 client.4217.objecter put_session s=0x7f023803cc00 osd=0 4 2024-09-16T05:07:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 client.4217.objecter _send_command 1 2024-09-16T05:07:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 1 -- 172.21.15.114:0/2455680133 --> v2:172.21.15.114:6808/269070361 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f025001ae50 con 0x7f023803d2b0 2024-09-16T05:07:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 20 client.4217.objecter dump_active .. 0 homeless 2024-09-16T05:07:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 1 -- 172.21.15.114:0/2455680133 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f025004c220 con 0x7f02540a5d00 2024-09-16T05:07:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T05:07:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024bfff640 4 set_mon_vals no callback set 2024-09-16T05:07:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025a04b640 1 --2- 172.21.15.114:0/2455680133 >> v2:172.21.15.114:6808/269070361 conn(0x7f023803d2b0 0x7f023803f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f0259049640 1 --2- 172.21.15.114:0/2455680133 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0238037810 0x7f0238039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025a04b640 10 monclient: get_auth_request con 0x7f023803d2b0 auth_method 0 2024-09-16T05:07:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f0259049640 10 monclient: get_auth_request con 0x7f0238037810 auth_method 0 2024-09-16T05:07:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 client.4217.objecter ms_handle_connect 0x7f023803d2b0 2024-09-16T05:07:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 10 monclient: shutdown 2024-09-16T05:07:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 1 -- 172.21.15.114:0/1414000961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16e406bcc0 msgr2=0x7f16e409f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 1 --2- 172.21.15.114:0/1414000961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16e406bcc0 0x7f16e409f070 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f16d40097c0 tx=0x7f16d401c360 comp rx=0 tx=0).stop 2024-09-16T05:07:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 1 -- 172.21.15.114:0/1414000961 shutdown_connections 2024-09-16T05:07:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 1 --2- 172.21.15.114:0/1414000961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16e406bcc0 0x7f16e409f070 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 1 -- 172.21.15.114:0/1414000961 >> 172.21.15.114:0/1414000961 conn(0x7f16e40a4140 msgr2=0x7f16e40a4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f025a04b640 1 --2- 172.21.15.114:0/2455680133 >> v2:172.21.15.114:6808/269070361 conn(0x7f023803d2b0 0x7f023803f6f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f024a7fc640 10 client.4217.objecter ms_handle_connect 0x7f0238037810 2024-09-16T05:07:49.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f0259049640 1 --2- 172.21.15.114:0/2455680133 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0238037810 0x7f0238039cd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f025413d990 tx=0x7f02440079b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 1 -- 172.21.15.114:0/1414000961 shutdown_connections 2024-09-16T05:07:49.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 1 -- 172.21.15.114:0/1414000961 wait complete. 2024-09-16T05:07:49.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 10 monclient: monmap: 2024-09-16T05:07:49.445 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.445 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:49.445 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:49.445 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:49.445 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.445 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.445 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:49.446 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:49.450 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 1 librados: starting msgr at 2024-09-16T05:07:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.398+0000 7f16ea668640 1 librados: starting objecter 2024-09-16T05:07:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 1 Processor -- start 2024-09-16T05:07:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f024a7fc640 1 -- 172.21.15.114:0/2455680133 <== osd.0 v2:172.21.15.114:6808/269070361 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f025001ae50 con 0x7f023803d2b0 2024-09-16T05:07:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f024a7fc640 10 client.4217.objecter ms_dispatch 0x7f0254000ba0 command_reply(tid 1: 0 ) 2024-09-16T05:07:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f024a7fc640 10 client.4217.objecter _finish_command 1 = system:0 2024-09-16T05:07:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f024a7fc640 20 client.4217.objecter put_session s=0x7f023803cc00 osd=0 4 2024-09-16T05:07:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f024a7fc640 15 client.4217.objecter _session_command_op_remove 0 1 2024-09-16T05:07:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 1 -- start start 2024-09-16T05:07:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 1 librados: setting wanted keys 2024-09-16T05:07:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 1 librados: calling monclient init 2024-09-16T05:07:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient: init 2024-09-16T05:07:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:07:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f16e406bcc0 0x7f16e413c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient(hunting): picked mon.noname-b con 0x7f16e406bcc0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16e406c0a0 0x7f16e4071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient(hunting): picked mon.noname-c con 0x7f16e406c0a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16e413c7e0 0x7f16e40723a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient(hunting): picked mon.noname-a con 0x7f16e413c7e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f16e4142ea0 con 0x7f16e413c7e0 2024-09-16T05:07:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f16e4142d20 con 0x7f16e406bcc0 2024-09-16T05:07:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f16e4143020 con 0x7f16e406c0a0 2024-09-16T05:07:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient(hunting): authenticate will time out at 954.032724s 2024-09-16T05:07:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e8c05640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16e413c7e0 0x7f16e40723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e8c05640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16e413c7e0 0x7f16e40723a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54788/0 (socket says 172.21.15.114:54788) 2024-09-16T05:07:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e8c05640 1 -- 172.21.15.114:0/1341292424 learned_addr learned my addr 172.21.15.114:0/1341292424 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 1 --2- 172.21.15.114:0/1341292424 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16e406c0a0 0x7f16e4071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 10 monclient(hunting): get_auth_request con 0x7f16e406c0a0 auth_method 0 2024-09-16T05:07:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e8c05640 10 monclient(hunting): get_auth_request con 0x7f16e413c7e0 auth_method 0 2024-09-16T05:07:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e8c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e8c05640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e8c05640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e3fff640 1 --2- 172.21.15.114:0/1341292424 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f16e406bcc0 0x7f16e413c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e8c05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e8c05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e8c05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e3fff640 10 monclient(hunting): get_auth_request con 0x7f16e406bcc0 auth_method 0 2024-09-16T05:07:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 10 monclient(hunting): handle_auth_done global_id 4220 payload 824 2024-09-16T05:07:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 1 -- 172.21.15.114:0/1341292424 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f16e406bcc0 msgr2=0x7f16e413c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 1 --2- 172.21.15.114:0/1341292424 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f16e406bcc0 0x7f16e413c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 1 -- 172.21.15.114:0/1341292424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16e413c7e0 msgr2=0x7f16e40723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 1 --2- 172.21.15.114:0/1341292424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16e413c7e0 0x7f16e40723a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 1 monclient: found mon.noname-c 2024-09-16T05:07:49.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 1 -- 172.21.15.114:0/1341292424 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e408e100 con 0x7f16e406c0a0 2024-09-16T05:07:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e37fe640 1 --2- 172.21.15.114:0/1341292424 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16e406c0a0 0x7f16e4071e60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f16d8001140 tx=0x7f16d800c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 10 client.?.objecter ms_handle_connect 0x7f16e406c0a0 2024-09-16T05:07:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 1 -- 172.21.15.114:0/1341292424 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d8017070 con 0x7f16e406c0a0 2024-09-16T05:07:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 10 monclient: dump: 2024-09-16T05:07:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.460 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.460 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.460 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.460 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.460 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.460 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.460 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 5 monclient: authenticate success, global_id 4220 2024-09-16T05:07:49.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient: _renew_subs 2024-09-16T05:07:49.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 1 -- 172.21.15.114:0/1341292424 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16e413ff10 con 0x7f16e406c0a0 2024-09-16T05:07:49.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 1 -- 172.21.15.114:0/1341292424 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16d8005cf0 con 0x7f16e406c0a0 2024-09-16T05:07:49.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 1 -- 172.21.15.114:0/1341292424 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d80072d0 con 0x7f16e406c0a0 2024-09-16T05:07:49.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 10 monclient: dump: 2024-09-16T05:07:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.462 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient: _renew_subs 2024-09-16T05:07:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 1 -- 172.21.15.114:0/1341292424 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f16e408db40 con 0x7f16e406c0a0 2024-09-16T05:07:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e27fc640 4 set_mon_vals no callback set 2024-09-16T05:07:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 1 librados: init done 2024-09-16T05:07:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 1 -- 172.21.15.114:0/1341292424 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f16d801a070 con 0x7f16e406c0a0 2024-09-16T05:07:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 1 --2- 172.21.15.114:0/1341292424 >> v2:172.21.15.149:6812/1185531135 conn(0x7f16d0037810 0x7f16d0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 1 -- 172.21.15.114:0/1341292424 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f16d804b120 con 0x7f16e406c0a0 2024-09-16T05:07:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 10 client.4220.objecter ms_dispatch 0x7f16e4000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 client.4220.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 15 client.4220.objecter _session_command_op_assign -1 1 2024-09-16T05:07:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 20 client.4220.objecter _get_session osd=-1 returning homeless 2024-09-16T05:07:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16ea668640 1 -- 172.21.15.114:0/1341292424 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f16e4073720 con 0x7f16e406c0a0 2024-09-16T05:07:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 3 client.4220.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 3 client.4220.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 10 client.4220.objecter checking command 1 2024-09-16T05:07:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 1 --2- 172.21.15.114:0/1341292424 >> v2:172.21.15.149:6804/1244703827 conn(0x7f16d003d320 0x7f16d003f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 20 client.4220.objecter _get_session s=0x7f16d003cc70 osd=5 3 2024-09-16T05:07:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 20 client.4220.objecter put_session s=0x7f16d003cc70 osd=5 3 2024-09-16T05:07:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 15 client.4220.objecter _session_command_op_remove -1 1 2024-09-16T05:07:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 20 client.4220.objecter _get_session s=0x7f16d003cc70 osd=5 3 2024-09-16T05:07:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 20 client.4220.objecter get_session s=0x7f16d003cc70 osd=5 3 2024-09-16T05:07:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 15 client.4220.objecter _session_command_op_assign 5 1 2024-09-16T05:07:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 20 client.4220.objecter put_session s=0x7f16d003cc70 osd=5 4 2024-09-16T05:07:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 10 client.4220.objecter _send_command 1 2024-09-16T05:07:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 1 -- 172.21.15.114:0/1341292424 --> v2:172.21.15.149:6804/1244703827 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16d8014ec0 con 0x7f16d003d320 2024-09-16T05:07:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 20 client.4220.objecter dump_active .. 0 homeless 2024-09-16T05:07:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.402+0000 7f16e0ff9640 1 -- 172.21.15.114:0/1341292424 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f16d804d060 con 0x7f16e406c0a0 2024-09-16T05:07:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.406+0000 7f16e0ff9640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T05:07:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.406+0000 7f16e3fff640 1 --2- 172.21.15.114:0/1341292424 >> v2:172.21.15.149:6812/1185531135 conn(0x7f16d0037810 0x7f16d0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.406+0000 7f16e3fff640 10 monclient: get_auth_request con 0x7f16d0037810 auth_method 0 2024-09-16T05:07:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.406+0000 7f16e8c05640 1 --2- 172.21.15.114:0/1341292424 >> v2:172.21.15.149:6804/1244703827 conn(0x7f16d003d320 0x7f16d003f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.406+0000 7f16e8c05640 10 monclient: get_auth_request con 0x7f16d003d320 auth_method 0 2024-09-16T05:07:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.406+0000 7f16e3fff640 1 --2- 172.21.15.114:0/1341292424 >> v2:172.21.15.149:6812/1185531135 conn(0x7f16d0037810 0x7f16d0039cd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f16d40ad000 tx=0x7f16d4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.406+0000 7f16e0ff9640 10 client.4220.objecter ms_handle_connect 0x7f16d0037810 2024-09-16T05:07:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.406+0000 7f16e0ff9640 10 client.4220.objecter ms_handle_connect 0x7f16d003d320 2024-09-16T05:07:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.406+0000 7f16e8c05640 1 --2- 172.21.15.114:0/1341292424 >> v2:172.21.15.149:6804/1244703827 conn(0x7f16d003d320 0x7f16d003f760 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.406+0000 7f16e0ff9640 1 -- 172.21.15.114:0/1341292424 <== osd.5 v2:172.21.15.149:6804/1244703827 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16d8014ec0 con 0x7f16d003d320 2024-09-16T05:07:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.406+0000 7f16e0ff9640 10 client.4220.objecter ms_dispatch 0x7f16e4000ba0 command_reply(tid 1: 0 ) 2024-09-16T05:07:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.406+0000 7f16e0ff9640 10 client.4220.objecter _finish_command 1 = system:0 2024-09-16T05:07:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.406+0000 7f16e0ff9640 20 client.4220.objecter put_session s=0x7f16d003cc70 osd=5 4 2024-09-16T05:07:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.406+0000 7f16e0ff9640 15 client.4220.objecter _session_command_op_remove 5 1 2024-09-16T05:07:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.414+0000 7f025baae640 10 client.4217.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T05:07:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.414+0000 7f025baae640 15 client.4217.objecter _session_command_op_assign -1 2 2024-09-16T05:07:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.414+0000 7f025baae640 20 client.4217.objecter _get_session s=0x7f023803cc00 osd=0 3 2024-09-16T05:07:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.414+0000 7f025baae640 20 client.4217.objecter put_session s=0x7f023803cc00 osd=0 3 2024-09-16T05:07:49.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.414+0000 7f025baae640 20 client.4217.objecter _get_session s=0x7f023803cc00 osd=0 3 2024-09-16T05:07:49.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.414+0000 7f025baae640 15 client.4217.objecter _session_command_op_remove -1 2 2024-09-16T05:07:49.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.414+0000 7f025baae640 20 client.4217.objecter get_session s=0x7f023803cc00 osd=0 3 2024-09-16T05:07:49.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.414+0000 7f025baae640 15 client.4217.objecter _session_command_op_assign 0 2 2024-09-16T05:07:49.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.414+0000 7f025baae640 20 client.4217.objecter put_session s=0x7f023803cc00 osd=0 4 2024-09-16T05:07:49.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.414+0000 7f025baae640 10 client.4217.objecter _send_command 2 2024-09-16T05:07:49.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.414+0000 7f025baae640 1 -- 172.21.15.114:0/2455680133 --> v2:172.21.15.114:6808/269070361 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f025413fd00 con 0x7f023803d2b0 2024-09-16T05:07:49.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f024a7fc640 1 -- 172.21.15.114:0/2455680133 <== osd.0 v2:172.21.15.114:6808/269070361 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f025413fd00 con 0x7f023803d2b0 2024-09-16T05:07:49.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f024a7fc640 10 client.4217.objecter ms_dispatch 0x7f0254000ba0 command_reply(tid 2: 0 ) 2024-09-16T05:07:49.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f024a7fc640 10 client.4217.objecter _finish_command 2 = system:0 2024-09-16T05:07:49.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f024a7fc640 20 client.4217.objecter put_session s=0x7f023803cc00 osd=0 4 2024-09-16T05:07:49.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f024a7fc640 15 client.4217.objecter _session_command_op_remove 0 2 2024-09-16T05:07:49.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f025baae640 10 librados: watch_flush enter 2024-09-16T05:07:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f025baae640 10 librados: watch_flush exit 2024-09-16T05:07:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f025baae640 10 client.4217.objecter close_session for osd.0 2024-09-16T05:07:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f025baae640 1 -- 172.21.15.114:0/2455680133 >> v2:172.21.15.114:6808/269070361 conn(0x7f023803d2b0 msgr2=0x7f023803f6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f025baae640 1 --2- 172.21.15.114:0/2455680133 >> v2:172.21.15.114:6808/269070361 conn(0x7f023803d2b0 0x7f023803f6f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f025baae640 20 client.4217.objecter put_session s=0x7f023803cc00 osd=0 1 2024-09-16T05:07:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f025baae640 20 client.4217.objecter shutdown clearing up homeless session... 2024-09-16T05:07:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f025baae640 10 client.4217.objecter successfully canceled tick 2024-09-16T05:07:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f025baae640 1 -- 172.21.15.114:0/2455680133 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0238037810 msgr2=0x7f0238039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f025baae640 1 --2- 172.21.15.114:0/2455680133 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0238037810 0x7f0238039cd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f025413d990 tx=0x7f02440079b0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f025baae640 10 monclient: shutdown 2024-09-16T05:07:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f025baae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f025baae640 1 -- 172.21.15.114:0/2455680133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02540a5d00 msgr2=0x7f02541d0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f025baae640 1 --2- 172.21.15.114:0/2455680133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02540a5d00 0x7f02541d0190 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f025000c410 tx=0x7f025000c8d0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.418+0000 7f025a04b640 1 -- 172.21.15.114:0/2455680133 reap_dead start 2024-09-16T05:07:49.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f025baae640 1 -- 172.21.15.114:0/2455680133 shutdown_connections 2024-09-16T05:07:49.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f025baae640 1 -- 172.21.15.114:0/2455680133 >> 172.21.15.114:0/2455680133 conn(0x7f02540a9a40 msgr2=0x7f02540a80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f025baae640 1 -- 172.21.15.114:0/2455680133 shutdown_connections 2024-09-16T05:07:49.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f025baae640 1 -- 172.21.15.114:0/2455680133 wait complete. 2024-09-16T05:07:49.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f025baae640 1 librados: shutdown 2024-09-16T05:07:49.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 10 client.4220.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T05:07:49.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 15 client.4220.objecter _session_command_op_assign -1 2 2024-09-16T05:07:49.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 20 client.4220.objecter _get_session s=0x7f16d003cc70 osd=5 3 2024-09-16T05:07:49.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 20 client.4220.objecter put_session s=0x7f16d003cc70 osd=5 3 2024-09-16T05:07:49.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 20 client.4220.objecter _get_session s=0x7f16d003cc70 osd=5 3 2024-09-16T05:07:49.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 15 client.4220.objecter _session_command_op_remove -1 2 2024-09-16T05:07:49.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 20 client.4220.objecter get_session s=0x7f16d003cc70 osd=5 3 2024-09-16T05:07:49.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 15 client.4220.objecter _session_command_op_assign 5 2 2024-09-16T05:07:49.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 20 client.4220.objecter put_session s=0x7f16d003cc70 osd=5 4 2024-09-16T05:07:49.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 10 client.4220.objecter _send_command 2 2024-09-16T05:07:49.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 1 -- 172.21.15.114:0/1341292424 --> v2:172.21.15.149:6804/1244703827 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f16e409f070 con 0x7f16d003d320 2024-09-16T05:07:49.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16e0ff9640 1 -- 172.21.15.114:0/1341292424 <== osd.5 v2:172.21.15.149:6804/1244703827 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f16e409f070 con 0x7f16d003d320 2024-09-16T05:07:49.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16e0ff9640 10 client.4220.objecter ms_dispatch 0x7f16e4000ba0 command_reply(tid 2: 0 ) 2024-09-16T05:07:49.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16e0ff9640 10 client.4220.objecter _finish_command 2 = system:0 2024-09-16T05:07:49.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16e0ff9640 20 client.4220.objecter put_session s=0x7f16d003cc70 osd=5 4 2024-09-16T05:07:49.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16e0ff9640 15 client.4220.objecter _session_command_op_remove 5 2 2024-09-16T05:07:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 10 librados: watch_flush enter 2024-09-16T05:07:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 10 librados: watch_flush exit 2024-09-16T05:07:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 10 client.4220.objecter close_session for osd.5 2024-09-16T05:07:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 1 -- 172.21.15.114:0/1341292424 >> v2:172.21.15.149:6804/1244703827 conn(0x7f16d003d320 msgr2=0x7f16d003f760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 1 --2- 172.21.15.114:0/1341292424 >> v2:172.21.15.149:6804/1244703827 conn(0x7f16d003d320 0x7f16d003f760 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 20 client.4220.objecter put_session s=0x7f16d003cc70 osd=5 1 2024-09-16T05:07:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 20 client.4220.objecter shutdown clearing up homeless session... 2024-09-16T05:07:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 10 client.4220.objecter successfully canceled tick 2024-09-16T05:07:49.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 1 -- 172.21.15.114:0/1341292424 >> v2:172.21.15.149:6812/1185531135 conn(0x7f16d0037810 msgr2=0x7f16d0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 1 --2- 172.21.15.114:0/1341292424 >> v2:172.21.15.149:6812/1185531135 conn(0x7f16d0037810 0x7f16d0039cd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f16d40ad000 tx=0x7f16d4008040 comp rx=0 tx=0).stop 2024-09-16T05:07:49.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 10 monclient: shutdown 2024-09-16T05:07:49.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 1 -- 172.21.15.114:0/1341292424 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16e406c0a0 msgr2=0x7f16e4071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 1 --2- 172.21.15.114:0/1341292424 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16e406c0a0 0x7f16e4071e60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f16d8001140 tx=0x7f16d800c960 comp rx=0 tx=0).stop 2024-09-16T05:07:49.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16e8c05640 1 -- 172.21.15.114:0/1341292424 reap_dead start 2024-09-16T05:07:49.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.422+0000 7f16ea668640 1 -- 172.21.15.114:0/1341292424 shutdown_connections 2024-09-16T05:07:49.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.426+0000 7f16ea668640 1 -- 172.21.15.114:0/1341292424 >> 172.21.15.114:0/1341292424 conn(0x7f16e40a4140 msgr2=0x7f16e413ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.426+0000 7f16ea668640 1 -- 172.21.15.114:0/1341292424 shutdown_connections 2024-09-16T05:07:49.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.426+0000 7f16ea668640 1 -- 172.21.15.114:0/1341292424 wait complete. 2024-09-16T05:07:49.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.426+0000 7f16ea668640 1 librados: shutdown 2024-09-16T05:07:49.480 INFO:teuthology.orchestra.run.smithi114.stdout:55834574852 2024-09-16T05:07:49.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T05:07:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 10 monclient: get_monmap_and_config 2024-09-16T05:07:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 10 monclient: monmap: 2024-09-16T05:07:49.481 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.482 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.482 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:49.448431+0000 2024-09-16T05:07:49.482 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:49.448431+0000 2024-09-16T05:07:49.482 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.482 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.482 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:49.482 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:49.482 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:49.482 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:49.482 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:49.483 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 1 Processor -- start 2024-09-16T05:07:49.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 1 -- start start 2024-09-16T05:07:49.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 10 monclient: init 2024-09-16T05:07:49.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f901c25a640 10 monclient: get_monmap_and_config 2024-09-16T05:07:49.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f901c25a640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f901c25a640 10 monclient: monmap: 2024-09-16T05:07:49.483 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.483 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.484 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:49.449299+0000 2024-09-16T05:07:49.484 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:49.449299+0000 2024-09-16T05:07:49.484 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.484 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.484 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:49.484 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:49.484 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:49.484 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:49.484 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:49.484 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-16T05:07:49.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b000a4270 0x7f2b0006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 10 monclient(hunting): picked mon.noname-b con 0x7f2b000a4270 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.486 INFO:teuthology.orchestra.run.smithi114.stdout:51539607557 2024-09-16T05:07:49.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T05:07:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b000a4650 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b00140ab0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b00073370 con 0x7f2b00140ab0 2024-09-16T05:07:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b000a52f0 con 0x7f2b000a4650 2024-09-16T05:07:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2b0006eab0 con 0x7f2b000a4270 2024-09-16T05:07:49.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b07aa3640 10 monclient(hunting): authenticate will time out at 954.077586s 2024-09-16T05:07:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b000a4270 0x7f2b0006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b06040640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2b00140ab0 0x7f2b00142ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38574/0 (socket says 172.21.15.114:38574) 2024-09-16T05:07:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b06040640 1 -- 172.21.15.114:0/2794729927 learned_addr learned my addr 172.21.15.114:0/2794729927 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 10 monclient(hunting): get_auth_request con 0x7f2b000a4270 auth_method 0 2024-09-16T05:07:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2024-09-16T05:07:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 1 -- 172.21.15.114:0/2794729927 >> v1:172.21.15.149:6789/0 conn(0x7f2b000a4650 legacy=0x7f2b000a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0483d640 1 -- 172.21.15.114:0/2794729927 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3229662384 0 0) 0x7f2b00073370 con 0x7f2b00140ab0 2024-09-16T05:07:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 1 -- 172.21.15.114:0/2794729927 >> v1:172.21.15.114:6789/0 conn(0x7f2b00140ab0 legacy=0x7f2b00142ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 1 monclient: found mon.noname-b 2024-09-16T05:07:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0583f640 1 -- 172.21.15.114:0/2794729927 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0006e180 con 0x7f2b000a4270 2024-09-16T05:07:49.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f2b0483d640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:07:49.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7c17db640 10 monclient: get_monmap_and_config 2024-09-16T05:07:49.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7c17db640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7c17db640 10 monclient: monmap: 2024-09-16T05:07:49.491 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.491 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.491 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:49.450773+0000 2024-09-16T05:07:49.492 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:49.450773+0000 2024-09-16T05:07:49.492 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.492 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.492 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:49.492 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:49.492 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:49.492 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:49.492 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:49.492 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7c17db640 1 Processor -- start 2024-09-16T05:07:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7c17db640 1 -- start start 2024-09-16T05:07:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7c17db640 10 monclient: init 2024-09-16T05:07:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7c17db640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7c17db640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-16T05:07:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f901c25a640 1 Processor -- start 2024-09-16T05:07:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7c17db640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff7bc1371d0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7c17db640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff7bc0a9220 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f901c25a640 1 -- start start 2024-09-16T05:07:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7f901c25a640 10 monclient: init 2024-09-16T05:07:49.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7c17db640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7bc137cc0 0x7ff7bc13fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7fb9ff395640 10 monclient: get_monmap_and_config 2024-09-16T05:07:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7baffd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff7bc1371d0 0x7ff7bc1375b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38576/0 (socket says 172.21.15.114:38576) 2024-09-16T05:07:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7c17db640 10 monclient(hunting): picked mon.noname-b con 0x7ff7bc137cc0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7baffd640 1 -- 172.21.15.114:0/1176145192 learned_addr learned my addr 172.21.15.114:0/1176145192 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7bb7fe640 1 --2- 172.21.15.114:0/1176145192 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7bc137cc0 0x7ff7bc13fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7c17db640 1 -- 172.21.15.114:0/1176145192 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7bc073330 con 0x7ff7bc1371d0 2024-09-16T05:07:49.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7fb9ff395640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7fb9ff395640 10 monclient: monmap: 2024-09-16T05:07:49.497 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.497 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.498 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:49.452022+0000 2024-09-16T05:07:49.498 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:49.452022+0000 2024-09-16T05:07:49.498 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.498 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.498 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:49.498 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:49.498 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:49.499 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:49.499 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:49.499 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.446+0000 7ff7c17db640 1 -- 172.21.15.114:0/1176145192 --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7bc0a8820 con 0x7ff7bc0a9220 2024-09-16T05:07:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7c17db640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7c17db640 1 -- 172.21.15.114:0/1176145192 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff7bc09bf60 con 0x7ff7bc137cc0 2024-09-16T05:07:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7c17db640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7c17db640 10 monclient(hunting): authenticate will time out at 954.080089s 2024-09-16T05:07:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 10 monclient(hunting): get_auth_request con 0x7ff7bc137cc0 auth_method 0 2024-09-16T05:07:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 1 -- 172.21.15.114:0/1176145192 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2090774587 0 0) 0x7ff7bc073330 con 0x7ff7bc1371d0 2024-09-16T05:07:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:07:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 10 monclient(hunting): my global_id is 4248 2024-09-16T05:07:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 1 -- 172.21.15.114:0/1176145192 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7a8003530 con 0x7ff7bc1371d0 2024-09-16T05:07:49.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901c25a640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901c25a640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-16T05:07:49.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901c25a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f901409cdb0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:49.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901c25a640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f901409d190 0x7f901413a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901c25a640 10 monclient(hunting): picked mon.noname-c con 0x7f901409d190 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:49.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901c25a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f901413a700 0x7f901413caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901c25a640 10 monclient(hunting): picked mon.noname-a con 0x7f901413a700 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901c25a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9014132410 con 0x7f901409cdb0 2024-09-16T05:07:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901c25a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901c25a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f901409d600 con 0x7f901413a700 2024-09-16T05:07:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901c25a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901c25a640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f90140766a0 con 0x7f901409d190 2024-09-16T05:07:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901c25a640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901c25a640 10 monclient(hunting): authenticate will time out at 954.080401s 2024-09-16T05:07:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 10 monclient(hunting): handle_auth_done global_id 4226 payload 274 2024-09-16T05:07:49.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 1 -- 172.21.15.114:0/1176145192 >> v1:172.21.15.149:6789/0 conn(0x7ff7bc0a9220 legacy=0x7ff7bc13c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 1 -- 172.21.15.114:0/1176145192 >> v1:172.21.15.114:6789/0 conn(0x7ff7bc1371d0 legacy=0x7ff7bc1375b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 1 monclient: found mon.noname-b 2024-09-16T05:07:49.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 1 -- 172.21.15.114:0/1176145192 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7bc09bc80 con 0x7ff7bc137cc0 2024-09-16T05:07:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f9019ff6640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f901409cdb0 0x7f9014131d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38584/0 (socket says 172.21.15.114:38584) 2024-09-16T05:07:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f9019ff6640 1 -- 172.21.15.114:0/223574415 learned_addr learned my addr 172.21.15.114:0/223574415 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 1 --2- 172.21.15.114:0/223574415 >> v2:172.21.15.114:3300/0 conn(0x7f901413a700 0x7f901413caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 10 monclient(hunting): get_auth_request con 0x7f901413a700 auth_method 0 2024-09-16T05:07:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f90197f5640 1 --2- 172.21.15.114:0/223574415 >> v2:172.21.15.149:3300/0 conn(0x7f901409d190 0x7f901413a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f90197f5640 10 monclient(hunting): get_auth_request con 0x7f901409d190 auth_method 0 2024-09-16T05:07:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f90197f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f90197f5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f90197f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b0583f640 1 --2- 172.21.15.114:0/2794729927 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b000a4270 0x7f2b0006ee40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2afc0097f0 tx=0x7f2afc01c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd0f83b456e4e2eb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f90197f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f90197f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f90197f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 10 monclient(hunting): handle_auth_done global_id 4251 payload 274 2024-09-16T05:07:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f9018ff4640 1 -- 172.21.15.114:0/223574415 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 188540549 0 0) 0x7f9014132410 con 0x7f901409cdb0 2024-09-16T05:07:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 1 -- 172.21.15.114:0/223574415 >> v2:172.21.15.149:3300/0 conn(0x7f901409d190 msgr2=0x7f901413a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 1 --2- 172.21.15.114:0/223574415 >> v2:172.21.15.149:3300/0 conn(0x7f901409d190 0x7f901413a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 1 -- 172.21.15.114:0/223574415 >> v1:172.21.15.114:6789/0 conn(0x7f901409cdb0 legacy=0x7f9014131d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 1 monclient: found mon.noname-a 2024-09-16T05:07:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 1 -- 172.21.15.114:0/223574415 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f901409a640 con 0x7f901413a700 2024-09-16T05:07:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f9018ff4640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:07:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b0483d640 1 -- 172.21.15.114:0/2794729927 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc0b0070 con 0x7f2b000a4270 2024-09-16T05:07:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b0483d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b0483d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b0483d640 10 monclient: dump: 2024-09-16T05:07:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.512 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.513 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.513 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.513 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.513 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.513 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.513 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.513 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.513 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b0483d640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b0483d640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b0483d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 5 monclient: authenticate success, global_id 4223 2024-09-16T05:07:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b0483d640 1 -- 172.21.15.114:0/2794729927 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2afc0ab600 con 0x7f2b000a4270 2024-09-16T05:07:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b0483d640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b0483d640 1 -- 172.21.15.114:0/2794729927 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc0ab9d0 con 0x7f2b000a4270 2024-09-16T05:07:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b0483d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 4 set_mon_vals no callback set 2024-09-16T05:07:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b0483d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b0483d640 10 monclient: dump: 2024-09-16T05:07:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.515 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.515 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.515 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.515 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.515 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.515 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.515 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9ff395640 1 Processor -- start 2024-09-16T05:07:49.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9ff395640 1 -- start start 2024-09-16T05:07:49.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9ff395640 10 monclient: init 2024-09-16T05:07:49.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 10 monclient: shutdown 2024-09-16T05:07:49.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 1 -- 172.21.15.114:0/2794729927 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b000a4270 msgr2=0x7f2b0006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 1 --2- 172.21.15.114:0/2794729927 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b000a4270 0x7f2b0006ee40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2afc0097f0 tx=0x7f2afc01c3f0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 1 -- 172.21.15.114:0/2794729927 shutdown_connections 2024-09-16T05:07:49.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 1 --2- 172.21.15.114:0/2794729927 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b000a4270 0x7f2b0006ee40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 1 -- 172.21.15.114:0/2794729927 >> 172.21.15.114:0/2794729927 conn(0x7f2b000a9a40 msgr2=0x7f2b000a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 1 -- 172.21.15.114:0/2794729927 shutdown_connections 2024-09-16T05:07:49.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9ff395640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9ff395640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-16T05:07:49.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9ff395640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb9f81376e0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:49.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 1 -- 172.21.15.114:0/2794729927 wait complete. 2024-09-16T05:07:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9ff395640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb9f81370a0 0x7fb9f8137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9ff395640 10 monclient(hunting): picked mon.noname-c con 0x7fb9f81370a0 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9ff395640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb9f80a5d00 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9ff395640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9f80a5090 con 0x7fb9f81376e0 2024-09-16T05:07:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9ff395640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9f809bd20 con 0x7fb9f80a5d00 2024-09-16T05:07:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9ff395640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9ff395640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fb9f8138000 con 0x7fb9f81370a0 2024-09-16T05:07:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9ff395640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9ff395640 10 monclient(hunting): authenticate will time out at 954.082079s 2024-09-16T05:07:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd131640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb9f81376e0 0x7fb9f80a4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38592/0 (socket says 172.21.15.114:38592) 2024-09-16T05:07:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd131640 1 -- 172.21.15.114:0/1511809488 learned_addr learned my addr 172.21.15.114:0/1511809488 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fc930640 1 --2- 172.21.15.114:0/1511809488 >> v2:172.21.15.149:3300/0 conn(0x7fb9f81370a0 0x7fb9f8137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fc930640 10 monclient(hunting): get_auth_request con 0x7fb9f81370a0 auth_method 0 2024-09-16T05:07:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fc930640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fc930640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fc930640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fc930640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fc930640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 -- 172.21.15.114:0/1511809488 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2288311071 0 0) 0x7fb9f80a5090 con 0x7fb9f81376e0 2024-09-16T05:07:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fc930640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:07:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient(hunting): my global_id is 4257 2024-09-16T05:07:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 -- 172.21.15.114:0/1511809488 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9e0003530 con 0x7fb9f81376e0 2024-09-16T05:07:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 -- 172.21.15.114:0/1511809488 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1078008795 0 0) 0x7fb9e0003530 con 0x7fb9f81376e0 2024-09-16T05:07:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 -- 172.21.15.114:0/1511809488 >> v2:172.21.15.149:3300/0 conn(0x7fb9f81370a0 msgr2=0x7fb9f8137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 --2- 172.21.15.114:0/1511809488 >> v2:172.21.15.149:3300/0 conn(0x7fb9f81370a0 0x7fb9f8137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 -- 172.21.15.114:0/1511809488 >> v1:172.21.15.149:6789/0 conn(0x7fb9f80a5d00 legacy=0x7fb9f80a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:07:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:07:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 -- 172.21.15.114:0/1511809488 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f809c000 con 0x7fb9f81376e0 2024-09-16T05:07:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 -- 172.21.15.114:0/1511809488 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7fb9f0001a00 con 0x7fb9f81376e0 2024-09-16T05:07:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:07:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient: dump: 2024-09-16T05:07:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.524 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.524 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.524 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.524 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.524 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.524 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.524 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:07:49.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 -- 172.21.15.114:0/1511809488 >> v1:172.21.15.114:6789/0 conn(0x7fb9f81376e0 legacy=0x7fb9f80a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:07:49.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 --2- 172.21.15.114:0/1511809488 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9e00039c0 0x7fb9e0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient(hunting): picked mon.b con 0x7fb9e00039c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:49.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 --2- 172.21.15.114:0/1511809488 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9e0004810 0x7fb9e0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient(hunting): picked mon.c con 0x7fb9e0004810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 --2- 172.21.15.114:0/1511809488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9e0005220 0x7fb9e000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 10 monclient: monmap: 2024-09-16T05:07:49.526 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.526 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:49.526 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:49.526 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:49.526 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.526 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.526 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:49.526 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:49.526 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:49.526 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient(hunting): picked mon.a con 0x7fb9e0005220 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:49.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 -- 172.21.15.114:0/1511809488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e000c140 con 0x7fb9e0005220 2024-09-16T05:07:49.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 -- 172.21.15.114:0/1511809488 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e000c2f0 con 0x7fb9e00039c0 2024-09-16T05:07:49.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 1 -- 172.21.15.114:0/1511809488 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e000c4f0 con 0x7fb9e0004810 2024-09-16T05:07:49.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9f7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:07:49.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fc930640 1 --2- 172.21.15.114:0/1511809488 >> v2:172.21.15.149:3300/0 conn(0x7fb9f81370a0 0x7fb9f8137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:49.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 1 --2- 172.21.15.114:0/1511809488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9e0005220 0x7fb9e000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 10 monclient(hunting): get_auth_request con 0x7fb9e0005220 auth_method 0 2024-09-16T05:07:49.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:49.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fc930640 1 --2- 172.21.15.114:0/1511809488 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9e0004810 0x7fb9e0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:07:49.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fc930640 10 monclient(hunting): get_auth_request con 0x7fb9e0004810 auth_method 0 2024-09-16T05:07:49.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fc930640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fc930640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fc930640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:49.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 10 monclient(hunting): handle_auth_done global_id 4257 payload 293 2024-09-16T05:07:49.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 1 librados: starting msgr at 2024-09-16T05:07:49.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 1 librados: starting objecter 2024-09-16T05:07:49.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 1 -- 172.21.15.114:0/1511809488 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9e0004810 msgr2=0x7fb9e0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:49.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 1 --2- 172.21.15.114:0/1511809488 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9e0004810 0x7fb9e0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 1 -- 172.21.15.114:0/1511809488 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9e00039c0 msgr2=0x7fb9e0003de0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T05:07:49.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 1 --2- 172.21.15.114:0/1511809488 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9e00039c0 0x7fb9e0003de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 1 Processor -- start 2024-09-16T05:07:49.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 1 monclient: found mon.a 2024-09-16T05:07:49.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 1 -- 172.21.15.114:0/1511809488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f809c000 con 0x7fb9e0005220 2024-09-16T05:07:49.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7fb9fd932640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 1 -- start start 2024-09-16T05:07:49.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 1 librados: setting wanted keys 2024-09-16T05:07:49.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 1 librados: calling monclient init 2024-09-16T05:07:49.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f2b07aa3640 10 monclient: init 2024-09-16T05:07:49.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7bb7fe640 1 --2- 172.21.15.114:0/1176145192 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7bc137cc0 0x7ff7bc13fd50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009dc0 tx=0x7ff7b401c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62b84c09d83084c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 1 -- 172.21.15.114:0/1176145192 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b40b2070 con 0x7ff7bc137cc0 2024-09-16T05:07:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 10 monclient: dump: 2024-09-16T05:07:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.536 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 1 -- 172.21.15.114:0/1176145192 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7b40a5900 con 0x7ff7bc137cc0 2024-09-16T05:07:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 1 -- 172.21.15.114:0/1176145192 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b40a5cd0 con 0x7ff7bc137cc0 2024-09-16T05:07:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7ba7fc640 10 monclient: dump: 2024-09-16T05:07:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.537 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.537 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.537 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.537 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.537 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.537 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.537 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.537 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7f901a7f7640 1 --2- 172.21.15.114:0/223574415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f901413a700 0x7f901413caf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f90141329e0 tx=0x7f901001c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286e15f98511b364 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7c17db640 5 monclient: authenticate success, global_id 4226 2024-09-16T05:07:49.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7c17db640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:49.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.450+0000 7ff7c17db640 4 set_mon_vals no callback set 2024-09-16T05:07:49.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9ff395640 5 monclient: authenticate success, global_id 4257 2024-09-16T05:07:49.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9ff395640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:07:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b000a4270 0x7f2b0013bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 monclient(hunting): picked mon.noname-a con 0x7f2b000a4270 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b000a4650 0x7f2b0013c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 monclient(hunting): picked mon.noname-c con 0x7f2b000a4650 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b00140ab0 0x7f2b0009c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9018ff4640 1 -- 172.21.15.114:0/223574415 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90100b0070 con 0x7f901413a700 2024-09-16T05:07:49.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9018ff4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9018ff4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:49.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9018ff4640 10 monclient: dump: 2024-09-16T05:07:49.541 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.541 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.541 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.541 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.541 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.541 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.541 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.541 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.541 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.541 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9018ff4640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9018ff4640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 monclient(hunting): picked mon.noname-b con 0x7f2b00140ab0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:49.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9018ff4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9018ff4640 1 -- 172.21.15.114:0/223574415 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90100ab600 con 0x7f901413a700 2024-09-16T05:07:49.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9018ff4640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9018ff4640 1 -- 172.21.15.114:0/223574415 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90100ab9d0 con 0x7f901413a700 2024-09-16T05:07:49.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9018ff4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9018ff4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:49.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9018ff4640 10 monclient: dump: 2024-09-16T05:07:49.543 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.543 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.543 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.544 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.544 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.544 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.544 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.544 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.544 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.544 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b00147170 con 0x7f2b000a4270 2024-09-16T05:07:49.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f2b00146ff0 con 0x7f2b00140ab0 2024-09-16T05:07:49.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2b001472f0 con 0x7f2b000a4650 2024-09-16T05:07:49.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 monclient(hunting): authenticate will time out at 954.084279s 2024-09-16T05:07:49.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 5 monclient: authenticate success, global_id 4251 2024-09-16T05:07:49.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:49.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 4 set_mon_vals no callback set 2024-09-16T05:07:49.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b000a4270 0x7f2b0013bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b000a4270 0x7f2b0013bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54802/0 (socket says 172.21.15.114:54802) 2024-09-16T05:07:49.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 1 -- 172.21.15.114:0/1920052488 learned_addr learned my addr 172.21.15.114:0/1920052488 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 10 monclient(hunting): get_auth_request con 0x7f2b000a4270 auth_method 0 2024-09-16T05:07:49.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b06040640 1 --2- 172.21.15.114:0/1920052488 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b00140ab0 0x7f2b0009c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b06040640 10 monclient(hunting): get_auth_request con 0x7f2b00140ab0 auth_method 0 2024-09-16T05:07:49.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b06040640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b06040640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b06040640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b06040640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b06040640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b06040640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 10 monclient(hunting): handle_auth_done global_id 4260 payload 824 2024-09-16T05:07:49.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 1 -- 172.21.15.114:0/1920052488 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b000a4650 msgr2=0x7f2b0013c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 1 --2- 172.21.15.114:0/1920052488 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b000a4650 0x7f2b0013c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 10 monclient: shutdown 2024-09-16T05:07:49.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 1 -- 172.21.15.114:0/1176145192 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7bc137cc0 msgr2=0x7ff7bc13fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 1 --2- 172.21.15.114:0/1176145192 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7bc137cc0 0x7ff7bc13fd50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009dc0 tx=0x7ff7b401c820 comp rx=0 tx=0).stop 2024-09-16T05:07:49.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 1 -- 172.21.15.114:0/1920052488 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b00140ab0 msgr2=0x7f2b0009c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 1 --2- 172.21.15.114:0/1920052488 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b00140ab0 0x7f2b0009c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 1 monclient: found mon.noname-a 2024-09-16T05:07:49.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 1 -- 172.21.15.114:0/1920052488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b000a11c0 con 0x7f2b000a4270 2024-09-16T05:07:49.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 10 monclient: shutdown 2024-09-16T05:07:49.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 -- 172.21.15.114:0/223574415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f901413a700 msgr2=0x7f901413caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 --2- 172.21.15.114:0/223574415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f901413a700 0x7f901413caf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f90141329e0 tx=0x7f901001c340 comp rx=0 tx=0).stop 2024-09-16T05:07:49.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0583f640 1 --2- 172.21.15.114:0/1920052488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b000a4270 0x7f2b0013bdb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2afc0097f0 tx=0x7f2afc01ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 1 -- 172.21.15.114:0/1176145192 shutdown_connections 2024-09-16T05:07:49.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 1 --2- 172.21.15.114:0/1176145192 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7bc137cc0 0x7ff7bc13fd50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 1 -- 172.21.15.114:0/1176145192 >> 172.21.15.114:0/1176145192 conn(0x7ff7bc0a9be0 msgr2=0x7ff7bc0a9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 client.?.objecter ms_handle_connect 0x7f2b000a4270 2024-09-16T05:07:49.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:49.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 -- 172.21.15.114:0/223574415 shutdown_connections 2024-09-16T05:07:49.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 --2- 172.21.15.114:0/223574415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f901413a700 0x7f901413caf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 --2- 172.21.15.114:0/223574415 >> v2:172.21.15.149:3300/0 conn(0x7f901409d190 0x7f901413a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 -- 172.21.15.114:0/223574415 >> 172.21.15.114:0/223574415 conn(0x7f90140a3270 msgr2=0x7f90140a3680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 -- 172.21.15.114:0/223574415 shutdown_connections 2024-09-16T05:07:49.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 1 -- 172.21.15.114:0/1920052488 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc0b0050 con 0x7f2b000a4270 2024-09-16T05:07:49.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:49.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 monclient: dump: 2024-09-16T05:07:49.553 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.553 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.554 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.554 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.554 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.554 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.554 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.554 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.554 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.554 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 1 -- 172.21.15.114:0/1920052488 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2afc0bc070 con 0x7f2b000a4270 2024-09-16T05:07:49.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 5 monclient: authenticate success, global_id 4260 2024-09-16T05:07:49.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 1 -- 172.21.15.114:0/1920052488 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc0b5a30 con 0x7f2b000a4270 2024-09-16T05:07:49.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:49.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 monclient: dump: 2024-09-16T05:07:49.555 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.556 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.556 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.556 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.556 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.556 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.556 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.556 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.556 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.556 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 monclient: _renew_subs 2024-09-16T05:07:49.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af7fff640 4 set_mon_vals no callback set 2024-09-16T05:07:49.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 1 -- 172.21.15.114:0/1920052488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b000a0220 con 0x7f2b000a4270 2024-09-16T05:07:49.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 -- 172.21.15.114:0/223574415 wait complete. 2024-09-16T05:07:49.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 monclient: _renew_subs 2024-09-16T05:07:49.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 1 -- 172.21.15.114:0/1920052488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b0009f050 con 0x7f2b000a4270 2024-09-16T05:07:49.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 1 librados: init done 2024-09-16T05:07:49.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9fd932640 1 -- 172.21.15.114:0/1511809488 reap_dead start 2024-09-16T05:07:49.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9fd932640 1 --2- 172.21.15.114:0/1511809488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9e0005220 0x7fb9e000b9d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8001750 tx=0x7fb9e801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b72ebb9f5d87a3b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 1 -- 172.21.15.114:0/1920052488 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2afc0b5bd0 con 0x7f2b000a4270 2024-09-16T05:07:49.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 10 monclient: monmap: 2024-09-16T05:07:49.559 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.559 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:49.559 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:49.560 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:49.560 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.560 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.560 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:49.560 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:49.560 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:49.561 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 1 -- 172.21.15.114:0/1176145192 shutdown_connections 2024-09-16T05:07:49.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 client.4260.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 15 client.4260.objecter _session_command_op_assign -1 1 2024-09-16T05:07:49.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 1 --2- 172.21.15.114:0/1920052488 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2ae4037840 0x7f2ae4039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 20 client.4260.objecter _get_session osd=-1 returning homeless 2024-09-16T05:07:49.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 librados: starting msgr at 2024-09-16T05:07:49.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 librados: starting objecter 2024-09-16T05:07:49.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b07aa3640 1 -- 172.21.15.114:0/1920052488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2b0006ee40 con 0x7f2b000a4270 2024-09-16T05:07:49.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 1 -- 172.21.15.114:0/1920052488 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f2afc0e2f20 con 0x7f2b000a4270 2024-09-16T05:07:49.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 client.4260.objecter ms_dispatch 0x7f2b00000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:49.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 3 client.4260.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:49.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 3 client.4260.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:49.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 Processor -- start 2024-09-16T05:07:49.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 client.4260.objecter checking command 1 2024-09-16T05:07:49.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 1 --2- 172.21.15.114:0/1920052488 >> v2:172.21.15.149:6805/1362772130 conn(0x7f2ae403d2e0 0x7f2ae403f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 20 client.4260.objecter _get_session s=0x7f2ae403cc30 osd=3 3 2024-09-16T05:07:49.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 20 client.4260.objecter put_session s=0x7f2ae403cc30 osd=3 3 2024-09-16T05:07:49.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 15 client.4260.objecter _session_command_op_remove -1 1 2024-09-16T05:07:49.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 20 client.4260.objecter _get_session s=0x7f2ae403cc30 osd=3 3 2024-09-16T05:07:49.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 20 client.4260.objecter get_session s=0x7f2ae403cc30 osd=3 3 2024-09-16T05:07:49.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 15 client.4260.objecter _session_command_op_assign 3 1 2024-09-16T05:07:49.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 20 client.4260.objecter put_session s=0x7f2ae403cc30 osd=3 4 2024-09-16T05:07:49.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 client.4260.objecter _send_command 1 2024-09-16T05:07:49.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 1 -- 172.21.15.114:0/1920052488 --> v2:172.21.15.149:6805/1362772130 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ae403fdb0 con 0x7f2ae403d2e0 2024-09-16T05:07:49.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 20 client.4260.objecter dump_active .. 0 homeless 2024-09-16T05:07:49.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 -- start start 2024-09-16T05:07:49.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 librados: setting wanted keys 2024-09-16T05:07:49.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 librados: calling monclient init 2024-09-16T05:07:49.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 10 monclient: init 2024-09-16T05:07:49.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 1 -- 172.21.15.114:0/1920052488 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f2afc0b4360 con 0x7f2b000a4270 2024-09-16T05:07:49.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T05:07:49.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0503e640 1 --2- 172.21.15.114:0/1920052488 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2ae4037840 0x7f2ae4039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0503e640 10 monclient: get_auth_request con 0x7f2ae4037840 auth_method 0 2024-09-16T05:07:49.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b06040640 1 --2- 172.21.15.114:0/1920052488 >> v2:172.21.15.149:6805/1362772130 conn(0x7f2ae403d2e0 0x7f2ae403f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b06040640 10 monclient: get_auth_request con 0x7f2ae403d2e0 auth_method 0 2024-09-16T05:07:49.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:07:49.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f901409cdb0 0x7f901412bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9f7fff640 1 -- 172.21.15.114:0/1511809488 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e80013a0 con 0x7fb9e0005220 2024-09-16T05:07:49.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:49.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9f7fff640 10 monclient: dump: 2024-09-16T05:07:49.571 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.571 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.571 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.571 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.571 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.571 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.571 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.571 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.571 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.571 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9f7fff640 1 -- 172.21.15.114:0/1511809488 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9e80052d0 con 0x7fb9e0005220 2024-09-16T05:07:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9f7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9f7fff640 1 -- 172.21.15.114:0/1511809488 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8003840 con 0x7fb9e0005220 2024-09-16T05:07:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9f7fff640 10 monclient: dump: 2024-09-16T05:07:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.573 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.573 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.573 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.573 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9ff395640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:49.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9ff395640 4 set_mon_vals no callback set 2024-09-16T05:07:49.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9ff395640 10 monclient: shutdown 2024-09-16T05:07:49.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9ff395640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9ff395640 1 -- 172.21.15.114:0/1511809488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9e0005220 msgr2=0x7fb9e000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9ff395640 1 --2- 172.21.15.114:0/1511809488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9e0005220 0x7fb9e000b9d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8001750 tx=0x7fb9e801c3a0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 10 monclient(hunting): picked mon.noname-a con 0x7f901409cdb0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:49.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9ff395640 1 -- 172.21.15.114:0/1511809488 shutdown_connections 2024-09-16T05:07:49.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9ff395640 1 --2- 172.21.15.114:0/1511809488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9e0005220 0x7fb9e000b9d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9ff395640 1 -- 172.21.15.114:0/1511809488 >> 172.21.15.114:0/1511809488 conn(0x7fb9f80a9a40 msgr2=0x7fb9f80a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9019ff6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f901409cdb0 0x7f901412bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9019ff6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f901409cdb0 0x7f901412bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54814/0 (socket says 172.21.15.114:54814) 2024-09-16T05:07:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9019ff6640 1 -- 172.21.15.114:0/2892680095 learned_addr learned my addr 172.21.15.114:0/2892680095 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 1 -- 172.21.15.114:0/1176145192 wait complete. 2024-09-16T05:07:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 --2- 172.21.15.114:0/2892680095 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f901409d190 0x7f901412c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 10 monclient: monmap: 2024-09-16T05:07:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.577 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.577 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:49.577 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:49.577 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:49.577 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 1 librados: starting msgr at 2024-09-16T05:07:49.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 1 librados: starting objecter 2024-09-16T05:07:49.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 1 Processor -- start 2024-09-16T05:07:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 10 monclient(hunting): picked mon.noname-b con 0x7f901409d190 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 --2- 172.21.15.114:0/2892680095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f901413a700 0x7f901412c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9ff395640 1 -- 172.21.15.114:0/1511809488 shutdown_connections 2024-09-16T05:07:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 10 monclient(hunting): picked mon.noname-c con 0x7f901413a700 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 -- 172.21.15.114:0/2892680095 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f901413fbe0 con 0x7f901409cdb0 2024-09-16T05:07:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 -- 172.21.15.114:0/2892680095 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f901413fa60 con 0x7f901409d190 2024-09-16T05:07:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 1 -- 172.21.15.114:0/2892680095 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f901413fd60 con 0x7f901413a700 2024-09-16T05:07:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f901c25a640 10 monclient(hunting): authenticate will time out at 954.087911s 2024-09-16T05:07:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9019ff6640 10 monclient(hunting): get_auth_request con 0x7f901409cdb0 auth_method 0 2024-09-16T05:07:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9019ff6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9019ff6640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f9019ff6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f9019ff6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f9019ff6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f90197f5640 1 --2- 172.21.15.114:0/2892680095 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f901409d190 0x7f901412c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f9019ff6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f90197f5640 10 monclient(hunting): get_auth_request con 0x7f901409d190 auth_method 0 2024-09-16T05:07:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f90197f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f90197f5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f90197f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f9019ff6640 10 monclient(hunting): handle_auth_done global_id 4263 payload 824 2024-09-16T05:07:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f9019ff6640 1 -- 172.21.15.114:0/2892680095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f901413a700 msgr2=0x7f901412c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f9019ff6640 1 --2- 172.21.15.114:0/2892680095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f901413a700 0x7f901412c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f9019ff6640 1 -- 172.21.15.114:0/2892680095 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f901409d190 msgr2=0x7f901412c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f9019ff6640 1 --2- 172.21.15.114:0/2892680095 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f901409d190 0x7f901412c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f9019ff6640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f9019ff6640 1 monclient: found mon.noname-a 2024-09-16T05:07:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f9019ff6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f9019ff6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f9019ff6640 1 -- 172.21.15.114:0/2892680095 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f901406f680 con 0x7f901409cdb0 2024-09-16T05:07:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f9019ff6640 1 --2- 172.21.15.114:0/2892680095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f901409cdb0 0x7f901412bf10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f900400c410 tx=0x7f900400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 10 client.?.objecter ms_handle_connect 0x7f901409cdb0 2024-09-16T05:07:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 1 -- 172.21.15.114:0/2892680095 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90040074e0 con 0x7f901409cdb0 2024-09-16T05:07:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:49.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 10 monclient: dump: 2024-09-16T05:07:49.582 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.582 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.582 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.582 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.582 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.582 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.582 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.582 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.582 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.582 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 1 -- start start 2024-09-16T05:07:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 1 librados: setting wanted keys 2024-09-16T05:07:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 1 librados: calling monclient init 2024-09-16T05:07:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7ff7c17db640 10 monclient: init 2024-09-16T05:07:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:07:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7bc0a9220 0x7ff7bc06e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 monclient(hunting): picked mon.noname-b con 0x7ff7bc0a9220 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7bc1371d0 0x7ff7bc13b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 monclient(hunting): picked mon.noname-a con 0x7ff7bc1371d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7bc137cc0 0x7ff7bc13b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 monclient(hunting): picked mon.noname-c con 0x7ff7bc137cc0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff7bc142c30 con 0x7ff7bc1371d0 2024-09-16T05:07:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7ff7bc142ab0 con 0x7ff7bc0a9220 2024-09-16T05:07:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff7bc142db0 con 0x7ff7bc137cc0 2024-09-16T05:07:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 monclient(hunting): authenticate will time out at 954.088193s 2024-09-16T05:07:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7bc1371d0 0x7ff7bc13b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7bc1371d0 0x7ff7bc13b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54820/0 (socket says 172.21.15.114:54820) 2024-09-16T05:07:49.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 1 -- 172.21.15.114:0/879980778 learned_addr learned my addr 172.21.15.114:0/879980778 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 10 monclient(hunting): get_auth_request con 0x7ff7bc1371d0 auth_method 0 2024-09-16T05:07:49.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7baffd640 1 --2- 172.21.15.114:0/879980778 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7bc0a9220 0x7ff7bc06e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7baffd640 10 monclient(hunting): get_auth_request con 0x7ff7bc0a9220 auth_method 0 2024-09-16T05:07:49.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7bb7fe640 1 --2- 172.21.15.114:0/879980778 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7bc137cc0 0x7ff7bc13b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 10 monclient(hunting): handle_auth_done global_id 4266 payload 824 2024-09-16T05:07:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 1 -- 172.21.15.114:0/879980778 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7bc137cc0 msgr2=0x7ff7bc13b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 1 --2- 172.21.15.114:0/879980778 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7bc137cc0 0x7ff7bc13b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 1 -- 172.21.15.114:0/879980778 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7bc0a9220 msgr2=0x7ff7bc06e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 1 --2- 172.21.15.114:0/879980778 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7bc0a9220 0x7ff7bc06e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 1 monclient: found mon.noname-a 2024-09-16T05:07:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 1 -- 172.21.15.114:0/879980778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7bc13f0a0 con 0x7ff7bc1371d0 2024-09-16T05:07:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7bb7fe640 10 monclient: get_auth_request con 0x7ff7bc137cc0 auth_method 0 2024-09-16T05:07:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7bb7fe640 1 --2- 172.21.15.114:0/879980778 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7bc137cc0 0x7ff7bc13b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:07:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 client.4260.objecter ms_handle_connect 0x7f2ae4037840 2024-09-16T05:07:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b0503e640 1 --2- 172.21.15.114:0/1920052488 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2ae4037840 0x7f2ae4039d00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2b0009e0f0 tx=0x7f2af00079c0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2af67fc640 10 client.4260.objecter ms_handle_connect 0x7f2ae403d2e0 2024-09-16T05:07:49.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7f2b06040640 1 --2- 172.21.15.114:0/1920052488 >> v2:172.21.15.149:6805/1362772130 conn(0x7f2ae403d2e0 0x7f2ae403f720 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.454+0000 7fb9ff395640 1 -- 172.21.15.114:0/1511809488 wait complete. 2024-09-16T05:07:49.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient: monmap: 2024-09-16T05:07:49.591 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.591 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:49.591 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:49.591 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:49.591 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.591 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.592 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:49.592 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:49.592 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:49.592 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 1 librados: starting msgr at 2024-09-16T05:07:49.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 1 librados: starting objecter 2024-09-16T05:07:49.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 1 Processor -- start 2024-09-16T05:07:49.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 1 -- 172.21.15.114:0/2892680095 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90040047f0 con 0x7f901409cdb0 2024-09-16T05:07:49.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f901c25a640 5 monclient: authenticate success, global_id 4263 2024-09-16T05:07:49.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f901c25a640 10 monclient: _renew_subs 2024-09-16T05:07:49.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f901c25a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f901c25a640 1 -- 172.21.15.114:0/2892680095 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f901412de90 con 0x7f901409cdb0 2024-09-16T05:07:49.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b3fff640 1 --2- 172.21.15.114:0/879980778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7bc1371d0 0x7ff7bc13b440 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff7a400c450 tx=0x7ff7a400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 10 client.?.objecter ms_handle_connect 0x7ff7bc1371d0 2024-09-16T05:07:49.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:49.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 1 -- 172.21.15.114:0/2892680095 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9004007680 con 0x7f901409cdb0 2024-09-16T05:07:49.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:49.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 10 monclient: dump: 2024-09-16T05:07:49.597 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.598 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.598 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.598 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.598 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.598 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.598 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.599 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.599 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.599 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900bfff640 4 set_mon_vals no callback set 2024-09-16T05:07:49.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f901c25a640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f901c25a640 10 monclient: _renew_subs 2024-09-16T05:07:49.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 1 -- start start 2024-09-16T05:07:49.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 1 librados: setting wanted keys 2024-09-16T05:07:49.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 1 librados: calling monclient init 2024-09-16T05:07:49.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient: init 2024-09-16T05:07:49.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f901c25a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f901c25a640 1 -- 172.21.15.114:0/2892680095 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f901412da50 con 0x7f901409cdb0 2024-09-16T05:07:49.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f901c25a640 1 librados: init done 2024-09-16T05:07:49.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 1 -- 172.21.15.114:0/879980778 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4016020 con 0x7ff7bc1371d0 2024-09-16T05:07:49.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:49.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 10 monclient: dump: 2024-09-16T05:07:49.604 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.604 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.604 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.605 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.605 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.605 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.605 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.605 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.605 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.606 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 1 -- 172.21.15.114:0/879980778 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7a40052d0 con 0x7ff7bc1371d0 2024-09-16T05:07:49.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 1 -- 172.21.15.114:0/879980778 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4007260 con 0x7ff7bc1371d0 2024-09-16T05:07:49.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:49.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 10 monclient: dump: 2024-09-16T05:07:49.607 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.607 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.608 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.608 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.608 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.608 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.608 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.608 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.609 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.609 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b9ffb640 4 set_mon_vals no callback set 2024-09-16T05:07:49.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 5 monclient: authenticate success, global_id 4266 2024-09-16T05:07:49.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 monclient: _renew_subs 2024-09-16T05:07:49.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 1 -- 172.21.15.114:0/879980778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7bc13d530 con 0x7ff7bc1371d0 2024-09-16T05:07:49.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 monclient: _renew_subs 2024-09-16T05:07:49.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 1 -- 172.21.15.114:0/879980778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7bc13f670 con 0x7ff7bc1371d0 2024-09-16T05:07:49.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 1 librados: init done 2024-09-16T05:07:49.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:07:49.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9e0005220 0x7fb9f8133780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient(hunting): picked mon.noname-b con 0x7fb9e0005220 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:49.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9f8130ed0 0x7fb9f8131310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient(hunting): picked mon.noname-a con 0x7fb9f8130ed0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:49.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9f8131850 0x7fb9f81e2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient(hunting): picked mon.noname-c con 0x7fb9f8131850 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f8137e50 con 0x7fb9f8130ed0 2024-09-16T05:07:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f8137cd0 con 0x7fb9e0005220 2024-09-16T05:07:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb9f80a4270 con 0x7fb9f8131850 2024-09-16T05:07:49.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient(hunting): authenticate will time out at 954.089755s 2024-09-16T05:07:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9f8131850 0x7fb9f81e2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9f8131850 0x7fb9f81e2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55534/0 (socket says 172.21.15.114:55534) 2024-09-16T05:07:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 1 -- 172.21.15.114:0/3220627640 learned_addr learned my addr 172.21.15.114:0/3220627640 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 10 monclient(hunting): get_auth_request con 0x7fb9f8131850 auth_method 0 2024-09-16T05:07:49.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 1 -- 172.21.15.114:0/2892680095 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9004011070 con 0x7f901409cdb0 2024-09-16T05:07:49.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd131640 1 --2- 172.21.15.114:0/3220627640 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9e0005220 0x7fb9f8133780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd131640 10 monclient(hunting): get_auth_request con 0x7fb9e0005220 auth_method 0 2024-09-16T05:07:49.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd131640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd131640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd131640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f2af67fc640 1 -- 172.21.15.114:0/1920052488 <== osd.3 v2:172.21.15.149:6805/1362772130 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2ae403fdb0 con 0x7f2ae403d2e0 2024-09-16T05:07:49.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f2af67fc640 10 client.4260.objecter ms_dispatch 0x7f2b00000ba0 command_reply(tid 1: 0 ) 2024-09-16T05:07:49.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f2af67fc640 10 client.4260.objecter _finish_command 1 = system:0 2024-09-16T05:07:49.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f2af67fc640 20 client.4260.objecter put_session s=0x7f2ae403cc30 osd=3 4 2024-09-16T05:07:49.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f2af67fc640 15 client.4260.objecter _session_command_op_remove 3 1 2024-09-16T05:07:49.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd131640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd131640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd131640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 client.4266.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 15 client.4266.objecter _session_command_op_assign -1 1 2024-09-16T05:07:49.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f900a7fc640 1 --2- 172.21.15.114:0/2892680095 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8ff8037840 0x7f8ff8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2024-09-16T05:07:49.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 20 client.4266.objecter _get_session osd=-1 returning homeless 2024-09-16T05:07:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 1 -- 172.21.15.114:0/3220627640 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9e0005220 msgr2=0x7fb9f8133780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 1 --2- 172.21.15.114:0/3220627640 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9e0005220 0x7fb9f8133780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 1 -- 172.21.15.114:0/3220627640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9f8130ed0 msgr2=0x7fb9f8131310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 1 --2- 172.21.15.114:0/3220627640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9f8130ed0 0x7fb9f8131310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 1 monclient: found mon.noname-c 2024-09-16T05:07:49.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f8ff7fff640 10 client.4263.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 1 -- 172.21.15.114:0/3220627640 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f81345e0 con 0x7fb9f8131850 2024-09-16T05:07:49.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f8ff7fff640 15 client.4263.objecter _session_command_op_assign -1 1 2024-09-16T05:07:49.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f8ff7fff640 20 client.4263.objecter _get_session osd=-1 returning homeless 2024-09-16T05:07:49.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f8ff7fff640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7c17db640 1 -- 172.21.15.114:0/879980778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff7bc0a5a80 con 0x7ff7bc1371d0 2024-09-16T05:07:49.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f8ff7fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f8ff7fff640 1 -- 172.21.15.114:0/2892680095 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9014131ce0 con 0x7f901409cdb0 2024-09-16T05:07:49.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f90197f5640 1 --2- 172.21.15.114:0/2892680095 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8ff8037840 0x7f8ff8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7f90197f5640 10 monclient: get_auth_request con 0x7f8ff8037840 auth_method 0 2024-09-16T05:07:49.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9fd932640 1 --2- 172.21.15.114:0/3220627640 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9f8131850 0x7fb9f81e2220 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb9e801cc40 tx=0x7fb9e0005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f5ffb640 10 client.?.objecter ms_handle_connect 0x7fb9f8131850 2024-09-16T05:07:49.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f5ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:49.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f5ffb640 1 -- 172.21.15.114:0/3220627640 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8007a40 con 0x7fb9f8131850 2024-09-16T05:07:49.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 1 -- 172.21.15.114:0/879980778 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff7a4007400 con 0x7ff7bc1371d0 2024-09-16T05:07:49.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:49.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f5ffb640 10 monclient: dump: 2024-09-16T05:07:49.631 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.631 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.631 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.631 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.631 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.631 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.632 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.632 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.632 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.632 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f5ffb640 1 -- 172.21.15.114:0/3220627640 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9e8005480 con 0x7fb9f8131850 2024-09-16T05:07:49.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f5ffb640 1 -- 172.21.15.114:0/3220627640 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8003c00 con 0x7fb9f8131850 2024-09-16T05:07:49.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:49.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f5ffb640 10 monclient: dump: 2024-09-16T05:07:49.634 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.634 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.634 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.634 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.634 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.634 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.635 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.635 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.635 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.635 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 5 monclient: authenticate success, global_id 4229 2024-09-16T05:07:49.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient: _renew_subs 2024-09-16T05:07:49.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 1 -- 172.21.15.114:0/3220627640 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9f8132a70 con 0x7fb9f8131850 2024-09-16T05:07:49.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9f77fe640 4 set_mon_vals no callback set 2024-09-16T05:07:49.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient: _renew_subs 2024-09-16T05:07:49.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 1 -- 172.21.15.114:0/3220627640 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9f81348c0 con 0x7fb9f8131850 2024-09-16T05:07:49.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7fb9ff395640 1 librados: init done 2024-09-16T05:07:49.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.458+0000 7ff7b37fe640 1 --2- 172.21.15.114:0/879980778 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff79c037810 0x7ff79c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 1 -- 172.21.15.114:0/879980778 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7ff7a4048db0 con 0x7ff7bc1371d0 2024-09-16T05:07:49.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 10 client.4266.objecter ms_dispatch 0x7ff7bc000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:49.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 3 client.4266.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:49.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 3 client.4266.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:49.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 10 client.4266.objecter checking command 1 2024-09-16T05:07:49.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 1 --2- 172.21.15.114:0/879980778 >> v2:172.21.15.114:6804/1932657732 conn(0x7ff79c03d2b0 0x7ff79c03f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 20 client.4266.objecter _get_session s=0x7ff79c03cc00 osd=2 3 2024-09-16T05:07:49.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 20 client.4266.objecter put_session s=0x7ff79c03cc00 osd=2 3 2024-09-16T05:07:49.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 15 client.4266.objecter _session_command_op_remove -1 1 2024-09-16T05:07:49.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 20 client.4266.objecter _get_session s=0x7ff79c03cc00 osd=2 3 2024-09-16T05:07:49.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 20 client.4266.objecter get_session s=0x7ff79c03cc00 osd=2 3 2024-09-16T05:07:49.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 15 client.4266.objecter _session_command_op_assign 2 1 2024-09-16T05:07:49.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 20 client.4266.objecter put_session s=0x7ff79c03cc00 osd=2 4 2024-09-16T05:07:49.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 10 client.4266.objecter _send_command 1 2024-09-16T05:07:49.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 1 -- 172.21.15.114:0/879980778 --> v2:172.21.15.114:6804/1932657732 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff7a4017e70 con 0x7ff79c03d2b0 2024-09-16T05:07:49.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 20 client.4266.objecter dump_active .. 0 homeless 2024-09-16T05:07:49.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 1 -- 172.21.15.114:0/879980778 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7ff7a4049210 con 0x7ff7bc1371d0 2024-09-16T05:07:49.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T05:07:49.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 1 -- 172.21.15.114:0/2892680095 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f9004011c60 con 0x7f901409cdb0 2024-09-16T05:07:49.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 10 client.4263.objecter ms_dispatch 0x7f9014000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:49.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 3 client.4263.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:49.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 3 client.4263.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:49.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 10 client.4263.objecter checking command 1 2024-09-16T05:07:49.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 1 --2- 172.21.15.114:0/2892680095 >> v2:172.21.15.114:6800/2903720358 conn(0x7f8ff803d350 0x7f8ff803f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 20 client.4263.objecter _get_session s=0x7f8ff803cca0 osd=1 3 2024-09-16T05:07:49.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 20 client.4263.objecter put_session s=0x7f8ff803cca0 osd=1 3 2024-09-16T05:07:49.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 15 client.4263.objecter _session_command_op_remove -1 1 2024-09-16T05:07:49.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 20 client.4263.objecter _get_session s=0x7f8ff803cca0 osd=1 3 2024-09-16T05:07:49.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 20 client.4263.objecter get_session s=0x7f8ff803cca0 osd=1 3 2024-09-16T05:07:49.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 15 client.4263.objecter _session_command_op_assign 1 1 2024-09-16T05:07:49.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 20 client.4263.objecter put_session s=0x7f8ff803cca0 osd=1 4 2024-09-16T05:07:49.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 10 client.4263.objecter _send_command 1 2024-09-16T05:07:49.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 1 -- 172.21.15.114:0/2892680095 --> v2:172.21.15.114:6800/2903720358 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9004011e70 con 0x7f8ff803d350 2024-09-16T05:07:49.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 20 client.4263.objecter dump_active .. 0 homeless 2024-09-16T05:07:49.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 1 -- 172.21.15.114:0/2892680095 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f900401be40 con 0x7f901409cdb0 2024-09-16T05:07:49.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7f900a7fc640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T05:07:49.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7baffd640 1 --2- 172.21.15.114:0/879980778 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff79c037810 0x7ff79c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7bb7fe640 1 --2- 172.21.15.114:0/879980778 >> v2:172.21.15.114:6804/1932657732 conn(0x7ff79c03d2b0 0x7ff79c03f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7baffd640 10 monclient: get_auth_request con 0x7ff79c037810 auth_method 0 2024-09-16T05:07:49.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7bb7fe640 10 monclient: get_auth_request con 0x7ff79c03d2b0 auth_method 0 2024-09-16T05:07:49.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9ff395640 10 client.4229.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9ff395640 15 client.4229.objecter _session_command_op_assign -1 1 2024-09-16T05:07:49.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9ff395640 20 client.4229.objecter _get_session osd=-1 returning homeless 2024-09-16T05:07:49.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9ff395640 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 1 -- 172.21.15.114:0/3220627640 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb9e8007030 con 0x7fb9f8131850 2024-09-16T05:07:49.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9ff395640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9ff395640 1 -- 172.21.15.114:0/3220627640 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb9f80a8b40 con 0x7fb9f8131850 2024-09-16T05:07:49.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 1 --2- 172.21.15.114:0/3220627640 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb9e4037810 0x7fb9e4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7bb7fe640 1 --2- 172.21.15.114:0/879980778 >> v2:172.21.15.114:6804/1932657732 conn(0x7ff79c03d2b0 0x7ff79c03f6f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 10 client.4266.objecter ms_handle_connect 0x7ff79c03d2b0 2024-09-16T05:07:49.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 1 -- 172.21.15.114:0/3220627640 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7fb9e8003dc0 con 0x7fb9f8131850 2024-09-16T05:07:49.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 10 client.4229.objecter ms_dispatch 0x7fb9f8000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:49.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 3 client.4229.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:49.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 3 client.4229.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:49.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 10 client.4229.objecter checking command 1 2024-09-16T05:07:49.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9fd131640 1 --2- 172.21.15.114:0/3220627640 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb9e4037810 0x7fb9e4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 1 --2- 172.21.15.114:0/3220627640 >> v2:172.21.15.149:6800/1092505261 conn(0x7fb9e403d320 0x7fb9e403f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 20 client.4229.objecter _get_session s=0x7fb9e403cc70 osd=4 3 2024-09-16T05:07:49.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 20 client.4229.objecter put_session s=0x7fb9e403cc70 osd=4 3 2024-09-16T05:07:49.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 15 client.4229.objecter _session_command_op_remove -1 1 2024-09-16T05:07:49.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 20 client.4229.objecter _get_session s=0x7fb9e403cc70 osd=4 3 2024-09-16T05:07:49.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 20 client.4229.objecter get_session s=0x7fb9e403cc70 osd=4 3 2024-09-16T05:07:49.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 15 client.4229.objecter _session_command_op_assign 4 1 2024-09-16T05:07:49.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 20 client.4229.objecter put_session s=0x7fb9e403cc70 osd=4 4 2024-09-16T05:07:49.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 10 client.4229.objecter _send_command 1 2024-09-16T05:07:49.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9fd131640 10 monclient: get_auth_request con 0x7fb9e4037810 auth_method 0 2024-09-16T05:07:49.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 1 -- 172.21.15.114:0/3220627640 --> v2:172.21.15.149:6800/1092505261 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9e80aee90 con 0x7fb9e403d320 2024-09-16T05:07:49.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 20 client.4229.objecter dump_active .. 0 homeless 2024-09-16T05:07:49.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 1 -- 172.21.15.114:0/3220627640 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fb9e8001460 con 0x7fb9f8131850 2024-09-16T05:07:49.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T05:07:49.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 10 client.4266.objecter ms_handle_connect 0x7ff79c037810 2024-09-16T05:07:49.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7baffd640 1 --2- 172.21.15.114:0/879980778 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff79c037810 0x7ff79c039cd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac0011f0 tx=0x7ff7ac007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9fc930640 1 --2- 172.21.15.114:0/3220627640 >> v2:172.21.15.149:6800/1092505261 conn(0x7fb9e403d320 0x7fb9e403f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 1 -- 172.21.15.114:0/879980778 <== osd.2 v2:172.21.15.114:6804/1932657732 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7a4017e70 con 0x7ff79c03d2b0 2024-09-16T05:07:49.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 10 client.4266.objecter ms_dispatch 0x7ff7bc000ba0 command_reply(tid 1: 0 ) 2024-09-16T05:07:49.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 10 client.4266.objecter _finish_command 1 = system:0 2024-09-16T05:07:49.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 20 client.4266.objecter put_session s=0x7ff79c03cc00 osd=2 4 2024-09-16T05:07:49.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7ff7b37fe640 15 client.4266.objecter _session_command_op_remove 2 1 2024-09-16T05:07:49.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9fc930640 10 monclient: get_auth_request con 0x7fb9e403d320 auth_method 0 2024-09-16T05:07:49.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 10 client.4229.objecter ms_handle_connect 0x7fb9e4037810 2024-09-16T05:07:49.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9fd131640 1 --2- 172.21.15.114:0/3220627640 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb9e4037810 0x7fb9e4039cd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0001570 tx=0x7fb9f0007ba0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9f5ffb640 10 client.4229.objecter ms_handle_connect 0x7fb9e403d320 2024-09-16T05:07:49.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.462+0000 7fb9fc930640 1 --2- 172.21.15.114:0/3220627640 >> v2:172.21.15.149:6800/1092505261 conn(0x7fb9e403d320 0x7fb9e403f760 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.466+0000 7f90197f5640 1 --2- 172.21.15.114:0/2892680095 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8ff8037840 0x7f8ff8039d00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f901406f890 tx=0x7f900c00b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.466+0000 7f900a7fc640 10 client.4263.objecter ms_handle_connect 0x7f8ff8037840 2024-09-16T05:07:49.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.466+0000 7fb9f5ffb640 1 -- 172.21.15.114:0/3220627640 <== osd.4 v2:172.21.15.149:6800/1092505261 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9e80aee90 con 0x7fb9e403d320 2024-09-16T05:07:49.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.466+0000 7fb9f5ffb640 10 client.4229.objecter ms_dispatch 0x7fb9f8000ba0 command_reply(tid 1: 0 ) 2024-09-16T05:07:49.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.466+0000 7fb9f5ffb640 10 client.4229.objecter _finish_command 1 = system:0 2024-09-16T05:07:49.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.466+0000 7fb9f5ffb640 20 client.4229.objecter put_session s=0x7fb9e403cc70 osd=4 4 2024-09-16T05:07:49.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.466+0000 7fb9f5ffb640 15 client.4229.objecter _session_command_op_remove 4 1 2024-09-16T05:07:49.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.470+0000 7f901a7f7640 1 --2- 172.21.15.114:0/2892680095 >> v2:172.21.15.114:6800/2903720358 conn(0x7f8ff803d350 0x7f8ff803f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.470+0000 7f901a7f7640 10 monclient: get_auth_request con 0x7f8ff803d350 auth_method 0 2024-09-16T05:07:49.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.470+0000 7f901a7f7640 1 --2- 172.21.15.114:0/2892680095 >> v2:172.21.15.114:6800/2903720358 conn(0x7f8ff803d350 0x7f8ff803f790 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.470+0000 7f900a7fc640 10 client.4263.objecter ms_handle_connect 0x7f8ff803d350 2024-09-16T05:07:49.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.470+0000 7f900a7fc640 1 -- 172.21.15.114:0/2892680095 <== osd.1 v2:172.21.15.114:6800/2903720358 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9004011e70 con 0x7f8ff803d350 2024-09-16T05:07:49.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.470+0000 7f900a7fc640 10 client.4263.objecter ms_dispatch 0x7f9014000ba0 command_reply(tid 1: 0 ) 2024-09-16T05:07:49.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.470+0000 7f900a7fc640 10 client.4263.objecter _finish_command 1 = system:0 2024-09-16T05:07:49.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.470+0000 7f900a7fc640 20 client.4263.objecter put_session s=0x7f8ff803cca0 osd=1 4 2024-09-16T05:07:49.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.470+0000 7f900a7fc640 15 client.4263.objecter _session_command_op_remove 1 1 2024-09-16T05:07:49.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.474+0000 7ff7c17db640 10 client.4266.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T05:07:49.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.474+0000 7ff7c17db640 15 client.4266.objecter _session_command_op_assign -1 2 2024-09-16T05:07:49.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.474+0000 7ff7c17db640 20 client.4266.objecter _get_session s=0x7ff79c03cc00 osd=2 3 2024-09-16T05:07:49.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.474+0000 7ff7c17db640 20 client.4266.objecter put_session s=0x7ff79c03cc00 osd=2 3 2024-09-16T05:07:49.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.474+0000 7ff7c17db640 20 client.4266.objecter _get_session s=0x7ff79c03cc00 osd=2 3 2024-09-16T05:07:49.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.474+0000 7ff7c17db640 15 client.4266.objecter _session_command_op_remove -1 2 2024-09-16T05:07:49.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.474+0000 7ff7c17db640 20 client.4266.objecter get_session s=0x7ff79c03cc00 osd=2 3 2024-09-16T05:07:49.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.474+0000 7ff7c17db640 15 client.4266.objecter _session_command_op_assign 2 2 2024-09-16T05:07:49.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.474+0000 7ff7c17db640 20 client.4266.objecter put_session s=0x7ff79c03cc00 osd=2 4 2024-09-16T05:07:49.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.474+0000 7ff7c17db640 10 client.4266.objecter _send_command 2 2024-09-16T05:07:49.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.474+0000 7ff7c17db640 1 -- 172.21.15.114:0/879980778 --> v2:172.21.15.114:6804/1932657732 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff7bc1375b0 con 0x7ff79c03d2b0 2024-09-16T05:07:49.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.474+0000 7ff7b37fe640 1 -- 172.21.15.114:0/879980778 <== osd.2 v2:172.21.15.114:6804/1932657732 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff7bc1375b0 con 0x7ff79c03d2b0 2024-09-16T05:07:49.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.474+0000 7ff7b37fe640 10 client.4266.objecter ms_dispatch 0x7ff7bc000ba0 command_reply(tid 2: 0 ) 2024-09-16T05:07:49.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.474+0000 7ff7b37fe640 10 client.4266.objecter _finish_command 2 = system:0 2024-09-16T05:07:49.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7b37fe640 20 client.4266.objecter put_session s=0x7ff79c03cc00 osd=2 4 2024-09-16T05:07:49.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7b37fe640 15 client.4266.objecter _session_command_op_remove 2 2 2024-09-16T05:07:49.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 10 librados: watch_flush enter 2024-09-16T05:07:49.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 10 librados: watch_flush exit 2024-09-16T05:07:49.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 10 client.4266.objecter close_session for osd.2 2024-09-16T05:07:49.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 1 -- 172.21.15.114:0/879980778 >> v2:172.21.15.114:6804/1932657732 conn(0x7ff79c03d2b0 msgr2=0x7ff79c03f6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 1 --2- 172.21.15.114:0/879980778 >> v2:172.21.15.114:6804/1932657732 conn(0x7ff79c03d2b0 0x7ff79c03f6f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 20 client.4266.objecter put_session s=0x7ff79c03cc00 osd=2 1 2024-09-16T05:07:49.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 20 client.4266.objecter shutdown clearing up homeless session... 2024-09-16T05:07:49.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 10 client.4266.objecter successfully canceled tick 2024-09-16T05:07:49.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 1 -- 172.21.15.114:0/879980778 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff79c037810 msgr2=0x7ff79c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 1 --2- 172.21.15.114:0/879980778 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff79c037810 0x7ff79c039cd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac0011f0 tx=0x7ff7ac007d40 comp rx=0 tx=0).stop 2024-09-16T05:07:49.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 10 monclient: shutdown 2024-09-16T05:07:49.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 1 -- 172.21.15.114:0/879980778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7bc1371d0 msgr2=0x7ff7bc13b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 1 --2- 172.21.15.114:0/879980778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7bc1371d0 0x7ff7bc13b440 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff7a400c450 tx=0x7ff7a400c910 comp rx=0 tx=0).stop 2024-09-16T05:07:49.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7bb7fe640 1 -- 172.21.15.114:0/879980778 reap_dead start 2024-09-16T05:07:49.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 1 -- 172.21.15.114:0/879980778 shutdown_connections 2024-09-16T05:07:49.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 1 -- 172.21.15.114:0/879980778 >> 172.21.15.114:0/879980778 conn(0x7ff7bc0a9be0 msgr2=0x7ff7bc09f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 1 -- 172.21.15.114:0/879980778 shutdown_connections 2024-09-16T05:07:49.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 1 -- 172.21.15.114:0/879980778 wait complete. 2024-09-16T05:07:49.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7ff7c17db640 1 librados: shutdown 2024-09-16T05:07:49.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 10 client.4260.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T05:07:49.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 15 client.4260.objecter _session_command_op_assign -1 2 2024-09-16T05:07:49.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 20 client.4260.objecter _get_session s=0x7f2ae403cc30 osd=3 3 2024-09-16T05:07:49.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 20 client.4260.objecter put_session s=0x7f2ae403cc30 osd=3 3 2024-09-16T05:07:49.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 20 client.4260.objecter _get_session s=0x7f2ae403cc30 osd=3 3 2024-09-16T05:07:49.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 15 client.4260.objecter _session_command_op_remove -1 2 2024-09-16T05:07:49.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 20 client.4260.objecter get_session s=0x7f2ae403cc30 osd=3 3 2024-09-16T05:07:49.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 15 client.4260.objecter _session_command_op_assign 3 2 2024-09-16T05:07:49.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 20 client.4260.objecter put_session s=0x7f2ae403cc30 osd=3 4 2024-09-16T05:07:49.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 10 client.4260.objecter _send_command 2 2024-09-16T05:07:49.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 1 -- 172.21.15.114:0/1920052488 --> v2:172.21.15.149:6805/1362772130 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2b0009edd0 con 0x7f2ae403d2e0 2024-09-16T05:07:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2af67fc640 1 -- 172.21.15.114:0/1920052488 <== osd.3 v2:172.21.15.149:6805/1362772130 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2b0009edd0 con 0x7f2ae403d2e0 2024-09-16T05:07:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2af67fc640 10 client.4260.objecter ms_dispatch 0x7f2b00000ba0 command_reply(tid 2: 0 ) 2024-09-16T05:07:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2af67fc640 10 client.4260.objecter _finish_command 2 = system:0 2024-09-16T05:07:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2af67fc640 20 client.4260.objecter put_session s=0x7f2ae403cc30 osd=3 4 2024-09-16T05:07:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2af67fc640 15 client.4260.objecter _session_command_op_remove 3 2 2024-09-16T05:07:49.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 10 librados: watch_flush enter 2024-09-16T05:07:49.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 10 librados: watch_flush exit 2024-09-16T05:07:49.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 10 client.4260.objecter close_session for osd.3 2024-09-16T05:07:49.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 1 -- 172.21.15.114:0/1920052488 >> v2:172.21.15.149:6805/1362772130 conn(0x7f2ae403d2e0 msgr2=0x7f2ae403f720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 1 --2- 172.21.15.114:0/1920052488 >> v2:172.21.15.149:6805/1362772130 conn(0x7f2ae403d2e0 0x7f2ae403f720 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 20 client.4260.objecter put_session s=0x7f2ae403cc30 osd=3 1 2024-09-16T05:07:49.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 20 client.4260.objecter shutdown clearing up homeless session... 2024-09-16T05:07:49.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 10 client.4260.objecter successfully canceled tick 2024-09-16T05:07:49.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 1 -- 172.21.15.114:0/1920052488 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2ae4037840 msgr2=0x7f2ae4039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 1 --2- 172.21.15.114:0/1920052488 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2ae4037840 0x7f2ae4039d00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2b0009e0f0 tx=0x7f2af00079c0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 10 monclient: shutdown 2024-09-16T05:07:49.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 1 -- 172.21.15.114:0/1920052488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b000a4270 msgr2=0x7f2b0013bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 1 --2- 172.21.15.114:0/1920052488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b000a4270 0x7f2b0013bdb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2afc0097f0 tx=0x7f2afc01ac30 comp rx=0 tx=0).stop 2024-09-16T05:07:49.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b06040640 1 -- 172.21.15.114:0/1920052488 reap_dead start 2024-09-16T05:07:49.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 1 -- 172.21.15.114:0/1920052488 shutdown_connections 2024-09-16T05:07:49.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 1 -- 172.21.15.114:0/1920052488 >> 172.21.15.114:0/1920052488 conn(0x7f2b000a9a40 msgr2=0x7f2b000a8600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 1 -- 172.21.15.114:0/1920052488 shutdown_connections 2024-09-16T05:07:49.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 1 -- 172.21.15.114:0/1920052488 wait complete. 2024-09-16T05:07:49.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7f2b07aa3640 1 librados: shutdown 2024-09-16T05:07:49.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7fb9ff395640 10 client.4229.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T05:07:49.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7fb9ff395640 15 client.4229.objecter _session_command_op_assign -1 2 2024-09-16T05:07:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7fb9ff395640 20 client.4229.objecter _get_session s=0x7fb9e403cc70 osd=4 3 2024-09-16T05:07:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7fb9ff395640 20 client.4229.objecter put_session s=0x7fb9e403cc70 osd=4 3 2024-09-16T05:07:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7fb9ff395640 20 client.4229.objecter _get_session s=0x7fb9e403cc70 osd=4 3 2024-09-16T05:07:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7fb9ff395640 15 client.4229.objecter _session_command_op_remove -1 2 2024-09-16T05:07:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7fb9ff395640 20 client.4229.objecter get_session s=0x7fb9e403cc70 osd=4 3 2024-09-16T05:07:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7fb9ff395640 15 client.4229.objecter _session_command_op_assign 4 2 2024-09-16T05:07:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7fb9ff395640 20 client.4229.objecter put_session s=0x7fb9e403cc70 osd=4 4 2024-09-16T05:07:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7fb9ff395640 10 client.4229.objecter _send_command 2 2024-09-16T05:07:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.478+0000 7fb9ff395640 1 -- 172.21.15.114:0/3220627640 --> v2:172.21.15.149:6800/1092505261 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb9f80a93e0 con 0x7fb9e403d320 2024-09-16T05:07:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9f5ffb640 1 -- 172.21.15.114:0/3220627640 <== osd.4 v2:172.21.15.149:6800/1092505261 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb9f80a93e0 con 0x7fb9e403d320 2024-09-16T05:07:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9f5ffb640 10 client.4229.objecter ms_dispatch 0x7fb9f8000ba0 command_reply(tid 2: 0 ) 2024-09-16T05:07:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9f5ffb640 10 client.4229.objecter _finish_command 2 = system:0 2024-09-16T05:07:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9f5ffb640 20 client.4229.objecter put_session s=0x7fb9e403cc70 osd=4 4 2024-09-16T05:07:49.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9f5ffb640 15 client.4229.objecter _session_command_op_remove 4 2 2024-09-16T05:07:49.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 10 librados: watch_flush enter 2024-09-16T05:07:49.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 10 librados: watch_flush exit 2024-09-16T05:07:49.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 10 client.4229.objecter close_session for osd.4 2024-09-16T05:07:49.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 1 -- 172.21.15.114:0/3220627640 >> v2:172.21.15.149:6800/1092505261 conn(0x7fb9e403d320 msgr2=0x7fb9e403f760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 1 --2- 172.21.15.114:0/3220627640 >> v2:172.21.15.149:6800/1092505261 conn(0x7fb9e403d320 0x7fb9e403f760 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 20 client.4229.objecter put_session s=0x7fb9e403cc70 osd=4 1 2024-09-16T05:07:49.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 20 client.4229.objecter shutdown clearing up homeless session... 2024-09-16T05:07:49.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 10 client.4229.objecter successfully canceled tick 2024-09-16T05:07:49.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 1 -- 172.21.15.114:0/3220627640 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb9e4037810 msgr2=0x7fb9e4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 1 --2- 172.21.15.114:0/3220627640 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb9e4037810 0x7fb9e4039cd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0001570 tx=0x7fb9f0007ba0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 10 monclient: shutdown 2024-09-16T05:07:49.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 1 -- 172.21.15.114:0/3220627640 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9f8131850 msgr2=0x7fb9f81e2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 1 --2- 172.21.15.114:0/3220627640 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9f8131850 0x7fb9f81e2220 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb9e801cc40 tx=0x7fb9e0005840 comp rx=0 tx=0).stop 2024-09-16T05:07:49.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9fd932640 1 -- 172.21.15.114:0/3220627640 reap_dead start 2024-09-16T05:07:49.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 1 -- 172.21.15.114:0/3220627640 shutdown_connections 2024-09-16T05:07:49.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 1 -- 172.21.15.114:0/3220627640 >> 172.21.15.114:0/3220627640 conn(0x7fb9f80a9a40 msgr2=0x7fb9f80a8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 1 -- 172.21.15.114:0/3220627640 shutdown_connections 2024-09-16T05:07:49.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 1 -- 172.21.15.114:0/3220627640 wait complete. 2024-09-16T05:07:49.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.482+0000 7fb9ff395640 1 librados: shutdown 2024-09-16T05:07:49.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f8ff7fff640 10 client.4263.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T05:07:49.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f8ff7fff640 15 client.4263.objecter _session_command_op_assign -1 2 2024-09-16T05:07:49.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f8ff7fff640 20 client.4263.objecter _get_session s=0x7f8ff803cca0 osd=1 3 2024-09-16T05:07:49.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f8ff7fff640 20 client.4263.objecter put_session s=0x7f8ff803cca0 osd=1 3 2024-09-16T05:07:49.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f8ff7fff640 20 client.4263.objecter _get_session s=0x7f8ff803cca0 osd=1 3 2024-09-16T05:07:49.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f8ff7fff640 15 client.4263.objecter _session_command_op_remove -1 2 2024-09-16T05:07:49.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f8ff7fff640 20 client.4263.objecter get_session s=0x7f8ff803cca0 osd=1 3 2024-09-16T05:07:49.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f8ff7fff640 15 client.4263.objecter _session_command_op_assign 1 2 2024-09-16T05:07:49.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f8ff7fff640 20 client.4263.objecter put_session s=0x7f8ff803cca0 osd=1 4 2024-09-16T05:07:49.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f8ff7fff640 10 client.4263.objecter _send_command 2 2024-09-16T05:07:49.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f8ff7fff640 1 -- 172.21.15.114:0/2892680095 --> v2:172.21.15.114:6800/2903720358 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f901412e330 con 0x7f8ff803d350 2024-09-16T05:07:49.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f900a7fc640 1 -- 172.21.15.114:0/2892680095 <== osd.1 v2:172.21.15.114:6800/2903720358 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f901412e330 con 0x7f8ff803d350 2024-09-16T05:07:49.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f900a7fc640 10 client.4263.objecter ms_dispatch 0x7f9014000ba0 command_reply(tid 2: 0 ) 2024-09-16T05:07:49.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f900a7fc640 10 client.4263.objecter _finish_command 2 = system:0 2024-09-16T05:07:49.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f900a7fc640 20 client.4263.objecter put_session s=0x7f8ff803cca0 osd=1 4 2024-09-16T05:07:49.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.494+0000 7f900a7fc640 15 client.4263.objecter _session_command_op_remove 1 2 2024-09-16T05:07:49.676 INFO:teuthology.orchestra.run.smithi114.stdout:55834574852 2024-09-16T05:07:49.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T05:07:49.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 10 librados: watch_flush enter 2024-09-16T05:07:49.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 10 librados: watch_flush exit 2024-09-16T05:07:49.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 10 client.4263.objecter close_session for osd.1 2024-09-16T05:07:49.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 1 -- 172.21.15.114:0/2892680095 >> v2:172.21.15.114:6800/2903720358 conn(0x7f8ff803d350 msgr2=0x7f8ff803f790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 1 --2- 172.21.15.114:0/2892680095 >> v2:172.21.15.114:6800/2903720358 conn(0x7f8ff803d350 0x7f8ff803f790 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 20 client.4263.objecter put_session s=0x7f8ff803cca0 osd=1 1 2024-09-16T05:07:49.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 20 client.4263.objecter shutdown clearing up homeless session... 2024-09-16T05:07:49.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 10 client.4263.objecter successfully canceled tick 2024-09-16T05:07:49.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 1 -- 172.21.15.114:0/2892680095 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8ff8037840 msgr2=0x7f8ff8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 1 --2- 172.21.15.114:0/2892680095 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8ff8037840 0x7f8ff8039d00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f901406f890 tx=0x7f900c00b040 comp rx=0 tx=0).stop 2024-09-16T05:07:49.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 10 monclient: shutdown 2024-09-16T05:07:49.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 1 -- 172.21.15.114:0/2892680095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f901409cdb0 msgr2=0x7f901412bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 1 --2- 172.21.15.114:0/2892680095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f901409cdb0 0x7f901412bf10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f900400c410 tx=0x7f900400c8d0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f901a7f7640 1 -- 172.21.15.114:0/2892680095 reap_dead start 2024-09-16T05:07:49.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 1 -- 172.21.15.114:0/2892680095 shutdown_connections 2024-09-16T05:07:49.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 1 -- 172.21.15.114:0/2892680095 >> 172.21.15.114:0/2892680095 conn(0x7f90140a3270 msgr2=0x7f90140a1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 1 -- 172.21.15.114:0/2892680095 shutdown_connections 2024-09-16T05:07:49.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 1 -- 172.21.15.114:0/2892680095 wait complete. 2024-09-16T05:07:49.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.498+0000 7f8ff7fff640 1 librados: shutdown 2024-09-16T05:07:49.680 INFO:teuthology.orchestra.run.smithi114.stdout:47244640261 2024-09-16T05:07:49.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T05:07:49.681 INFO:teuthology.orchestra.run.smithi114.stdout:47244640261 2024-09-16T05:07:49.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T05:07:49.682 INFO:teuthology.orchestra.run.smithi114.stdout:51539607557 2024-09-16T05:07:49.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T05:07:49.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.810+0000 7f9e7f4f7640 10 monclient: get_monmap_and_config 2024-09-16T05:07:49.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.810+0000 7f9e7f4f7640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.810+0000 7f9e7f4f7640 10 monclient: monmap: 2024-09-16T05:07:49.811 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.812 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.812 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:49.814396+0000 2024-09-16T05:07:49.812 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:49.814396+0000 2024-09-16T05:07:49.812 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.812 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.812 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:49.812 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:49.812 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:49.812 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:49.812 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:49.812 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.810+0000 7f9e7f4f7640 1 Processor -- start 2024-09-16T05:07:49.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.810+0000 7f9e7f4f7640 1 -- start start 2024-09-16T05:07:49.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.810+0000 7f9e7f4f7640 10 monclient: init 2024-09-16T05:07:49.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.810+0000 7f9e7f4f7640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.810+0000 7f9e7f4f7640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-16T05:07:49.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.810+0000 7f9e7f4f7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9e781376e0 0x7f9e78137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.810+0000 7f9e7f4f7640 10 monclient(hunting): picked mon.noname-a con 0x7f9e781376e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:49.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.810+0000 7f9e7d293640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9e781376e0 0x7f9e78137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9e781376e0 0x7f9e78137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54830/0 (socket says 172.21.15.114:54830) 2024-09-16T05:07:49.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 1 -- 172.21.15.114:0/4145786225 learned_addr learned my addr 172.21.15.114:0/4145786225 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7f4f7640 1 --2- 172.21.15.114:0/4145786225 >> v2:172.21.15.149:3300/0 conn(0x7f9e781370a0 0x7f9e780a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7f4f7640 10 monclient(hunting): picked mon.noname-c con 0x7f9e781370a0 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:49.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7f4f7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e780a5d00 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:49.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/4145786225 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e7809bd20 con 0x7f9e780a5d00 2024-09-16T05:07:49.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7f4f7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/4145786225 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f9e78138000 con 0x7f9e781376e0 2024-09-16T05:07:49.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7f4f7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/4145786225 --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f9e7809c5c0 con 0x7f9e781370a0 2024-09-16T05:07:49.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7f4f7640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7f4f7640 10 monclient(hunting): authenticate will time out at 954.444345s 2024-09-16T05:07:49.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 10 monclient(hunting): get_auth_request con 0x7f9e781376e0 auth_method 0 2024-09-16T05:07:49.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 10 monclient(hunting): handle_auth_done global_id 4269 payload 274 2024-09-16T05:07:49.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7ca92640 1 --2- 172.21.15.114:0/4145786225 >> v2:172.21.15.149:3300/0 conn(0x7f9e781370a0 0x7f9e780a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 1 -- 172.21.15.114:0/4145786225 >> v2:172.21.15.149:3300/0 conn(0x7f9e781370a0 msgr2=0x7f9e780a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:49.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 1 --2- 172.21.15.114:0/4145786225 >> v2:172.21.15.149:3300/0 conn(0x7f9e781370a0 0x7f9e780a4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 1 -- 172.21.15.114:0/4145786225 >> v1:172.21.15.114:6789/0 conn(0x7f9e780a5d00 legacy=0x7f9e780a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 1 monclient: found mon.noname-a 2024-09-16T05:07:49.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 1 -- 172.21.15.114:0/4145786225 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e7809c000 con 0x7f9e781376e0 2024-09-16T05:07:49.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7d293640 1 --2- 172.21.15.114:0/4145786225 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e781376e0 0x7f9e78137ac0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9e78073760 tx=0x7f9e7401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6fee8a176bd3fcb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e6ffff640 1 -- 172.21.15.114:0/4145786225 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e740b1070 con 0x7f9e781376e0 2024-09-16T05:07:49.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e6ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:49.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e6ffff640 10 monclient: dump: 2024-09-16T05:07:49.819 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.819 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.819 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.819 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.820 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.820 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.820 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.820 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.820 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.820 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e6ffff640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e6ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e6ffff640 1 -- 172.21.15.114:0/4145786225 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e740ac640 con 0x7f9e781376e0 2024-09-16T05:07:49.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7f4f7640 5 monclient: authenticate success, global_id 4269 2024-09-16T05:07:49.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.814+0000 7f9e7f4f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:49.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e6ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:49.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 4 set_mon_vals no callback set 2024-09-16T05:07:49.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e6ffff640 1 -- 172.21.15.114:0/4145786225 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e740aca50 con 0x7f9e781376e0 2024-09-16T05:07:49.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:49.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e6ffff640 10 monclient: dump: 2024-09-16T05:07:49.821 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.821 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.821 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.821 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.821 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.822 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.822 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.822 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.822 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.822 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 10 monclient: shutdown 2024-09-16T05:07:49.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/4145786225 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e781376e0 msgr2=0x7f9e78137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 1 --2- 172.21.15.114:0/4145786225 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e781376e0 0x7f9e78137ac0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9e78073760 tx=0x7f9e7401c400 comp rx=0 tx=0).stop 2024-09-16T05:07:49.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/4145786225 shutdown_connections 2024-09-16T05:07:49.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 1 --2- 172.21.15.114:0/4145786225 >> v2:172.21.15.149:3300/0 conn(0x7f9e781370a0 0x7f9e780a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 1 --2- 172.21.15.114:0/4145786225 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e781376e0 0x7f9e78137ac0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/4145786225 >> 172.21.15.114:0/4145786225 conn(0x7f9e780a9a40 msgr2=0x7f9e780a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/4145786225 shutdown_connections 2024-09-16T05:07:49.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/4145786225 wait complete. 2024-09-16T05:07:49.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 10 monclient: monmap: 2024-09-16T05:07:49.823 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.824 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:49.824 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:49.824 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:49.824 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.824 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.824 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:49.824 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:49.824 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:49.824 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 1 librados: starting msgr at 2024-09-16T05:07:49.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7fab44c24640 10 monclient: get_monmap_and_config 2024-09-16T05:07:49.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 1 librados: starting objecter 2024-09-16T05:07:49.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7fab44c24640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7fab44c24640 10 monclient: monmap: 2024-09-16T05:07:49.825 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.826 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.826 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:49.823237+0000 2024-09-16T05:07:49.826 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:49.823237+0000 2024-09-16T05:07:49.826 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.826 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.826 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:49.826 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:49.826 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:49.826 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:49.826 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:49.826 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 1 Processor -- start 2024-09-16T05:07:49.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 1 -- start start 2024-09-16T05:07:49.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 1 librados: setting wanted keys 2024-09-16T05:07:49.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 1 librados: calling monclient init 2024-09-16T05:07:49.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7f9e7f4f7640 10 monclient: init 2024-09-16T05:07:49.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7fab44c24640 1 Processor -- start 2024-09-16T05:07:49.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7fab44c24640 1 -- start start 2024-09-16T05:07:49.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.818+0000 7fab44c24640 10 monclient: init 2024-09-16T05:07:49.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:07:49.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e780a5d00 0x7f9e781357a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7d293640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e780a5d00 0x7f9e781357a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7d293640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e780a5d00 0x7f9e781357a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54834/0 (socket says 172.21.15.114:54834) 2024-09-16T05:07:49.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7d293640 1 -- 172.21.15.114:0/55919518 learned_addr learned my addr 172.21.15.114:0/55919518 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 10 monclient(hunting): picked mon.noname-a con 0x7f9e780a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:49.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 1 --2- 172.21.15.114:0/55919518 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e781370a0 0x7f9e78135ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 10 monclient(hunting): picked mon.noname-c con 0x7f9e781370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 1 --2- 172.21.15.114:0/55919518 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e781376e0 0x7f9e7813bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab44c24640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 10 monclient(hunting): picked mon.noname-b con 0x7f9e781376e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:49.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab44c24640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-16T05:07:49.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/55919518 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9e78141d80 con 0x7f9e780a5d00 2024-09-16T05:07:49.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/55919518 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f9e78141c00 con 0x7f9e781376e0 2024-09-16T05:07:49.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab44c24640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fab401376e0 0x7fab40137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab44c24640 10 monclient(hunting): picked mon.noname-a con 0x7fab401376e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:49.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/55919518 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9e78141f00 con 0x7f9e781370a0 2024-09-16T05:07:49.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab44c24640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab401370a0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:49.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab44c24640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab400a5d00 0x7fab400a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3e59c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fab401376e0 0x7fab40137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 10 monclient(hunting): authenticate will time out at 954.453006s 2024-09-16T05:07:49.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 1 --2- 172.21.15.114:0/55919518 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e781370a0 0x7f9e78135ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3e59c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fab401376e0 0x7fab40137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54842/0 (socket says 172.21.15.114:54842) 2024-09-16T05:07:49.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7d293640 10 monclient(hunting): get_auth_request con 0x7f9e780a5d00 auth_method 0 2024-09-16T05:07:49.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7d293640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7d293640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab400a5d00 0x7fab400a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab400a5d00 0x7fab400a4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55552/0 (socket says 172.21.15.114:55552) 2024-09-16T05:07:49.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7d293640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 10 monclient(hunting): get_auth_request con 0x7f9e781370a0 auth_method 0 2024-09-16T05:07:49.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7da94640 1 --2- 172.21.15.114:0/55919518 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e781376e0 0x7f9e7813bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3e59c640 1 -- 172.21.15.114:0/1218379820 learned_addr learned my addr 172.21.15.114:0/1218379820 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7da94640 10 monclient(hunting): get_auth_request con 0x7f9e781376e0 auth_method 0 2024-09-16T05:07:49.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7da94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7da94640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7da94640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab44c24640 10 monclient(hunting): picked mon.noname-b con 0x7fab400a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab44c24640 1 -- 172.21.15.114:0/1218379820 --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab4009bd20 con 0x7fab401370a0 2024-09-16T05:07:49.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab44c24640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab44c24640 1 -- 172.21.15.114:0/1218379820 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fab40138000 con 0x7fab401376e0 2024-09-16T05:07:49.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab44c24640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab44c24640 1 -- 172.21.15.114:0/1218379820 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fab4009c5c0 con 0x7fab400a5d00 2024-09-16T05:07:49.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab44c24640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab44c24640 10 monclient(hunting): authenticate will time out at 954.453559s 2024-09-16T05:07:49.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3e59c640 10 monclient(hunting): get_auth_request con 0x7fab401376e0 auth_method 0 2024-09-16T05:07:49.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7d293640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7d293640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7d293640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7da94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7da94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7da94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2024-09-16T05:07:49.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 1 -- 172.21.15.114:0/55919518 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e781376e0 msgr2=0x7f9e7813bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 1 --2- 172.21.15.114:0/55919518 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e781376e0 0x7f9e7813bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 1 -- 172.21.15.114:0/55919518 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e780a5d00 msgr2=0x7f9e781357a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 1 --2- 172.21.15.114:0/55919518 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e780a5d00 0x7f9e781357a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 1 monclient: found mon.noname-c 2024-09-16T05:07:49.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 1 -- 172.21.15.114:0/55919518 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e781201d0 con 0x7f9e781370a0 2024-09-16T05:07:49.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 10 monclient(hunting): get_auth_request con 0x7fab400a5d00 auth_method 0 2024-09-16T05:07:49.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3dd9b640 1 -- 172.21.15.114:0/1218379820 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2170110987 0 0) 0x7fab4009bd20 con 0x7fab401370a0 2024-09-16T05:07:49.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3dd9b640 10 monclient(hunting): _negotiate 2024-09-16T05:07:49.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3dd9b640 10 monclient(hunting): my global_id is 4291 2024-09-16T05:07:49.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3dd9b640 1 -- 172.21.15.114:0/1218379820 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab2c003530 con 0x7fab401370a0 2024-09-16T05:07:49.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2024-09-16T05:07:49.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 1 -- 172.21.15.114:0/1218379820 >> v2:172.21.15.114:3300/0 conn(0x7fab401376e0 msgr2=0x7fab40137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:49.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 1 --2- 172.21.15.114:0/1218379820 >> v2:172.21.15.114:3300/0 conn(0x7fab401376e0 0x7fab40137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 1 -- 172.21.15.114:0/1218379820 >> v1:172.21.15.149:6789/0 conn(0x7fab401370a0 legacy=0x7fab400a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 1 monclient: found mon.noname-b 2024-09-16T05:07:49.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7fab3ed9d640 1 -- 172.21.15.114:0/1218379820 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4009c000 con 0x7fab400a5d00 2024-09-16T05:07:49.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7ca92640 1 --2- 172.21.15.114:0/55919518 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e781370a0 0x7f9e78135ce0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800c820 tx=0x7f9e6800cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6dffb640 10 client.?.objecter ms_handle_connect 0x7f9e781370a0 2024-09-16T05:07:49.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:49.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6dffb640 1 -- 172.21.15.114:0/55919518 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e68015070 con 0x7f9e781370a0 2024-09-16T05:07:49.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:49.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6dffb640 10 monclient: dump: 2024-09-16T05:07:49.846 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.846 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.846 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.846 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.846 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.847 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.847 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.847 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.847 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.847 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6dffb640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6dffb640 1 -- 172.21.15.114:0/55919518 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e68004020 con 0x7f9e781370a0 2024-09-16T05:07:49.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6dffb640 1 -- 172.21.15.114:0/55919518 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e680043f0 con 0x7f9e781370a0 2024-09-16T05:07:49.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6dffb640 10 monclient: dump: 2024-09-16T05:07:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.848 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e6f7fe640 4 set_mon_vals no callback set 2024-09-16T05:07:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 5 monclient: authenticate success, global_id 4232 2024-09-16T05:07:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 10 monclient: _renew_subs 2024-09-16T05:07:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/55919518 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e7813e270 con 0x7f9e781370a0 2024-09-16T05:07:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 10 monclient: _renew_subs 2024-09-16T05:07:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.822+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/55919518 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e781204b0 con 0x7f9e781370a0 2024-09-16T05:07:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e7f4f7640 1 librados: init done 2024-09-16T05:07:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e6dffb640 1 -- 172.21.15.114:0/55919518 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9e68007830 con 0x7f9e781370a0 2024-09-16T05:07:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e6dffb640 1 --2- 172.21.15.114:0/55919518 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9e5c037810 0x7f9e5c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e6dffb640 1 -- 172.21.15.114:0/55919518 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f9e68048490 con 0x7f9e781370a0 2024-09-16T05:07:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e6dffb640 10 client.4232.objecter ms_dispatch 0x7f9e78000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e6dffb640 3 client.4232.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e6dffb640 3 client.4232.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e6dffb640 20 client.4232.objecter dump_active .. 0 homeless 2024-09-16T05:07:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e7f4f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e7f4f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e7f4f7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/55919518 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e780a2060 con 0x7f9e781370a0 2024-09-16T05:07:49.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7fab3ed9d640 1 --2- 172.21.15.114:0/1218379820 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab400a5d00 0x7fab400a4980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fab38009d90 tx=0x7fab3801c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a646d1d84df0658 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e7d293640 1 --2- 172.21.15.114:0/55919518 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9e5c037810 0x7f9e5c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e7d293640 10 monclient: get_auth_request con 0x7f9e5c037810 auth_method 0 2024-09-16T05:07:49.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e7d293640 1 --2- 172.21.15.114:0/55919518 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9e5c037810 0x7f9e5c039cd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9e74000c10 tx=0x7f9e74008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.826+0000 7f9e6dffb640 10 client.4232.objecter ms_handle_connect 0x7f9e5c037810 2024-09-16T05:07:49.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab3dd9b640 1 -- 172.21.15.114:0/1218379820 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab380b2070 con 0x7fab400a5d00 2024-09-16T05:07:49.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab3dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab3dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:49.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab3dd9b640 10 monclient: dump: 2024-09-16T05:07:49.852 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.852 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.852 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.852 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.853 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.853 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.853 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.853 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.853 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.853 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab3dd9b640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab3dd9b640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab3dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab44c24640 5 monclient: authenticate success, global_id 4235 2024-09-16T05:07:49.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab44c24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:49.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab3dd9b640 1 -- 172.21.15.114:0/1218379820 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab380a5800 con 0x7fab400a5d00 2024-09-16T05:07:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab3dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab44c24640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab44c24640 4 set_mon_vals no callback set 2024-09-16T05:07:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7f9e6dffb640 1 -- 172.21.15.114:0/55919518 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f9e68010350 con 0x7f9e781370a0 2024-09-16T05:07:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7f9e6dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7f9e6dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:49.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab3dd9b640 1 -- 172.21.15.114:0/1218379820 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab380a5bb0 con 0x7fab400a5d00 2024-09-16T05:07:49.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab3dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab3dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:49.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab3dd9b640 10 monclient: dump: 2024-09-16T05:07:49.859 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.860 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.860 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.860 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.860 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.860 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.860 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.860 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.860 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.860 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab44c24640 10 monclient: shutdown 2024-09-16T05:07:49.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab44c24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab44c24640 1 -- 172.21.15.114:0/1218379820 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab400a5d00 msgr2=0x7fab400a4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab44c24640 1 --2- 172.21.15.114:0/1218379820 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab400a5d00 0x7fab400a4980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fab38009d90 tx=0x7fab3801c830 comp rx=0 tx=0).stop 2024-09-16T05:07:49.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab44c24640 1 -- 172.21.15.114:0/1218379820 shutdown_connections 2024-09-16T05:07:49.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab44c24640 1 --2- 172.21.15.114:0/1218379820 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab400a5d00 0x7fab400a4980 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab44c24640 1 --2- 172.21.15.114:0/1218379820 >> v2:172.21.15.114:3300/0 conn(0x7fab401376e0 0x7fab40137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab44c24640 1 -- 172.21.15.114:0/1218379820 >> 172.21.15.114:0/1218379820 conn(0x7fab400a9a40 msgr2=0x7fab400a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab44c24640 1 -- 172.21.15.114:0/1218379820 shutdown_connections 2024-09-16T05:07:49.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.834+0000 7fab44c24640 1 -- 172.21.15.114:0/1218379820 wait complete. 2024-09-16T05:07:49.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient: monmap: 2024-09-16T05:07:49.861 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.862 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:49.862 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:49.862 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:49.862 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.862 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.862 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:49.862 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:49.862 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:49.862 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 1 librados: starting msgr at 2024-09-16T05:07:49.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 1 librados: starting objecter 2024-09-16T05:07:49.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 1 Processor -- start 2024-09-16T05:07:49.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 1 -- start start 2024-09-16T05:07:49.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 1 librados: setting wanted keys 2024-09-16T05:07:49.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 1 librados: calling monclient init 2024-09-16T05:07:49.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient: init 2024-09-16T05:07:49.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:07:49.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab400a5d00 0x7fab40138650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient(hunting): picked mon.noname-c con 0x7fab400a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab401370a0 0x7fab40138b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient(hunting): picked mon.noname-a con 0x7fab401370a0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:49.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fab401376e0 0x7fab401d3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient(hunting): picked mon.noname-b con 0x7fab401376e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:49.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fab40141d80 con 0x7fab401370a0 2024-09-16T05:07:49.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fab40141c00 con 0x7fab401376e0 2024-09-16T05:07:49.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fab40141f00 con 0x7fab400a5d00 2024-09-16T05:07:49.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient(hunting): authenticate will time out at 954.469683s 2024-09-16T05:07:49.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab401370a0 0x7fab40138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab401370a0 0x7fab40138b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54850/0 (socket says 172.21.15.114:54850) 2024-09-16T05:07:49.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 1 -- 172.21.15.114:0/2483134955 learned_addr learned my addr 172.21.15.114:0/2483134955 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 10 monclient(hunting): get_auth_request con 0x7fab401370a0 auth_method 0 2024-09-16T05:07:49.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab3e59c640 1 --2- 172.21.15.114:0/2483134955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab400a5d00 0x7fab40138650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab3e59c640 10 monclient(hunting): get_auth_request con 0x7fab400a5d00 auth_method 0 2024-09-16T05:07:49.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2024-09-16T05:07:49.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 1 -- 172.21.15.114:0/2483134955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab400a5d00 msgr2=0x7fab40138650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab3ed9d640 1 --2- 172.21.15.114:0/2483134955 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fab401376e0 0x7fab401d3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 1 --2- 172.21.15.114:0/2483134955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab400a5d00 0x7fab40138650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 1 -- 172.21.15.114:0/2483134955 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fab401376e0 msgr2=0x7fab401d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 1 --2- 172.21.15.114:0/2483134955 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fab401376e0 0x7fab401d3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 1 monclient: found mon.noname-a 2024-09-16T05:07:49.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 1 -- 172.21.15.114:0/2483134955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4013afb0 con 0x7fab401370a0 2024-09-16T05:07:49.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab3e59c640 1 --2- 172.21.15.114:0/2483134955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab400a5d00 0x7fab40138650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:07:49.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab37fff640 1 --2- 172.21.15.114:0/2483134955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab401370a0 0x7fab40138b90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fab2800c490 tx=0x7fab2800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab36ffd640 10 client.?.objecter ms_handle_connect 0x7fab401370a0 2024-09-16T05:07:49.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab36ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:49.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab36ffd640 1 -- 172.21.15.114:0/2483134955 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab280052d0 con 0x7fab401370a0 2024-09-16T05:07:49.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:49.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab36ffd640 10 monclient: dump: 2024-09-16T05:07:49.867 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.867 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.867 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.867 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.868 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.868 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.868 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.868 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.868 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.868 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab36ffd640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab36ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 5 monclient: authenticate success, global_id 4281 2024-09-16T05:07:49.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient: _renew_subs 2024-09-16T05:07:49.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab36ffd640 1 -- 172.21.15.114:0/2483134955 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab28005470 con 0x7fab401370a0 2024-09-16T05:07:49.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7f4443668640 10 monclient: get_monmap_and_config 2024-09-16T05:07:49.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 1 -- 172.21.15.114:0/2483134955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab400a8c40 con 0x7fab401370a0 2024-09-16T05:07:49.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab36ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab36ffd640 1 -- 172.21.15.114:0/2483134955 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab28012870 con 0x7fab401370a0 2024-09-16T05:07:49.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7f4443668640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7f4443668640 10 monclient: monmap: 2024-09-16T05:07:49.870 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.870 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.870 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:49.843305+0000 2024-09-16T05:07:49.870 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:49.843305+0000 2024-09-16T05:07:49.870 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.870 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.870 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:49.870 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:49.871 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:49.871 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:49.871 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:49.871 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:49.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab36ffd640 10 monclient: dump: 2024-09-16T05:07:49.872 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.872 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.872 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.872 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.872 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.872 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.872 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.873 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient: _renew_subs 2024-09-16T05:07:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 1 -- 172.21.15.114:0/2483134955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab400a1d40 con 0x7fab401370a0 2024-09-16T05:07:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 1 librados: init done 2024-09-16T05:07:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab3d59a640 4 set_mon_vals no callback set 2024-09-16T05:07:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.838+0000 7fab44c24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fab44c24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fab44c24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fab44c24640 1 -- 172.21.15.114:0/2483134955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab40137ac0 con 0x7fab401370a0 2024-09-16T05:07:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fab36ffd640 1 -- 172.21.15.114:0/2483134955 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fab28012a10 con 0x7fab401370a0 2024-09-16T05:07:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fab36ffd640 1 --2- 172.21.15.114:0/2483134955 >> v2:172.21.15.149:6812/1185531135 conn(0x7fab20037810 0x7fab20039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fab36ffd640 1 -- 172.21.15.114:0/2483134955 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7fab28050350 con 0x7fab401370a0 2024-09-16T05:07:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fab36ffd640 10 client.4281.objecter ms_dispatch 0x7fab40000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fab36ffd640 3 client.4281.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fab36ffd640 3 client.4281.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fab36ffd640 20 client.4281.objecter dump_active .. 0 homeless 2024-09-16T05:07:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fd4b0a0c640 10 monclient: get_monmap_and_config 2024-09-16T05:07:49.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fd4b0a0c640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fd4b0a0c640 10 monclient: monmap: 2024-09-16T05:07:49.875 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.875 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.875 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:49.845285+0000 2024-09-16T05:07:49.875 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:49.845285+0000 2024-09-16T05:07:49.875 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.875 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.875 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:49.875 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:49.876 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:49.876 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:49.876 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:49.876 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 1 Processor -- start 2024-09-16T05:07:49.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fab3e59c640 1 --2- 172.21.15.114:0/2483134955 >> v2:172.21.15.149:6812/1185531135 conn(0x7fab20037810 0x7fab20039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 1 -- start start 2024-09-16T05:07:49.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 10 monclient: init 2024-09-16T05:07:49.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fab3e59c640 10 monclient: get_auth_request con 0x7fab20037810 auth_method 0 2024-09-16T05:07:49.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-16T05:07:49.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f443c1376e0 0x7f443c137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 10 monclient(hunting): picked mon.noname-b con 0x7f443c1376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f443c1370a0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:49.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f443c0a5d00 0x7f443c0a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 10 monclient(hunting): picked mon.noname-c con 0x7f443c0a5d00 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:49.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f443c09bd20 con 0x7f443c1370a0 2024-09-16T05:07:49.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f443c138000 con 0x7f443c0a5d00 2024-09-16T05:07:49.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f443c09c5c0 con 0x7f443c1376e0 2024-09-16T05:07:49.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4443668640 10 monclient(hunting): authenticate will time out at 954.474037s 2024-09-16T05:07:49.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fab36ffd640 10 client.4281.objecter ms_handle_connect 0x7fab20037810 2024-09-16T05:07:49.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fab3e59c640 1 --2- 172.21.15.114:0/2483134955 >> v2:172.21.15.149:6812/1185531135 conn(0x7fab20037810 0x7fab20039cd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fab30009ba0 tx=0x7fab30008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f443c0a5d00 0x7f443c0a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f443c0a5d00 0x7f443c0a4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.114:45074/0 (socket says 172.21.15.114:45074) 2024-09-16T05:07:49.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 1 -- 172.21.15.114:0/1891554342 learned_addr learned my addr 172.21.15.114:0/1891554342 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 10 monclient(hunting): get_auth_request con 0x7f443c0a5d00 auth_method 0 2024-09-16T05:07:49.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 10 monclient(hunting): handle_auth_done global_id 4294 payload 274 2024-09-16T05:07:49.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 1 -- 172.21.15.114:0/1891554342 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f443c1376e0 msgr2=0x7f443c137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:49.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 1 --2- 172.21.15.114:0/1891554342 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f443c1376e0 0x7f443c137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 1 -- 172.21.15.114:0/1891554342 >> v1:172.21.15.149:6789/0 conn(0x7f443c1370a0 legacy=0x7f443c0a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 1 monclient: found mon.noname-c 2024-09-16T05:07:49.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:07:49.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7f4441c05640 1 -- 172.21.15.114:0/1891554342 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443c09c000 con 0x7f443c0a5d00 2024-09-16T05:07:49.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fd4b0a0c640 1 Processor -- start 2024-09-16T05:07:49.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fd4b0a0c640 1 -- start start 2024-09-16T05:07:49.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fd4b0a0c640 10 monclient: init 2024-09-16T05:07:49.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fd4b0a0c640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fd4b0a0c640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-16T05:07:49.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fd4b0a0c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4a4023cb0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:49.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fd4b0a0c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd4a4024090 0x7fd4a40244a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fd4b0a0c640 10 monclient(hunting): picked mon.noname-a con 0x7fd4a4024090 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fd4b0a0c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4a40c0990 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fd4b0a0c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4a4027e30 con 0x7fd4a40c0990 2024-09-16T05:07:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.842+0000 7fd4b0a0c640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4a4027590 con 0x7fd4a4023cb0 2024-09-16T05:07:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4b0a0c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4b0a0c640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd4a4027be0 con 0x7fd4a4024090 2024-09-16T05:07:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4b0a0c640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4b0a0c640 10 monclient(hunting): authenticate will time out at 954.476017s 2024-09-16T05:07:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4ab59e640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd4a40c0990 0x7fd4a40c2d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38616/0 (socket says 172.21.15.114:38616) 2024-09-16T05:07:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4ab59e640 1 -- 172.21.15.114:0/1165185508 learned_addr learned my addr 172.21.15.114:0/1165185508 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 1 --2- 172.21.15.114:0/1165185508 >> v2:172.21.15.114:3300/0 conn(0x7fd4a4024090 0x7fd4a40244a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 10 monclient(hunting): get_auth_request con 0x7fd4a4024090 auth_method 0 2024-09-16T05:07:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4441c05640 1 --2- 172.21.15.114:0/1891554342 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f443c0a5d00 0x7f443c0a4980 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4434009e50 tx=0x7f443401c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43587beebf002a1e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4433fff640 1 -- 172.21.15.114:0/1891554342 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44340a56c0 con 0x7f443c0a5d00 2024-09-16T05:07:49.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4433fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4433fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:49.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4433fff640 10 monclient: dump: 2024-09-16T05:07:49.885 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.885 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.885 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.885 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.885 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.886 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.886 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.886 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.886 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.886 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4433fff640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4433fff640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4433fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4433fff640 1 -- 172.21.15.114:0/1891554342 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44340a5860 con 0x7f443c0a5d00 2024-09-16T05:07:49.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4433fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4433fff640 1 -- 172.21.15.114:0/1891554342 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44340a5c30 con 0x7f443c0a5d00 2024-09-16T05:07:49.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4433fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4433fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:49.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4433fff640 10 monclient: dump: 2024-09-16T05:07:49.887 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.887 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.887 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.887 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.887 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.887 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.887 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.887 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.887 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.887 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 5 monclient: authenticate success, global_id 4294 2024-09-16T05:07:49.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:49.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 4 set_mon_vals no callback set 2024-09-16T05:07:49.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 10 monclient: shutdown 2024-09-16T05:07:49.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 1 -- 172.21.15.114:0/1891554342 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f443c0a5d00 msgr2=0x7f443c0a4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 1 --2- 172.21.15.114:0/1891554342 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f443c0a5d00 0x7f443c0a4980 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4434009e50 tx=0x7f443401c8f0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 1 -- 172.21.15.114:0/1891554342 shutdown_connections 2024-09-16T05:07:49.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 1 --2- 172.21.15.114:0/1891554342 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f443c0a5d00 0x7f443c0a4980 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 1 --2- 172.21.15.114:0/1891554342 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f443c1376e0 0x7f443c137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 1 -- 172.21.15.114:0/1891554342 >> 172.21.15.114:0/1891554342 conn(0x7f443c0a9a40 msgr2=0x7f443c0a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 1 -- 172.21.15.114:0/1891554342 shutdown_connections 2024-09-16T05:07:49.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 1 -- 172.21.15.114:0/1891554342 wait complete. 2024-09-16T05:07:49.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2024-09-16T05:07:49.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 1 -- 172.21.15.114:0/1165185508 >> v1:172.21.15.149:6789/0 conn(0x7fd4a4023cb0 legacy=0x7fd4a4025370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 1 -- 172.21.15.114:0/1165185508 >> v1:172.21.15.114:6789/0 conn(0x7fd4a40c0990 legacy=0x7fd4a40c2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 1 monclient: found mon.noname-a 2024-09-16T05:07:49.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 1 -- 172.21.15.114:0/1165185508 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a40296d0 con 0x7fd4a4024090 2024-09-16T05:07:49.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 10 monclient: monmap: 2024-09-16T05:07:49.890 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.890 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:49.890 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:49.890 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:49.890 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.890 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.890 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:49.890 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:49.890 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:49.890 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 1 librados: starting msgr at 2024-09-16T05:07:49.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7f4443668640 1 librados: starting objecter 2024-09-16T05:07:49.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.846+0000 7fd4aa59c640 1 --2- 172.21.15.114:0/1165185508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4a4024090 0x7fd4a40244a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd4a00097c0 tx=0x7fd4a001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b98970ccf7ee3f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 1 Processor -- start 2024-09-16T05:07:49.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 1 -- start start 2024-09-16T05:07:49.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 1 librados: setting wanted keys 2024-09-16T05:07:49.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 1 librados: calling monclient init 2024-09-16T05:07:49.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient: init 2024-09-16T05:07:49.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:07:49.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f443c0a5d00 0x7f443c1cf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient(hunting): picked mon.noname-a con 0x7f443c0a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:49.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f443c1370a0 0x7f443c1362a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient(hunting): picked mon.noname-c con 0x7f443c1370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f443c1376e0 0x7f443c13c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient(hunting): picked mon.noname-b con 0x7f443c1376e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:49.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f443c141d80 con 0x7f443c0a5d00 2024-09-16T05:07:49.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f443c141c00 con 0x7f443c1376e0 2024-09-16T05:07:49.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f443c141f00 con 0x7f443c1370a0 2024-09-16T05:07:49.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient(hunting): authenticate will time out at 954.480954s 2024-09-16T05:07:49.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fab36ffd640 1 -- 172.21.15.114:0/2483134955 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fab280182b0 con 0x7fab401370a0 2024-09-16T05:07:49.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fab36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fab36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:49.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4a9d9b640 1 -- 172.21.15.114:0/1165185508 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a00b0070 con 0x7fd4a4024090 2024-09-16T05:07:49.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4a9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4a9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:49.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4a9d9b640 10 monclient: dump: 2024-09-16T05:07:49.895 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.895 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.895 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.895 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.895 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.895 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.895 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.895 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.895 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.895 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4a9d9b640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4a9d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4b0a0c640 5 monclient: authenticate success, global_id 4284 2024-09-16T05:07:49.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4b0a0c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:49.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f443c1370a0 0x7f443c1362a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f443c1370a0 0x7f443c1362a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55574/0 (socket says 172.21.15.114:55574) 2024-09-16T05:07:49.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 1 -- 172.21.15.114:0/2159788967 learned_addr learned my addr 172.21.15.114:0/2159788967 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 10 monclient(hunting): get_auth_request con 0x7f443c1370a0 auth_method 0 2024-09-16T05:07:49.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441404640 1 --2- 172.21.15.114:0/2159788967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f443c0a5d00 0x7f443c1cf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441404640 10 monclient(hunting): get_auth_request con 0x7f443c0a5d00 auth_method 0 2024-09-16T05:07:49.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441404640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441404640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441c05640 1 --2- 172.21.15.114:0/2159788967 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f443c1376e0 0x7f443c13c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441c05640 10 monclient(hunting): get_auth_request con 0x7f443c1376e0 auth_method 0 2024-09-16T05:07:49.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441c05640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441c05640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441404640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441404640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441404640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441c05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441c05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4441c05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 10 monclient(hunting): handle_auth_done global_id 4241 payload 824 2024-09-16T05:07:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 1 -- 172.21.15.114:0/2159788967 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f443c1376e0 msgr2=0x7f443c13c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 1 --2- 172.21.15.114:0/2159788967 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f443c1376e0 0x7f443c13c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 1 -- 172.21.15.114:0/2159788967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f443c0a5d00 msgr2=0x7f443c1cf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 1 --2- 172.21.15.114:0/2159788967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f443c0a5d00 0x7f443c1cf1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 1 monclient: found mon.noname-c 2024-09-16T05:07:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 1 -- 172.21.15.114:0/2159788967 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443c0a2600 con 0x7f443c1370a0 2024-09-16T05:07:49.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4440c03640 1 --2- 172.21.15.114:0/2159788967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f443c1370a0 0x7f443c1362a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f442c00c430 tx=0x7f442c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4431ffb640 10 client.?.objecter ms_handle_connect 0x7f443c1370a0 2024-09-16T05:07:49.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4431ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:49.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4a9d9b640 1 -- 172.21.15.114:0/1165185508 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4a00ab600 con 0x7fd4a4024090 2024-09-16T05:07:49.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4a9d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4b0a0c640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:49.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4b0a0c640 4 set_mon_vals no callback set 2024-09-16T05:07:49.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4a9d9b640 1 -- 172.21.15.114:0/1165185508 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a00aba10 con 0x7fd4a4024090 2024-09-16T05:07:49.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4a9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4a9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:49.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4a9d9b640 10 monclient: dump: 2024-09-16T05:07:49.901 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.901 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.901 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.901 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.901 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.901 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.901 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.901 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.901 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.902 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4b0a0c640 10 monclient: shutdown 2024-09-16T05:07:49.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4b0a0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4b0a0c640 1 -- 172.21.15.114:0/1165185508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4a4024090 msgr2=0x7fd4a40244a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7fd4b0a0c640 1 --2- 172.21.15.114:0/1165185508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4a4024090 0x7fd4a40244a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd4a00097c0 tx=0x7fd4a001c410 comp rx=0 tx=0).stop 2024-09-16T05:07:49.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4431ffb640 1 -- 172.21.15.114:0/2159788967 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c017070 con 0x7f443c1370a0 2024-09-16T05:07:49.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4431ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4431ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:49.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4431ffb640 10 monclient: dump: 2024-09-16T05:07:49.903 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.903 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.903 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.903 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.903 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.903 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.903 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.903 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.904 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.904 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4431ffb640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4431ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4431ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4431ffb640 1 -- 172.21.15.114:0/2159788967 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f442c0052d0 con 0x7f443c1370a0 2024-09-16T05:07:49.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 5 monclient: authenticate success, global_id 4241 2024-09-16T05:07:49.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient: _renew_subs 2024-09-16T05:07:49.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 1 -- 172.21.15.114:0/2159788967 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f443c073d20 con 0x7f443c1370a0 2024-09-16T05:07:49.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4431ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4431ffb640 1 -- 172.21.15.114:0/2159788967 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c003840 con 0x7f443c1370a0 2024-09-16T05:07:49.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4431ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4431ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:49.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4431ffb640 10 monclient: dump: 2024-09-16T05:07:49.905 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.905 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.905 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.905 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.905 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.905 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.905 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.905 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.905 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.905 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient: _renew_subs 2024-09-16T05:07:49.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 1 -- 172.21.15.114:0/2159788967 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f443c120c30 con 0x7f443c1370a0 2024-09-16T05:07:49.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 1 librados: init done 2024-09-16T05:07:49.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f44337fe640 4 set_mon_vals no callback set 2024-09-16T05:07:49.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.850+0000 7f4443668640 1 -- 172.21.15.114:0/2159788967 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f443c137ac0 con 0x7f443c1370a0 2024-09-16T05:07:49.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.854+0000 7fd4b0a0c640 1 -- 172.21.15.114:0/1165185508 shutdown_connections 2024-09-16T05:07:49.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.854+0000 7fd4b0a0c640 1 --2- 172.21.15.114:0/1165185508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4a4024090 0x7fd4a40244a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.854+0000 7fd4b0a0c640 1 -- 172.21.15.114:0/1165185508 >> 172.21.15.114:0/1165185508 conn(0x7fd4a4006320 msgr2=0x7fd4a4006730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.854+0000 7f4431ffb640 1 -- 172.21.15.114:0/2159788967 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f442c01a070 con 0x7f443c1370a0 2024-09-16T05:07:49.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.854+0000 7f4431ffb640 1 --2- 172.21.15.114:0/2159788967 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4420037810 0x7f4420039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.854+0000 7f4431ffb640 1 -- 172.21.15.114:0/2159788967 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f442c048050 con 0x7f443c1370a0 2024-09-16T05:07:49.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.854+0000 7f4431ffb640 10 client.4241.objecter ms_dispatch 0x7f443c000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:49.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.854+0000 7f4431ffb640 3 client.4241.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:49.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.854+0000 7f4431ffb640 3 client.4241.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:49.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.854+0000 7f4431ffb640 20 client.4241.objecter dump_active .. 0 homeless 2024-09-16T05:07:49.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.854+0000 7f4441404640 1 --2- 172.21.15.114:0/2159788967 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4420037810 0x7f4420039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.854+0000 7f4441404640 10 monclient: get_auth_request con 0x7f4420037810 auth_method 0 2024-09-16T05:07:49.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.854+0000 7f4431ffb640 10 client.4241.objecter ms_handle_connect 0x7f4420037810 2024-09-16T05:07:49.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.854+0000 7f4441404640 1 --2- 172.21.15.114:0/2159788967 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4420037810 0x7f4420039cd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4438009990 tx=0x7f4438009110 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 -- 172.21.15.114:0/1165185508 shutdown_connections 2024-09-16T05:07:49.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 -- 172.21.15.114:0/1165185508 wait complete. 2024-09-16T05:07:49.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7f4431ffb640 1 -- 172.21.15.114:0/2159788967 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f442c021020 con 0x7f443c1370a0 2024-09-16T05:07:49.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7f4431ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7f4431ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:49.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient: monmap: 2024-09-16T05:07:49.912 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.912 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:49.912 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:49.912 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:49.912 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.913 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.913 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:49.913 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:49.913 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:49.913 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 librados: starting msgr at 2024-09-16T05:07:49.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 librados: starting objecter 2024-09-16T05:07:49.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 Processor -- start 2024-09-16T05:07:49.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 -- start start 2024-09-16T05:07:49.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 librados: setting wanted keys 2024-09-16T05:07:49.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 librados: calling monclient init 2024-09-16T05:07:49.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient: init 2024-09-16T05:07:49.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:07:49.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4a4023cb0 0x7fd4a40bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient(hunting): picked mon.noname-c con 0x7fd4a4023cb0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4a4024090 0x7fd4a401e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient(hunting): picked mon.noname-b con 0x7fd4a4024090 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:49.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4a40c0990 0x7fd4a401e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient(hunting): picked mon.noname-a con 0x7fd4a40c0990 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:49.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a40c7050 con 0x7fd4a40c0990 2024-09-16T05:07:49.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a40c6ed0 con 0x7fd4a4024090 2024-09-16T05:07:49.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd4a40c71d0 con 0x7fd4a4023cb0 2024-09-16T05:07:49.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient(hunting): authenticate will time out at 954.489870s 2024-09-16T05:07:49.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4a40c0990 0x7fd4a401e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4a40c0990 0x7fd4a401e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54886/0 (socket says 172.21.15.114:54886) 2024-09-16T05:07:49.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 1 -- 172.21.15.114:0/3555570503 learned_addr learned my addr 172.21.15.114:0/3555570503 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 10 monclient(hunting): get_auth_request con 0x7fd4a40c0990 auth_method 0 2024-09-16T05:07:49.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2024-09-16T05:07:49.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 1 -- 172.21.15.114:0/3555570503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4a4023cb0 msgr2=0x7fd4a40bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 1 --2- 172.21.15.114:0/3555570503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4a4023cb0 0x7fd4a40bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 1 -- 172.21.15.114:0/3555570503 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4a4024090 msgr2=0x7fd4a401e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 1 --2- 172.21.15.114:0/3555570503 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4a4024090 0x7fd4a401e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 1 monclient: found mon.noname-a 2024-09-16T05:07:49.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 1 -- 172.21.15.114:0/3555570503 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a40c3f10 con 0x7fd4a40c0990 2024-09-16T05:07:49.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4ab59e640 1 --2- 172.21.15.114:0/3555570503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4a40c0990 0x7fd4a401e8f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac075580 tx=0x7fd4ac069950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4937fe640 10 client.?.objecter ms_handle_connect 0x7fd4a40c0990 2024-09-16T05:07:49.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4937fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:49.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4937fe640 1 -- 172.21.15.114:0/3555570503 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac091070 con 0x7fd4a40c0990 2024-09-16T05:07:49.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:49.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4937fe640 10 monclient: dump: 2024-09-16T05:07:49.917 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.917 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.917 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.917 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.917 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.917 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.917 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.917 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.917 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.917 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4937fe640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4937fe640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4937fe640 1 -- 172.21.15.114:0/3555570503 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4ac0759d0 con 0x7fd4a40c0990 2024-09-16T05:07:49.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4937fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4937fe640 1 -- 172.21.15.114:0/3555570503 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac075dc0 con 0x7fd4a40c0990 2024-09-16T05:07:49.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:49.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4937fe640 10 monclient: dump: 2024-09-16T05:07:49.918 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.918 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.918 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.918 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.918 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.918 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.918 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.919 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.919 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.919 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4a959a640 4 set_mon_vals no callback set 2024-09-16T05:07:49.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 5 monclient: authenticate success, global_id 4290 2024-09-16T05:07:49.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient: _renew_subs 2024-09-16T05:07:49.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 -- 172.21.15.114:0/3555570503 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4a40ad580 con 0x7fd4a40c0990 2024-09-16T05:07:49.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient: _renew_subs 2024-09-16T05:07:49.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 -- 172.21.15.114:0/3555570503 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4a4020000 con 0x7fd4a40c0990 2024-09-16T05:07:49.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.858+0000 7fd4b0a0c640 1 librados: init done 2024-09-16T05:07:49.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.862+0000 7fd4b0a0c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.862+0000 7fd4b0a0c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.862+0000 7fd4b0a0c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.862+0000 7fd4b0a0c640 1 -- 172.21.15.114:0/3555570503 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4a40acce0 con 0x7fd4a40c0990 2024-09-16T05:07:49.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.866+0000 7fd4937fe640 1 -- 172.21.15.114:0/3555570503 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd4ac076030 con 0x7fd4a40c0990 2024-09-16T05:07:49.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.866+0000 7fd4937fe640 1 --2- 172.21.15.114:0/3555570503 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd498037810 0x7fd498039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.866+0000 7fd4937fe640 1 -- 172.21.15.114:0/3555570503 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7fd4ac0c3df0 con 0x7fd4a40c0990 2024-09-16T05:07:49.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.866+0000 7fd4937fe640 10 client.4290.objecter ms_dispatch 0x7fd4ac000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:49.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.866+0000 7fd4937fe640 3 client.4290.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:49.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.866+0000 7fd4937fe640 3 client.4290.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:49.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.866+0000 7fd4937fe640 20 client.4290.objecter dump_active .. 0 homeless 2024-09-16T05:07:49.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.866+0000 7fd4aad9d640 1 --2- 172.21.15.114:0/3555570503 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd498037810 0x7fd498039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.866+0000 7fd4aad9d640 10 monclient: get_auth_request con 0x7fd498037810 auth_method 0 2024-09-16T05:07:49.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.866+0000 7fd4aad9d640 1 --2- 172.21.15.114:0/3555570503 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd498037810 0x7fd498039cd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd49c000f70 tx=0x7fd49c0079e0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.866+0000 7fd4937fe640 1 -- 172.21.15.114:0/3555570503 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd4a40c3f10 con 0x7fd4a40c0990 2024-09-16T05:07:49.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.866+0000 7fd4937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.866+0000 7fd4937fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:49.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.866+0000 7fd4937fe640 10 client.4290.objecter ms_handle_connect 0x7fd498037810 2024-09-16T05:07:49.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 10 monclient: get_monmap_and_config 2024-09-16T05:07:49.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 10 monclient: monmap: 2024-09-16T05:07:49.930 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.931 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.931 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:49.873087+0000 2024-09-16T05:07:49.931 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:49.873087+0000 2024-09-16T05:07:49.931 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.931 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.931 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:49.931 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:49.931 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:49.931 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:49.931 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:49.931 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 1 Processor -- start 2024-09-16T05:07:49.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 1 -- start start 2024-09-16T05:07:49.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 10 monclient: init 2024-09-16T05:07:49.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-16T05:07:49.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f41b00a1fb0 0x7f41b009e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 10 monclient(hunting): picked mon.noname-c con 0x7f41b00a1fb0 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:49.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f41b00a2390 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:49.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41b013ca40 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:49.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41b009e810 con 0x7f41b00a2390 2024-09-16T05:07:49.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41b00a0a70 con 0x7f41b013ca40 2024-09-16T05:07:49.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f41b0073ed0 con 0x7f41b00a1fb0 2024-09-16T05:07:49.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41b4f1e640 10 monclient(hunting): authenticate will time out at 954.502709s 2024-09-16T05:07:49.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41add9b640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f41b00a2390 0x7f41b00a27a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38618/0 (socket says 172.21.15.114:38618) 2024-09-16T05:07:49.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41add9b640 1 -- 172.21.15.114:0/2992020534 learned_addr learned my addr 172.21.15.114:0/2992020534 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:49.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ae59c640 1 --2- 172.21.15.114:0/2992020534 >> v2:172.21.15.149:3300/0 conn(0x7f41b00a1fb0 0x7f41b009e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ae59c640 10 monclient(hunting): get_auth_request con 0x7f41b00a1fb0 auth_method 0 2024-09-16T05:07:49.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 -- 172.21.15.114:0/2992020534 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4276194206 0 0) 0x7f41b009e810 con 0x7f41b00a2390 2024-09-16T05:07:49.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient(hunting): _negotiate 2024-09-16T05:07:49.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient(hunting): my global_id is 4293 2024-09-16T05:07:49.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 -- 172.21.15.114:0/2992020534 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4198003530 con 0x7f41b00a2390 2024-09-16T05:07:49.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 -- 172.21.15.114:0/2992020534 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1166768582 0 0) 0x7f4198003530 con 0x7f41b00a2390 2024-09-16T05:07:49.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 -- 172.21.15.114:0/2992020534 >> v2:172.21.15.149:3300/0 conn(0x7f41b00a1fb0 msgr2=0x7f41b009e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:49.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 --2- 172.21.15.114:0/2992020534 >> v2:172.21.15.149:3300/0 conn(0x7f41b00a1fb0 0x7f41b009e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 -- 172.21.15.114:0/2992020534 >> v1:172.21.15.149:6789/0 conn(0x7f41b013ca40 legacy=0x7f41b013ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:07:49.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:07:49.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 -- 172.21.15.114:0/2992020534 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b0076400 con 0x7f41b00a2390 2024-09-16T05:07:49.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 -- 172.21.15.114:0/2992020534 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f41a40016a0 con 0x7f41b00a2390 2024-09-16T05:07:49.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:07:49.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient: dump: 2024-09-16T05:07:49.935 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.935 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.935 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.935 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.935 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.935 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.935 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.935 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.935 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.935 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:07:49.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 -- 172.21.15.114:0/2992020534 >> v1:172.21.15.114:6789/0 conn(0x7f41b00a2390 legacy=0x7f41b00a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:07:49.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 --2- 172.21.15.114:0/2992020534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4198003a10 0x7f4198003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient(hunting): picked mon.c con 0x7f4198003a10 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 --2- 172.21.15.114:0/2992020534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4198004810 0x7f4198004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient(hunting): picked mon.a con 0x7f4198004810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:49.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 --2- 172.21.15.114:0/2992020534 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41980052f0 0x7f419800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient(hunting): picked mon.b con 0x7f41980052f0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:49.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 -- 172.21.15.114:0/2992020534 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f41b0073ed0 con 0x7f4198004810 2024-09-16T05:07:49.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 -- 172.21.15.114:0/2992020534 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f419800c1f0 con 0x7f41980052f0 2024-09-16T05:07:49.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 1 -- 172.21.15.114:0/2992020534 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f419800c3f0 con 0x7f4198003a10 2024-09-16T05:07:49.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.870+0000 7f41ad59a640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:07:49.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41ae59c640 1 --2- 172.21.15.114:0/2992020534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4198003a10 0x7f4198003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 1 --2- 172.21.15.114:0/2992020534 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41980052f0 0x7f419800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41add9b640 1 --2- 172.21.15.114:0/2992020534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4198004810 0x7f4198004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 10 monclient(hunting): get_auth_request con 0x7f41980052f0 auth_method 0 2024-09-16T05:07:49.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:49.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41add9b640 10 monclient(hunting): get_auth_request con 0x7f4198004810 auth_method 0 2024-09-16T05:07:49.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41add9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41add9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:49.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41ae59c640 10 monclient(hunting): get_auth_request con 0x7f4198003a10 auth_method 0 2024-09-16T05:07:49.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41ae59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:49.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:07:49.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41add9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:07:49.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:07:49.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 10 monclient(hunting): handle_auth_done global_id 4293 payload 293 2024-09-16T05:07:49.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 1 -- 172.21.15.114:0/2992020534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4198003a10 msgr2=0x7f4198003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:49.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 1 --2- 172.21.15.114:0/2992020534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4198003a10 0x7f4198003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 1 -- 172.21.15.114:0/2992020534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4198004810 msgr2=0x7f4198004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:49.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 1 --2- 172.21.15.114:0/2992020534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4198004810 0x7f4198004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 1 monclient: found mon.b 2024-09-16T05:07:49.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:49.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 1 -- 172.21.15.114:0/2992020534 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4198003530 con 0x7f41980052f0 2024-09-16T05:07:49.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41add9b640 1 --2- 172.21.15.114:0/2992020534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4198004810 0x7f4198004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:49.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41ae59c640 1 --2- 172.21.15.114:0/2992020534 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4198003a10 0x7f4198003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:49.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41aed9d640 1 -- 172.21.15.114:0/2992020534 reap_dead start 2024-09-16T05:07:49.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41b4f1e640 5 monclient: authenticate success, global_id 4293 2024-09-16T05:07:49.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.874+0000 7f41b4f1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:49.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41aed9d640 1 --2- 172.21.15.114:0/2992020534 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41980052f0 0x7f419800b9d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f41a8001e70 tx=0x7f41a801c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0f0737a309b91f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41ad59a640 1 -- 172.21.15.114:0/2992020534 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a00096b0 con 0x7f41980052f0 2024-09-16T05:07:49.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41ad59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:49.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41ad59a640 10 monclient: dump: 2024-09-16T05:07:49.941 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.941 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.941 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.941 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.941 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.941 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.941 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.941 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.941 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.941 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:49.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41ad59a640 1 -- 172.21.15.114:0/2992020534 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a8003ba0 con 0x7f41980052f0 2024-09-16T05:07:49.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41ad59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:49.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41ad59a640 1 -- 172.21.15.114:0/2992020534 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4007c90 con 0x7f41980052f0 2024-09-16T05:07:49.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 4 set_mon_vals no callback set 2024-09-16T05:07:49.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41ad59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:49.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41ad59a640 10 monclient: dump: 2024-09-16T05:07:49.942 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.942 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.942 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.942 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.942 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.942 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.942 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.942 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.942 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.943 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 10 monclient: shutdown 2024-09-16T05:07:49.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2992020534 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41980052f0 msgr2=0x7f419800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 1 --2- 172.21.15.114:0/2992020534 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41980052f0 0x7f419800b9d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f41a8001e70 tx=0x7f41a801c7f0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2992020534 shutdown_connections 2024-09-16T05:07:49.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 1 --2- 172.21.15.114:0/2992020534 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41980052f0 0x7f419800b9d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2992020534 >> 172.21.15.114:0/2992020534 conn(0x7f41b00a3270 msgr2=0x7f41b00a3680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2992020534 shutdown_connections 2024-09-16T05:07:49.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2992020534 wait complete. 2024-09-16T05:07:49.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 10 monclient: monmap: 2024-09-16T05:07:49.943 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.943 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:49.944 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:49.944 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:49.944 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.944 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.944 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:49.944 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:49.944 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:49.944 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 1 librados: starting msgr at 2024-09-16T05:07:49.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 1 librados: starting objecter 2024-09-16T05:07:49.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 1 Processor -- start 2024-09-16T05:07:49.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 1 -- start start 2024-09-16T05:07:49.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 1 librados: setting wanted keys 2024-09-16T05:07:49.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 1 librados: calling monclient init 2024-09-16T05:07:49.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.878+0000 7f41b4f1e640 10 monclient: init 2024-09-16T05:07:49.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:07:49.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41980052f0 0x7f41b013ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient(hunting): picked mon.noname-a con 0x7f41980052f0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:49.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41b0099530 0x7f41b00999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient(hunting): picked mon.noname-b con 0x7f41b0099530 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:49.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41b008d000 0x7f41b008d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41980052f0 0x7f41b013ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41980052f0 0x7f41b013ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54914/0 (socket says 172.21.15.114:54914) 2024-09-16T05:07:49.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41aed9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41b008d000 0x7f41b008d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41aed9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41b008d000 0x7f41b008d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55610/0 (socket says 172.21.15.114:55610) 2024-09-16T05:07:49.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient(hunting): picked mon.noname-c con 0x7f41b008d000 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 1 -- 172.21.15.114:0/2353673701 learned_addr learned my addr 172.21.15.114:0/2353673701 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2353673701 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f41b009e660 con 0x7f41980052f0 2024-09-16T05:07:49.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2353673701 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f41b009e4e0 con 0x7f41b0099530 2024-09-16T05:07:49.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2353673701 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f41b00a27a0 con 0x7f41b008d000 2024-09-16T05:07:49.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient(hunting): authenticate will time out at 954.512657s 2024-09-16T05:07:49.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 10 monclient(hunting): get_auth_request con 0x7f41980052f0 auth_method 0 2024-09-16T05:07:49.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41add9b640 1 --2- 172.21.15.114:0/2353673701 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41b0099530 0x7f41b00999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41aed9d640 10 monclient(hunting): get_auth_request con 0x7f41b008d000 auth_method 0 2024-09-16T05:07:49.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41add9b640 10 monclient(hunting): get_auth_request con 0x7f41b0099530 auth_method 0 2024-09-16T05:07:49.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41add9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41add9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 10 monclient(hunting): handle_auth_done global_id 4296 payload 824 2024-09-16T05:07:49.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 1 -- 172.21.15.114:0/2353673701 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41b008d000 msgr2=0x7f41b008d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 1 --2- 172.21.15.114:0/2353673701 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41b008d000 0x7f41b008d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 1 -- 172.21.15.114:0/2353673701 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41b0099530 msgr2=0x7f41b00999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 1 --2- 172.21.15.114:0/2353673701 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41b0099530 0x7f41b00999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 1 monclient: found mon.noname-a 2024-09-16T05:07:49.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 1 -- 172.21.15.114:0/2353673701 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b00a59e0 con 0x7f41980052f0 2024-09-16T05:07:49.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41add9b640 1 --2- 172.21.15.114:0/2353673701 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41b0099530 0x7f41b00999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:07:49.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41ae59c640 1 --2- 172.21.15.114:0/2353673701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41980052f0 0x7f41b013ac70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f41a0011910 tx=0x7f41a0011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 10 client.?.objecter ms_handle_connect 0x7f41980052f0 2024-09-16T05:07:49.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:49.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 1 -- 172.21.15.114:0/2353673701 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a000faf0 con 0x7f41980052f0 2024-09-16T05:07:49.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:49.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 10 monclient: dump: 2024-09-16T05:07:49.949 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.949 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.949 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.949 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.949 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.950 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.950 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.950 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.950 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.950 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 1 -- 172.21.15.114:0/2353673701 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a000fc90 con 0x7f41980052f0 2024-09-16T05:07:49.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 5 monclient: authenticate success, global_id 4296 2024-09-16T05:07:49.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient: _renew_subs 2024-09-16T05:07:49.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2353673701 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b00a55b0 con 0x7f41980052f0 2024-09-16T05:07:49.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 1 -- 172.21.15.114:0/2353673701 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0004020 con 0x7f41980052f0 2024-09-16T05:07:49.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:49.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 10 monclient: dump: 2024-09-16T05:07:49.951 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.951 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.951 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.951 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.951 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.951 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.951 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.951 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.951 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.951 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient: _renew_subs 2024-09-16T05:07:49.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41acd99640 4 set_mon_vals no callback set 2024-09-16T05:07:49.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2353673701 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b01cf2f0 con 0x7f41980052f0 2024-09-16T05:07:49.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 1 librados: init done 2024-09-16T05:07:49.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 1 -- 172.21.15.114:0/2353673701 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f41a0012040 con 0x7f41980052f0 2024-09-16T05:07:49.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2353673701 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b013b880 con 0x7f41980052f0 2024-09-16T05:07:49.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 1 --2- 172.21.15.114:0/2353673701 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4190037810 0x7f4190039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 1 -- 172.21.15.114:0/2353673701 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f41a0048b00 con 0x7f41980052f0 2024-09-16T05:07:49.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 10 client.4296.objecter ms_dispatch 0x7f41b0000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:49.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 3 client.4296.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:49.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 3 client.4296.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:49.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f419effd640 20 client.4296.objecter dump_active .. 0 homeless 2024-09-16T05:07:49.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41add9b640 1 --2- 172.21.15.114:0/2353673701 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4190037810 0x7f4190039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.882+0000 7f41add9b640 10 monclient: get_auth_request con 0x7f4190037810 auth_method 0 2024-09-16T05:07:49.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.886+0000 7f419effd640 10 client.4296.objecter ms_handle_connect 0x7f4190037810 2024-09-16T05:07:49.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.886+0000 7f41add9b640 1 --2- 172.21.15.114:0/2353673701 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4190037810 0x7f4190039cd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f41a40010f0 tx=0x7f41a4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.894+0000 7f419effd640 1 -- 172.21.15.114:0/2353673701 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f41a0019030 con 0x7f41980052f0 2024-09-16T05:07:49.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.894+0000 7f419effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.894+0000 7f419effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:49.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.898+0000 7f7510969640 10 monclient: get_monmap_and_config 2024-09-16T05:07:49.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.898+0000 7f7510969640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.898+0000 7f7510969640 10 monclient: monmap: 2024-09-16T05:07:49.957 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.957 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.958 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:49.904034+0000 2024-09-16T05:07:49.958 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:49.904034+0000 2024-09-16T05:07:49.958 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.958 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.958 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:49.958 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:49.958 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:49.958 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:49.958 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:49.958 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 1 Processor -- start 2024-09-16T05:07:49.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 1 -- start start 2024-09-16T05:07:49.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 10 monclient: init 2024-09-16T05:07:49.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-16T05:07:49.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f750c1376e0 0x7f750c137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 10 monclient(hunting): picked mon.noname-a con 0x7f750c1376e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:49.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f750c1370a0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:49.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f750c0a5d00 0x7f750c0a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 10 monclient(hunting): picked mon.noname-c con 0x7f750c0a5d00 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:49.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f750c09bd20 con 0x7f750c1370a0 2024-09-16T05:07:49.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f750c138000 con 0x7f750c1376e0 2024-09-16T05:07:49.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f750c09c5c0 con 0x7f750c0a5d00 2024-09-16T05:07:49.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f7510969640 10 monclient(hunting): authenticate will time out at 954.533527s 2024-09-16T05:07:49.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f750c1376e0 0x7f750c137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f750c1376e0 0x7f750c137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54926/0 (socket says 172.21.15.114:54926) 2024-09-16T05:07:49.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 1 -- 172.21.15.114:0/675249312 learned_addr learned my addr 172.21.15.114:0/675249312 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 10 monclient(hunting): get_auth_request con 0x7f750c1376e0 auth_method 0 2024-09-16T05:07:49.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2024-09-16T05:07:49.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 1 -- 172.21.15.114:0/675249312 >> v2:172.21.15.149:3300/0 conn(0x7f750c0a5d00 msgr2=0x7f750c0a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:49.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 1 --2- 172.21.15.114:0/675249312 >> v2:172.21.15.149:3300/0 conn(0x7f750c0a5d00 0x7f750c0a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 1 -- 172.21.15.114:0/675249312 >> v1:172.21.15.114:6789/0 conn(0x7f750c1370a0 legacy=0x7f750c0a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:49.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 1 monclient: found mon.noname-a 2024-09-16T05:07:49.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:49.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.902+0000 7f750a59c640 1 -- 172.21.15.114:0/675249312 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750c09c000 con 0x7f750c1376e0 2024-09-16T05:07:49.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f750a59c640 1 --2- 172.21.15.114:0/675249312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f750c1376e0 0x7f750c137ac0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f750c06d8b0 tx=0x7f74fc01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56480eb9d05ecede server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7509d9b640 1 -- 172.21.15.114:0/675249312 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74fc0b0070 con 0x7f750c1376e0 2024-09-16T05:07:49.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7509d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7509d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:49.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7509d9b640 10 monclient: dump: 2024-09-16T05:07:49.961 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.961 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.961 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.962 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.962 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.962 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.962 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.962 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.962 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.962 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7509d9b640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7509d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7509d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 5 monclient: authenticate success, global_id 4299 2024-09-16T05:07:49.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:49.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7509d9b640 1 -- 172.21.15.114:0/675249312 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74fc0ab600 con 0x7f750c1376e0 2024-09-16T05:07:49.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7509d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7509d9b640 1 -- 172.21.15.114:0/675249312 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74fc0ab9d0 con 0x7f750c1376e0 2024-09-16T05:07:49.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:49.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 4 set_mon_vals no callback set 2024-09-16T05:07:49.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7509d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7509d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:49.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7509d9b640 10 monclient: dump: 2024-09-16T05:07:49.963 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.963 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.963 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.963 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.963 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.963 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.963 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.963 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.964 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.964 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 10 monclient: shutdown 2024-09-16T05:07:49.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:49.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 1 -- 172.21.15.114:0/675249312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f750c1376e0 msgr2=0x7f750c137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 1 --2- 172.21.15.114:0/675249312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f750c1376e0 0x7f750c137ac0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f750c06d8b0 tx=0x7f74fc01c3a0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 1 -- 172.21.15.114:0/675249312 shutdown_connections 2024-09-16T05:07:49.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 1 --2- 172.21.15.114:0/675249312 >> v2:172.21.15.149:3300/0 conn(0x7f750c0a5d00 0x7f750c0a4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 1 --2- 172.21.15.114:0/675249312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f750c1376e0 0x7f750c137ac0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 1 -- 172.21.15.114:0/675249312 >> 172.21.15.114:0/675249312 conn(0x7f750c0a9a40 msgr2=0x7f750c0a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:49.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 1 -- 172.21.15.114:0/675249312 shutdown_connections 2024-09-16T05:07:49.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 1 -- 172.21.15.114:0/675249312 wait complete. 2024-09-16T05:07:49.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 10 monclient: build_initial_monmap 2024-09-16T05:07:49.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.906+0000 7f7510969640 10 monclient: monmap: 2024-09-16T05:07:49.965 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:49.965 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:49.965 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:49.965 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:49.965 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:49.965 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.965 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:49.965 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:49.965 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:49.965 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 1 librados: starting msgr at 2024-09-16T05:07:49.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 1 librados: starting objecter 2024-09-16T05:07:49.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 1 Processor -- start 2024-09-16T05:07:49.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 1 -- start start 2024-09-16T05:07:49.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 1 librados: setting wanted keys 2024-09-16T05:07:49.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 1 librados: calling monclient init 2024-09-16T05:07:49.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 monclient: init 2024-09-16T05:07:49.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:49.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:07:49.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f750c0a5d00 0x7f750c1cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 monclient(hunting): picked mon.noname-a con 0x7f750c0a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:49.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f750c1370a0 0x7f750c136450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 monclient(hunting): picked mon.noname-b con 0x7f750c1370a0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:49.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f750c1376e0 0x7f750c13c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 monclient(hunting): picked mon.noname-c con 0x7f750c1376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:49.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f750c141d80 con 0x7f750c0a5d00 2024-09-16T05:07:49.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f750c141c00 con 0x7f750c1370a0 2024-09-16T05:07:49.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:49.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f750c141f00 con 0x7f750c1376e0 2024-09-16T05:07:49.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:49.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 monclient(hunting): authenticate will time out at 954.541356s 2024-09-16T05:07:49.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f750c1376e0 0x7f750c13c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f750c1376e0 0x7f750c13c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55614/0 (socket says 172.21.15.114:55614) 2024-09-16T05:07:49.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 1 -- 172.21.15.114:0/333475471 learned_addr learned my addr 172.21.15.114:0/333475471 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:49.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 10 monclient(hunting): get_auth_request con 0x7f750c1376e0 auth_method 0 2024-09-16T05:07:49.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750a59c640 1 --2- 172.21.15.114:0/333475471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f750c0a5d00 0x7f750c1cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750a59c640 10 monclient(hunting): get_auth_request con 0x7f750c0a5d00 auth_method 0 2024-09-16T05:07:49.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7503fff640 1 --2- 172.21.15.114:0/333475471 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f750c1370a0 0x7f750c136450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7503fff640 10 monclient(hunting): get_auth_request con 0x7f750c1370a0 auth_method 0 2024-09-16T05:07:49.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7503fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:49.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7503fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:49.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7503fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:49.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7503fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:49.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7503fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:49.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7503fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:49.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2024-09-16T05:07:49.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 1 -- 172.21.15.114:0/333475471 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f750c1370a0 msgr2=0x7f750c136450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 1 --2- 172.21.15.114:0/333475471 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f750c1370a0 0x7f750c136450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 1 -- 172.21.15.114:0/333475471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f750c0a5d00 msgr2=0x7f750c1cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:49.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 1 --2- 172.21.15.114:0/333475471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f750c0a5d00 0x7f750c1cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:49.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 10 monclient: _finish_hunting 0 2024-09-16T05:07:49.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 1 monclient: found mon.noname-c 2024-09-16T05:07:49.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:49.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 1 -- 172.21.15.114:0/333475471 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750c09d5d0 con 0x7f750c1376e0 2024-09-16T05:07:49.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750ad9d640 1 --2- 172.21.15.114:0/333475471 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f750c1376e0 0x7f750c13c360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f750400b9a0 tx=0x7f750400be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 10 client.?.objecter ms_handle_connect 0x7f750c1376e0 2024-09-16T05:07:49.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:49.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 1 -- 172.21.15.114:0/333475471 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7504019070 con 0x7f750c1376e0 2024-09-16T05:07:49.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:49.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 10 monclient: dump: 2024-09-16T05:07:49.970 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.970 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.970 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.970 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.970 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.970 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.970 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.971 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.971 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.971 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 10 monclient: _finish_auth 0 2024-09-16T05:07:49.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:07:49.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:49.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 1 -- 172.21.15.114:0/333475471 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f750400cbc0 con 0x7f750c1376e0 2024-09-16T05:07:49.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 5 monclient: authenticate success, global_id 4247 2024-09-16T05:07:49.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 monclient: _renew_subs 2024-09-16T05:07:49.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 1 -- 172.21.15.114:0/333475471 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f750c1d02b0 con 0x7f750c1376e0 2024-09-16T05:07:49.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:49.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 1 -- 172.21.15.114:0/333475471 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7504014440 con 0x7f750c1376e0 2024-09-16T05:07:49.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:49.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:49.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 10 monclient: dump: 2024-09-16T05:07:49.972 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:49.972 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:49.972 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.972 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:49.972 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:49.972 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:49.972 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:49.972 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:49.972 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:49.972 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:49.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:49.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 monclient: _renew_subs 2024-09-16T05:07:49.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 1 -- 172.21.15.114:0/333475471 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f750c0a8fa0 con 0x7f750c1376e0 2024-09-16T05:07:49.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f750959a640 4 set_mon_vals no callback set 2024-09-16T05:07:49.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7510969640 1 librados: init done 2024-09-16T05:07:49.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.910+0000 7f7502ffd640 1 -- 172.21.15.114:0/333475471 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f750401c070 con 0x7f750c1376e0 2024-09-16T05:07:49.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.914+0000 7f7502ffd640 1 --2- 172.21.15.114:0/333475471 >> v2:172.21.15.149:6812/1185531135 conn(0x7f74ec037810 0x7f74ec039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:49.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.914+0000 7f7500ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.914+0000 7f7500ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.914+0000 7f7500ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:49.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.914+0000 7f7502ffd640 1 -- 172.21.15.114:0/333475471 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f7504019de0 con 0x7f750c1376e0 2024-09-16T05:07:49.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.914+0000 7f7502ffd640 10 client.4247.objecter ms_dispatch 0x7f750c000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:49.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.914+0000 7f7502ffd640 3 client.4247.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:49.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.914+0000 7f7502ffd640 3 client.4247.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:49.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.914+0000 7f7500ff9640 1 -- 172.21.15.114:0/333475471 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74e4005090 con 0x7f750c1376e0 2024-09-16T05:07:49.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.914+0000 7f7502ffd640 20 client.4247.objecter dump_active .. 0 homeless 2024-09-16T05:07:49.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.914+0000 7f750a59c640 1 --2- 172.21.15.114:0/333475471 >> v2:172.21.15.149:6812/1185531135 conn(0x7f74ec037810 0x7f74ec039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:49.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.914+0000 7f750a59c640 10 monclient: get_auth_request con 0x7f74ec037810 auth_method 0 2024-09-16T05:07:49.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.914+0000 7f750a59c640 1 --2- 172.21.15.114:0/333475471 >> v2:172.21.15.149:6812/1185531135 conn(0x7f74ec037810 0x7f74ec039cd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0ad000 tx=0x7f74fc0a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:49.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.918+0000 7f7502ffd640 10 client.4247.objecter ms_handle_connect 0x7f74ec037810 2024-09-16T05:07:49.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.918+0000 7f7502ffd640 1 -- 172.21.15.114:0/333475471 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f750c1d02b0 con 0x7f750c1376e0 2024-09-16T05:07:49.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.918+0000 7f7502ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:49.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:49.918+0000 7f7502ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.142+0000 7f9e7f4f7640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T05:07:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.142+0000 7f9e7f4f7640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T05:07:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.142+0000 7f9e7f4f7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.142+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/55919518 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f9e780a7460 con 0x7f9e781370a0 2024-09-16T05:07:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.146+0000 7f9e6dffb640 1 -- 172.21.15.114:0/55919518 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9e68011e10 con 0x7f9e781370a0 2024-09-16T05:07:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.146+0000 7f9e6dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T05:07:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.146+0000 7f9e6dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:50.145 INFO:teuthology.orchestra.run.smithi114.stdout:51539607555 2024-09-16T05:07:50.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.146+0000 7f9e7f4f7640 10 librados: watch_flush enter 2024-09-16T05:07:50.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.146+0000 7f9e7f4f7640 10 librados: watch_flush exit 2024-09-16T05:07:50.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.146+0000 7f9e7f4f7640 20 client.4232.objecter shutdown clearing up homeless session... 2024-09-16T05:07:50.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.146+0000 7f9e7f4f7640 10 client.4232.objecter successfully canceled tick 2024-09-16T05:07:50.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.146+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/55919518 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9e5c037810 msgr2=0x7f9e5c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:50.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.146+0000 7f9e7f4f7640 1 --2- 172.21.15.114:0/55919518 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9e5c037810 0x7f9e5c039cd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9e74000c10 tx=0x7f9e74008040 comp rx=0 tx=0).stop 2024-09-16T05:07:50.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7f9e7f4f7640 10 monclient: shutdown 2024-09-16T05:07:50.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7f9e7f4f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:50.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/55919518 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e781370a0 msgr2=0x7f9e78135ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:50.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7f9e7f4f7640 1 --2- 172.21.15.114:0/55919518 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e781370a0 0x7f9e78135ce0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800c820 tx=0x7f9e6800cce0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7fab44c24640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T05:07:50.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7fab44c24640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T05:07:50.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7fab44c24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:50.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7fab44c24640 1 -- 172.21.15.114:0/2483134955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fab4013ee70 con 0x7fab401370a0 2024-09-16T05:07:50.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/55919518 shutdown_connections 2024-09-16T05:07:50.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7fab36ffd640 1 -- 172.21.15.114:0/2483134955 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fab28046b20 con 0x7fab401370a0 2024-09-16T05:07:50.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7fab36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T05:07:50.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7fab36ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7f9e7f4f7640 1 --2- 172.21.15.114:0/55919518 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9e5c037810 0x7f9e5c039cd0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7f9e7f4f7640 1 --2- 172.21.15.114:0/55919518 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e781376e0 0x7f9e7813bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7f9e7f4f7640 1 --2- 172.21.15.114:0/55919518 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e781370a0 0x7f9e78135ce0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7f9e7f4f7640 1 --2- 172.21.15.114:0/55919518 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e780a5d00 0x7f9e781357a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/55919518 >> 172.21.15.114:0/55919518 conn(0x7f9e780a9a40 msgr2=0x7f9e7809f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:50.153 INFO:teuthology.orchestra.run.smithi114.stdout:55834574851 2024-09-16T05:07:50.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/55919518 shutdown_connections 2024-09-16T05:07:50.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7f9e7f4f7640 1 -- 172.21.15.114:0/55919518 wait complete. 2024-09-16T05:07:50.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7f9e7f4f7640 1 librados: shutdown 2024-09-16T05:07:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7fab44c24640 10 librados: watch_flush enter 2024-09-16T05:07:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.150+0000 7fab44c24640 10 librados: watch_flush exit 2024-09-16T05:07:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 20 client.4281.objecter shutdown clearing up homeless session... 2024-09-16T05:07:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 10 client.4281.objecter successfully canceled tick 2024-09-16T05:07:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 1 -- 172.21.15.114:0/2483134955 >> v2:172.21.15.149:6812/1185531135 conn(0x7fab20037810 msgr2=0x7fab20039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 1 --2- 172.21.15.114:0/2483134955 >> v2:172.21.15.149:6812/1185531135 conn(0x7fab20037810 0x7fab20039cd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fab30009ba0 tx=0x7fab30008040 comp rx=0 tx=0).stop 2024-09-16T05:07:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 10 monclient: shutdown 2024-09-16T05:07:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 1 -- 172.21.15.114:0/2483134955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab401370a0 msgr2=0x7fab40138b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 1 --2- 172.21.15.114:0/2483134955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab401370a0 0x7fab40138b90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fab2800c490 tx=0x7fab2800c950 comp rx=0 tx=0).stop 2024-09-16T05:07:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 1 -- 172.21.15.114:0/2483134955 shutdown_connections 2024-09-16T05:07:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 1 --2- 172.21.15.114:0/2483134955 >> v2:172.21.15.149:6812/1185531135 conn(0x7fab20037810 0x7fab20039cd0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 1 --2- 172.21.15.114:0/2483134955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab401370a0 0x7fab40138b90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 1 --2- 172.21.15.114:0/2483134955 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fab401376e0 0x7fab401d3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 1 --2- 172.21.15.114:0/2483134955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab400a5d00 0x7fab40138650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 1 -- 172.21.15.114:0/2483134955 >> 172.21.15.114:0/2483134955 conn(0x7fab400a9a40 msgr2=0x7fab4006fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:50.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 1 -- 172.21.15.114:0/2483134955 shutdown_connections 2024-09-16T05:07:50.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 1 -- 172.21.15.114:0/2483134955 wait complete. 2024-09-16T05:07:50.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7fab44c24640 1 librados: shutdown 2024-09-16T05:07:50.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7f4443668640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T05:07:50.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7f4443668640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T05:07:50.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7f4443668640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:50.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7f4443668640 1 -- 172.21.15.114:0/2159788967 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f443c136e80 con 0x7f443c1370a0 2024-09-16T05:07:50.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7f4431ffb640 1 -- 172.21.15.114:0/2159788967 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f442c005480 con 0x7f443c1370a0 2024-09-16T05:07:50.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7f4431ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T05:07:50.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.154+0000 7f4431ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:50.157 INFO:teuthology.orchestra.run.smithi114.stdout:47244640259 2024-09-16T05:07:50.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 10 librados: watch_flush enter 2024-09-16T05:07:50.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 10 librados: watch_flush exit 2024-09-16T05:07:50.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 20 client.4241.objecter shutdown clearing up homeless session... 2024-09-16T05:07:50.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 10 client.4241.objecter successfully canceled tick 2024-09-16T05:07:50.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 1 -- 172.21.15.114:0/2159788967 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4420037810 msgr2=0x7f4420039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:50.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 1 --2- 172.21.15.114:0/2159788967 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4420037810 0x7f4420039cd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4438009990 tx=0x7f4438009110 comp rx=0 tx=0).stop 2024-09-16T05:07:50.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 10 monclient: shutdown 2024-09-16T05:07:50.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:50.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 1 -- 172.21.15.114:0/2159788967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f443c1370a0 msgr2=0x7f443c1362a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:50.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 1 --2- 172.21.15.114:0/2159788967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f443c1370a0 0x7f443c1362a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f442c00c430 tx=0x7f442c00c8f0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 1 -- 172.21.15.114:0/2159788967 shutdown_connections 2024-09-16T05:07:50.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 1 --2- 172.21.15.114:0/2159788967 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4420037810 0x7f4420039cd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 1 --2- 172.21.15.114:0/2159788967 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f443c1376e0 0x7f443c13c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 1 --2- 172.21.15.114:0/2159788967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f443c1370a0 0x7f443c1362a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 1 --2- 172.21.15.114:0/2159788967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f443c0a5d00 0x7f443c1cf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 1 -- 172.21.15.114:0/2159788967 >> 172.21.15.114:0/2159788967 conn(0x7f443c0a9a40 msgr2=0x7f443c0a7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:50.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 1 -- 172.21.15.114:0/2159788967 shutdown_connections 2024-09-16T05:07:50.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 1 -- 172.21.15.114:0/2159788967 wait complete. 2024-09-16T05:07:50.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.158+0000 7f4443668640 1 librados: shutdown 2024-09-16T05:07:50.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.174+0000 7fd4b0a0c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T05:07:50.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.174+0000 7fd4b0a0c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T05:07:50.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.174+0000 7fd4b0a0c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:50.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.174+0000 7fd4b0a0c640 1 -- 172.21.15.114:0/3555570503 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd4a40244a0 con 0x7fd4a40c0990 2024-09-16T05:07:50.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.174+0000 7fd4937fe640 1 -- 172.21.15.114:0/3555570503 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd4ac09de10 con 0x7fd4a40c0990 2024-09-16T05:07:50.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.174+0000 7fd4937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T05:07:50.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.174+0000 7fd4937fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:50.177 INFO:teuthology.orchestra.run.smithi114.stdout:47244640259 2024-09-16T05:07:50.178 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.5 2024-09-16T05:07:50.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 10 librados: watch_flush enter 2024-09-16T05:07:50.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 10 librados: watch_flush exit 2024-09-16T05:07:50.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 20 client.4290.objecter shutdown clearing up homeless session... 2024-09-16T05:07:50.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 10 client.4290.objecter successfully canceled tick 2024-09-16T05:07:50.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 1 -- 172.21.15.114:0/3555570503 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd498037810 msgr2=0x7fd498039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:50.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 1 --2- 172.21.15.114:0/3555570503 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd498037810 0x7fd498039cd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd49c000f70 tx=0x7fd49c0079e0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 10 monclient: shutdown 2024-09-16T05:07:50.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:50.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 1 -- 172.21.15.114:0/3555570503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4a40c0990 msgr2=0x7fd4a401e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:50.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 1 --2- 172.21.15.114:0/3555570503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4a40c0990 0x7fd4a401e8f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac075580 tx=0x7fd4ac069950 comp rx=0 tx=0).stop 2024-09-16T05:07:50.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 1 -- 172.21.15.114:0/3555570503 shutdown_connections 2024-09-16T05:07:50.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 1 --2- 172.21.15.114:0/3555570503 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd498037810 0x7fd498039cd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 1 --2- 172.21.15.114:0/3555570503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4a40c0990 0x7fd4a401e8f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 1 --2- 172.21.15.114:0/3555570503 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4a4024090 0x7fd4a401e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 1 --2- 172.21.15.114:0/3555570503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4a4023cb0 0x7fd4a40bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 1 -- 172.21.15.114:0/3555570503 >> 172.21.15.114:0/3555570503 conn(0x7fd4a4006320 msgr2=0x7fd4a40c2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:50.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 1 -- 172.21.15.114:0/3555570503 shutdown_connections 2024-09-16T05:07:50.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 1 -- 172.21.15.114:0/3555570503 wait complete. 2024-09-16T05:07:50.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.178+0000 7fd4b0a0c640 1 librados: shutdown 2024-09-16T05:07:50.182 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.0 2024-09-16T05:07:50.187 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640259 for osd.3 2024-09-16T05:07:50.196 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640259 for osd.4 2024-09-16T05:07:50.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T05:07:50.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T05:07:50.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:50.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2353673701 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f41b0074180 con 0x7f41980052f0 2024-09-16T05:07:50.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f419effd640 1 -- 172.21.15.114:0/2353673701 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f41a00133a0 con 0x7f41980052f0 2024-09-16T05:07:50.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f419effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T05:07:50.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f419effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:50.201 INFO:teuthology.orchestra.run.smithi114.stdout:55834574851 2024-09-16T05:07:50.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 10 librados: watch_flush enter 2024-09-16T05:07:50.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 10 librados: watch_flush exit 2024-09-16T05:07:50.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 20 client.4296.objecter shutdown clearing up homeless session... 2024-09-16T05:07:50.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 10 client.4296.objecter successfully canceled tick 2024-09-16T05:07:50.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2353673701 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4190037810 msgr2=0x7f4190039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:50.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 1 --2- 172.21.15.114:0/2353673701 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4190037810 0x7f4190039cd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f41a40010f0 tx=0x7f41a4009040 comp rx=0 tx=0).stop 2024-09-16T05:07:50.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 10 monclient: shutdown 2024-09-16T05:07:50.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:50.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2353673701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41980052f0 msgr2=0x7f41b013ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:50.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 1 --2- 172.21.15.114:0/2353673701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41980052f0 0x7f41b013ac70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f41a0011910 tx=0x7f41a0011dd0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2353673701 shutdown_connections 2024-09-16T05:07:50.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 1 --2- 172.21.15.114:0/2353673701 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4190037810 0x7f4190039cd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 1 --2- 172.21.15.114:0/2353673701 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41b008d000 0x7f41b008d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 1 --2- 172.21.15.114:0/2353673701 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41b0099530 0x7f41b00999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 1 --2- 172.21.15.114:0/2353673701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41980052f0 0x7f41b013ac70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2353673701 >> 172.21.15.114:0/2353673701 conn(0x7f41b00a3270 msgr2=0x7f41b00a43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:50.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2353673701 shutdown_connections 2024-09-16T05:07:50.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 1 -- 172.21.15.114:0/2353673701 wait complete. 2024-09-16T05:07:50.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.202+0000 7f41b4f1e640 1 librados: shutdown 2024-09-16T05:07:50.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T05:07:50.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T05:07:50.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:50.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 1 -- 172.21.15.114:0/333475471 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f74e4005650 con 0x7f750c1376e0 2024-09-16T05:07:50.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7502ffd640 1 -- 172.21.15.114:0/333475471 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7504019ae0 con 0x7f750c1376e0 2024-09-16T05:07:50.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7502ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T05:07:50.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7502ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:50.210 INFO:teuthology.orchestra.run.smithi114.stdout:51539607555 2024-09-16T05:07:50.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 10 librados: watch_flush enter 2024-09-16T05:07:50.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 10 librados: watch_flush exit 2024-09-16T05:07:50.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 20 client.4247.objecter shutdown clearing up homeless session... 2024-09-16T05:07:50.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 10 client.4247.objecter successfully canceled tick 2024-09-16T05:07:50.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 1 -- 172.21.15.114:0/333475471 >> v2:172.21.15.149:6812/1185531135 conn(0x7f74ec037810 msgr2=0x7f74ec039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:50.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 1 --2- 172.21.15.114:0/333475471 >> v2:172.21.15.149:6812/1185531135 conn(0x7f74ec037810 0x7f74ec039cd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0ad000 tx=0x7f74fc0a6000 comp rx=0 tx=0).stop 2024-09-16T05:07:50.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 10 monclient: shutdown 2024-09-16T05:07:50.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:50.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 1 -- 172.21.15.114:0/333475471 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f750c1376e0 msgr2=0x7f750c13c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:50.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 1 --2- 172.21.15.114:0/333475471 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f750c1376e0 0x7f750c13c360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f750400b9a0 tx=0x7f750400be60 comp rx=0 tx=0).stop 2024-09-16T05:07:50.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 1 -- 172.21.15.114:0/333475471 shutdown_connections 2024-09-16T05:07:50.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 1 --2- 172.21.15.114:0/333475471 >> v2:172.21.15.149:6812/1185531135 conn(0x7f74ec037810 0x7f74ec039cd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 1 --2- 172.21.15.114:0/333475471 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f750c1376e0 0x7f750c13c360 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 1 --2- 172.21.15.114:0/333475471 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f750c1370a0 0x7f750c136450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 1 --2- 172.21.15.114:0/333475471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f750c0a5d00 0x7f750c1cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:50.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 1 -- 172.21.15.114:0/333475471 >> 172.21.15.114:0/333475471 conn(0x7f750c0a9a40 msgr2=0x7f750c09fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:50.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 1 -- 172.21.15.114:0/333475471 shutdown_connections 2024-09-16T05:07:50.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 1 -- 172.21.15.114:0/333475471 wait complete. 2024-09-16T05:07:50.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:50.210+0000 7f7500ff9640 1 librados: shutdown 2024-09-16T05:07:50.219 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.2 2024-09-16T05:07:50.228 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.1 2024-09-16T05:07:51.179 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T05:07:51.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T05:07:51.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T05:07:51.197 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T05:07:51.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T05:07:51.228 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T05:07:51.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 10 monclient: get_monmap_and_config 2024-09-16T05:07:51.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 10 monclient: build_initial_monmap 2024-09-16T05:07:51.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 10 monclient: monmap: 2024-09-16T05:07:51.320 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:51.320 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.320 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:51.322875+0000 2024-09-16T05:07:51.320 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:51.322875+0000 2024-09-16T05:07:51.320 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:51.320 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.320 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:51.320 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:51.320 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:51.320 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:51.320 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:51.320 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 1 Processor -- start 2024-09-16T05:07:51.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 1 -- start start 2024-09-16T05:07:51.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 10 monclient: init 2024-09-16T05:07:51.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:51.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-16T05:07:51.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f181376e0 0x7f9f18137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 10 monclient(hunting): picked mon.noname-b con 0x7f9f181376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:51.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9f181370a0 0x7f9f180a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 10 monclient(hunting): picked mon.noname-a con 0x7f9f181370a0 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:51.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9f180a5d00 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:51.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f1809bd20 con 0x7f9f180a5d00 2024-09-16T05:07:51.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f9f18138000 con 0x7f9f181370a0 2024-09-16T05:07:51.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9f1809c5c0 con 0x7f9f181376e0 2024-09-16T05:07:51.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:51.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.318+0000 7f9f1f464640 10 monclient(hunting): authenticate will time out at 955.951866s 2024-09-16T05:07:51.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1d200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f181376e0 0x7f9f18137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9f181370a0 0x7f9f180a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1d200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f181376e0 0x7f9f18137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55618/0 (socket says 172.21.15.114:55618) 2024-09-16T05:07:51.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1d200640 1 -- 172.21.15.114:0/2148361294 learned_addr learned my addr 172.21.15.114:0/2148361294 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:51.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1d200640 10 monclient(hunting): get_auth_request con 0x7f9f181376e0 auth_method 0 2024-09-16T05:07:51.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1d200640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1d200640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1d200640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 10 monclient(hunting): get_auth_request con 0x7f9f181370a0 auth_method 0 2024-09-16T05:07:51.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1d200640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1d200640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1d200640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2024-09-16T05:07:51.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 1 -- 172.21.15.114:0/2148361294 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f181376e0 msgr2=0x7f9f18137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:51.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 1 --2- 172.21.15.114:0/2148361294 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f181376e0 0x7f9f18137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 1 -- 172.21.15.114:0/2148361294 >> v1:172.21.15.149:6789/0 conn(0x7f9f180a5d00 legacy=0x7f9f180a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 10 monclient: _finish_hunting 0 2024-09-16T05:07:51.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 1 monclient: found mon.noname-a 2024-09-16T05:07:51.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:51.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 1 -- 172.21.15.114:0/2148361294 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f1809c000 con 0x7f9f181370a0 2024-09-16T05:07:51.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1d200640 1 --2- 172.21.15.114:0/2148361294 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f181376e0 0x7f9f18137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:51.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1c9ff640 1 --2- 172.21.15.114:0/2148361294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f181370a0 0x7f9f180a4270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9f080097c0 tx=0x7f9f0801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5ecfbcf7b78b73 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f0ffff640 1 -- 172.21.15.114:0/2148361294 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f080b0070 con 0x7f9f181370a0 2024-09-16T05:07:51.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f0ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:51.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f0ffff640 10 monclient: dump: 2024-09-16T05:07:51.333 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.333 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.333 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.333 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.333 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.334 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.334 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.334 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.334 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.334 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f0ffff640 10 monclient: _finish_auth 0 2024-09-16T05:07:51.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f0ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:07:51.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:51.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f0ffff640 1 -- 172.21.15.114:0/2148361294 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f080ab600 con 0x7f9f181370a0 2024-09-16T05:07:51.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1f464640 5 monclient: authenticate success, global_id 4305 2024-09-16T05:07:51.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1f464640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:51.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f0ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:51.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1f464640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:51.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1f464640 4 set_mon_vals no callback set 2024-09-16T05:07:51.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f0ffff640 1 -- 172.21.15.114:0/2148361294 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f080ab9d0 con 0x7f9f181370a0 2024-09-16T05:07:51.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f0ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:51.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f0ffff640 10 monclient: dump: 2024-09-16T05:07:51.335 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.335 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.335 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.335 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.335 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.335 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.336 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.336 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.336 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.336 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1f464640 10 monclient: shutdown 2024-09-16T05:07:51.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1f464640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:51.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1f464640 1 -- 172.21.15.114:0/2148361294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f181370a0 msgr2=0x7f9f180a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1f464640 1 --2- 172.21.15.114:0/2148361294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f181370a0 0x7f9f180a4270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9f080097c0 tx=0x7f9f0801c360 comp rx=0 tx=0).stop 2024-09-16T05:07:51.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1f464640 1 -- 172.21.15.114:0/2148361294 shutdown_connections 2024-09-16T05:07:51.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1f464640 1 --2- 172.21.15.114:0/2148361294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f181370a0 0x7f9f180a4270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1f464640 1 --2- 172.21.15.114:0/2148361294 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f181376e0 0x7f9f18137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.322+0000 7f9f1f464640 1 -- 172.21.15.114:0/2148361294 >> 172.21.15.114:0/2148361294 conn(0x7f9f180a9a40 msgr2=0x7f9f180a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:51.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 -- 172.21.15.114:0/2148361294 shutdown_connections 2024-09-16T05:07:51.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 -- 172.21.15.114:0/2148361294 wait complete. 2024-09-16T05:07:51.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient: build_initial_monmap 2024-09-16T05:07:51.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient: monmap: 2024-09-16T05:07:51.337 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:51.337 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:51.337 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:51.337 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:51.337 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:51.337 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.337 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:51.337 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:51.337 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:51.337 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 librados: starting msgr at 2024-09-16T05:07:51.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 librados: starting objecter 2024-09-16T05:07:51.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 Processor -- start 2024-09-16T05:07:51.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 -- start start 2024-09-16T05:07:51.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 librados: setting wanted keys 2024-09-16T05:07:51.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 librados: calling monclient init 2024-09-16T05:07:51.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient: init 2024-09-16T05:07:51.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:51.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:07:51.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9f180a5d00 0x7f9f181cf110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient(hunting): picked mon.noname-b con 0x7f9f180a5d00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:51.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f181370a0 0x7f9f18136090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient(hunting): picked mon.noname-c con 0x7f9f181370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:51.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f181376e0 0x7f9f1813bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient(hunting): picked mon.noname-a con 0x7f9f181376e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:51.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9f18141d80 con 0x7f9f181376e0 2024-09-16T05:07:51.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f9f18141c00 con 0x7f9f180a5d00 2024-09-16T05:07:51.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9f18141f00 con 0x7f9f181370a0 2024-09-16T05:07:51.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:51.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient(hunting): authenticate will time out at 955.957704s 2024-09-16T05:07:51.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1c9ff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f181370a0 0x7f9f18136090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1c9ff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f181370a0 0x7f9f18136090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55622/0 (socket says 172.21.15.114:55622) 2024-09-16T05:07:51.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1c9ff640 1 -- 172.21.15.114:0/880744628 learned_addr learned my addr 172.21.15.114:0/880744628 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:51.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1c9ff640 10 monclient(hunting): get_auth_request con 0x7f9f181370a0 auth_method 0 2024-09-16T05:07:51.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1c9ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1c9ff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1c9ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 1 --2- 172.21.15.114:0/880744628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f181376e0 0x7f9f1813bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 10 monclient(hunting): get_auth_request con 0x7f9f181376e0 auth_method 0 2024-09-16T05:07:51.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1d200640 1 --2- 172.21.15.114:0/880744628 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9f180a5d00 0x7f9f181cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 10 monclient(hunting): handle_auth_done global_id 4308 payload 824 2024-09-16T05:07:51.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 1 -- 172.21.15.114:0/880744628 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f181370a0 msgr2=0x7f9f18136090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 1 --2- 172.21.15.114:0/880744628 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f181370a0 0x7f9f18136090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 1 -- 172.21.15.114:0/880744628 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9f180a5d00 msgr2=0x7f9f181cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 1 --2- 172.21.15.114:0/880744628 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9f180a5d00 0x7f9f181cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 10 monclient: _finish_hunting 0 2024-09-16T05:07:51.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 1 monclient: found mon.noname-a 2024-09-16T05:07:51.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:51.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 1 -- 172.21.15.114:0/880744628 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f180a2560 con 0x7f9f181376e0 2024-09-16T05:07:51.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1d200640 10 monclient: get_auth_request con 0x7f9f180a5d00 auth_method 0 2024-09-16T05:07:51.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1d200640 1 --2- 172.21.15.114:0/880744628 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9f180a5d00 0x7f9f181cf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:07:51.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1da01640 1 --2- 172.21.15.114:0/880744628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f181376e0 0x7f9f1813bfa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000c730 tx=0x7f9f1000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0dffb640 10 client.?.objecter ms_handle_connect 0x7f9f181376e0 2024-09-16T05:07:51.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:51.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0dffb640 1 -- 172.21.15.114:0/880744628 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f10016020 con 0x7f9f181376e0 2024-09-16T05:07:51.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:51.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0dffb640 10 monclient: dump: 2024-09-16T05:07:51.342 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.342 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.342 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.342 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.342 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.342 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.342 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.343 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.343 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.343 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0dffb640 10 monclient: _finish_auth 0 2024-09-16T05:07:51.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:07:51.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:51.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 5 monclient: authenticate success, global_id 4308 2024-09-16T05:07:51.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0dffb640 1 -- 172.21.15.114:0/880744628 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f10006d60 con 0x7f9f181376e0 2024-09-16T05:07:51.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:51.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0dffb640 1 -- 172.21.15.114:0/880744628 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f10003bc0 con 0x7f9f181376e0 2024-09-16T05:07:51.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:51.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0dffb640 10 monclient: dump: 2024-09-16T05:07:51.344 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.344 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.344 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.344 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.344 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.344 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.344 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.344 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.344 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.344 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f0f7fe640 4 set_mon_vals no callback set 2024-09-16T05:07:51.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient: _renew_subs 2024-09-16T05:07:51.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 -- 172.21.15.114:0/880744628 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f18120740 con 0x7f9f181376e0 2024-09-16T05:07:51.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:51.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient: _renew_subs 2024-09-16T05:07:51.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 -- 172.21.15.114:0/880744628 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f18120180 con 0x7f9f181376e0 2024-09-16T05:07:51.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.326+0000 7f9f1f464640 1 librados: init done 2024-09-16T05:07:51.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.330+0000 7f9f1f464640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.330+0000 7f9f1f464640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.330+0000 7f9f1f464640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.330+0000 7f9f1f464640 1 -- 172.21.15.114:0/880744628 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f18137ac0 con 0x7f9f181376e0 2024-09-16T05:07:51.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.330+0000 7f9f0dffb640 1 -- 172.21.15.114:0/880744628 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9f100050d0 con 0x7f9f181376e0 2024-09-16T05:07:51.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.330+0000 7f9f0dffb640 1 --2- 172.21.15.114:0/880744628 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9efc037810 0x7f9efc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.330+0000 7f9f1d200640 1 --2- 172.21.15.114:0/880744628 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9efc037810 0x7f9efc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.330+0000 7f9f1d200640 10 monclient: get_auth_request con 0x7f9efc037810 auth_method 0 2024-09-16T05:07:51.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.330+0000 7f9f0dffb640 10 client.4308.objecter ms_handle_connect 0x7f9efc037810 2024-09-16T05:07:51.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.330+0000 7f9f1d200640 1 --2- 172.21.15.114:0/880744628 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9efc037810 0x7f9efc039cd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9f1813d7b0 tx=0x7f9f14008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.334+0000 7f9f0dffb640 1 -- 172.21.15.114:0/880744628 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f9f10046c20 con 0x7f9f181376e0 2024-09-16T05:07:51.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.334+0000 7f9f0dffb640 10 client.4308.objecter ms_dispatch 0x7f9f18000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:51.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.334+0000 7f9f0dffb640 3 client.4308.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:51.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.334+0000 7f9f0dffb640 3 client.4308.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:51.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.334+0000 7f9f0dffb640 20 client.4308.objecter dump_active .. 0 homeless 2024-09-16T05:07:51.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.334+0000 7f9f0dffb640 1 -- 172.21.15.114:0/880744628 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f9f180a2560 con 0x7f9f181376e0 2024-09-16T05:07:51.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.334+0000 7f9f0dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.334+0000 7f9f0dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:51.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.354+0000 7f4981877640 10 monclient: get_monmap_and_config 2024-09-16T05:07:51.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.354+0000 7f4981877640 10 monclient: build_initial_monmap 2024-09-16T05:07:51.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.354+0000 7f4981877640 10 monclient: monmap: 2024-09-16T05:07:51.355 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:51.355 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.356 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:51.358436+0000 2024-09-16T05:07:51.356 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:51.358436+0000 2024-09-16T05:07:51.356 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:51.356 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.356 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:51.356 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:51.356 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:51.356 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:51.356 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:51.356 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.354+0000 7f4981877640 1 Processor -- start 2024-09-16T05:07:51.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.354+0000 7f4981877640 1 -- start start 2024-09-16T05:07:51.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.354+0000 7f4981877640 10 monclient: init 2024-09-16T05:07:51.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.354+0000 7f4981877640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:51.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.354+0000 7f4981877640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-16T05:07:51.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.354+0000 7f4981877640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f497c1376e0 0x7f497c137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.354+0000 7f4981877640 10 monclient(hunting): picked mon.noname-b con 0x7f497c1376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:51.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497affd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f497c1376e0 0x7f497c137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497affd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f497c1376e0 0x7f497c137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55632/0 (socket says 172.21.15.114:55632) 2024-09-16T05:07:51.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497a7fc640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f497c1370a0 0x7f497c0a4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38648/0 (socket says 172.21.15.114:38648) 2024-09-16T05:07:51.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4981877640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f497c1370a0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:51.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4981877640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f497c0a5d00 0x7f497c0a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497a7fc640 1 -- 172.21.15.114:0/2979036984 learned_addr learned my addr 172.21.15.114:0/2979036984 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:51.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4981877640 10 monclient(hunting): picked mon.noname-c con 0x7f497c0a5d00 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:51.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4981877640 1 -- 172.21.15.114:0/2979036984 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f497c09bd20 con 0x7f497c1370a0 2024-09-16T05:07:51.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 1 --2- 172.21.15.114:0/2979036984 >> v2:172.21.15.149:3300/0 conn(0x7f497c0a5d00 0x7f497c0a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4981877640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4981877640 1 -- 172.21.15.114:0/2979036984 --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f497c138000 con 0x7f497c0a5d00 2024-09-16T05:07:51.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4981877640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4979ffb640 1 -- 172.21.15.114:0/2979036984 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 147146077 0 0) 0x7f497c09bd20 con 0x7f497c1370a0 2024-09-16T05:07:51.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4981877640 1 -- 172.21.15.114:0/2979036984 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f497c09c5c0 con 0x7f497c1376e0 2024-09-16T05:07:51.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4981877640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:51.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4981877640 10 monclient(hunting): authenticate will time out at 955.988765s 2024-09-16T05:07:51.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 10 monclient(hunting): get_auth_request con 0x7f497c0a5d00 auth_method 0 2024-09-16T05:07:51.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497affd640 10 monclient(hunting): get_auth_request con 0x7f497c1376e0 auth_method 0 2024-09-16T05:07:51.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4979ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:07:51.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4979ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4979ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4979ffb640 10 monclient(hunting): my global_id is 4311 2024-09-16T05:07:51.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4979ffb640 1 -- 172.21.15.114:0/2979036984 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4964003530 con 0x7f497c1370a0 2024-09-16T05:07:51.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 10 monclient(hunting): handle_auth_done global_id 4321 payload 274 2024-09-16T05:07:51.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 1 -- 172.21.15.114:0/2979036984 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f497c1376e0 msgr2=0x7f497c137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:51.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 1 --2- 172.21.15.114:0/2979036984 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f497c1376e0 0x7f497c137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 1 -- 172.21.15.114:0/2979036984 >> v1:172.21.15.114:6789/0 conn(0x7f497c1370a0 legacy=0x7f497c0a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:07:51.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 1 monclient: found mon.noname-c 2024-09-16T05:07:51.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:51.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:07:51.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f497b7fe640 1 -- 172.21.15.114:0/2979036984 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497c09c000 con 0x7f497c0a5d00 2024-09-16T05:07:51.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4979ffb640 1 -- 172.21.15.114:0/2979036984 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3236854218 0 0) 0x7f4964003530 con 0x7f497c1370a0 2024-09-16T05:07:51.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.358+0000 7f4979ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:07:51.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f497b7fe640 1 --2- 172.21.15.114:0/2979036984 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f497c0a5d00 0x7f497c0a4980 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4974009d90 tx=0x7f497401c750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39bdd233d3701d8f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4979ffb640 1 -- 172.21.15.114:0/2979036984 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49740ac070 con 0x7f497c0a5d00 2024-09-16T05:07:51.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4979ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4979ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:51.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4979ffb640 10 monclient: dump: 2024-09-16T05:07:51.368 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.368 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.368 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.368 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.368 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.368 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.369 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.369 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.369 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.369 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4979ffb640 10 monclient: _finish_auth 0 2024-09-16T05:07:51.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4979ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:07:51.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4979ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:51.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4979ffb640 1 -- 172.21.15.114:0/2979036984 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49740a6470 con 0x7f497c0a5d00 2024-09-16T05:07:51.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 5 monclient: authenticate success, global_id 4321 2024-09-16T05:07:51.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:51.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4979ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:51.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:51.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4979ffb640 1 -- 172.21.15.114:0/2979036984 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970001680 con 0x7f497c0a5d00 2024-09-16T05:07:51.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 4 set_mon_vals no callback set 2024-09-16T05:07:51.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4979ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4979ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:51.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4979ffb640 10 monclient: dump: 2024-09-16T05:07:51.371 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.371 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.371 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.371 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.371 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.371 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.372 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.372 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.372 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.372 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 10 monclient: shutdown 2024-09-16T05:07:51.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:51.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 1 -- 172.21.15.114:0/2979036984 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f497c0a5d00 msgr2=0x7f497c0a4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 1 --2- 172.21.15.114:0/2979036984 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f497c0a5d00 0x7f497c0a4980 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4974009d90 tx=0x7f497401c750 comp rx=0 tx=0).stop 2024-09-16T05:07:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 1 -- 172.21.15.114:0/2979036984 shutdown_connections 2024-09-16T05:07:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 1 --2- 172.21.15.114:0/2979036984 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f497c0a5d00 0x7f497c0a4980 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 1 --2- 172.21.15.114:0/2979036984 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f497c1376e0 0x7f497c137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 1 -- 172.21.15.114:0/2979036984 >> 172.21.15.114:0/2979036984 conn(0x7f497c0a9a40 msgr2=0x7f497c0a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 1 -- 172.21.15.114:0/2979036984 shutdown_connections 2024-09-16T05:07:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 1 -- 172.21.15.114:0/2979036984 wait complete. 2024-09-16T05:07:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 10 monclient: build_initial_monmap 2024-09-16T05:07:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 10 monclient: monmap: 2024-09-16T05:07:51.374 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:51.374 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:51.374 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:51.374 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:51.374 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:51.374 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.375 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:51.375 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:51.375 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:51.375 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 1 librados: starting msgr at 2024-09-16T05:07:51.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 1 librados: starting objecter 2024-09-16T05:07:51.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 1 Processor -- start 2024-09-16T05:07:51.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 1 -- start start 2024-09-16T05:07:51.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 1 librados: setting wanted keys 2024-09-16T05:07:51.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 1 librados: calling monclient init 2024-09-16T05:07:51.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.362+0000 7f4981877640 10 monclient: init 2024-09-16T05:07:51.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:51.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:07:51.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f497c0a5d00 0x7f497c1cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 10 monclient(hunting): picked mon.noname-c con 0x7f497c0a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:51.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f497c1370a0 0x7f497c1cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 10 monclient(hunting): picked mon.noname-a con 0x7f497c1370a0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:51.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f497c1376e0 0x7f497c13bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 10 monclient(hunting): picked mon.noname-b con 0x7f497c1376e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:51.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f497c141d80 con 0x7f497c1370a0 2024-09-16T05:07:51.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f497c141c00 con 0x7f497c1376e0 2024-09-16T05:07:51.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f497c141f00 con 0x7f497c0a5d00 2024-09-16T05:07:51.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:51.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 10 monclient(hunting): authenticate will time out at 955.996564s 2024-09-16T05:07:51.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f497c0a5d00 0x7f497c1cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f497c1370a0 0x7f497c1cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f497c0a5d00 0x7f497c1cf2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55648/0 (socket says 172.21.15.114:55648) 2024-09-16T05:07:51.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 1 -- 172.21.15.114:0/1845599169 learned_addr learned my addr 172.21.15.114:0/1845599169 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:51.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497a7fc640 10 monclient(hunting): get_auth_request con 0x7f497c1370a0 auth_method 0 2024-09-16T05:07:51.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497b7fe640 1 --2- 172.21.15.114:0/1845599169 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f497c1376e0 0x7f497c13bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497b7fe640 10 monclient(hunting): get_auth_request con 0x7f497c1376e0 auth_method 0 2024-09-16T05:07:51.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 10 monclient(hunting): get_auth_request con 0x7f497c0a5d00 auth_method 0 2024-09-16T05:07:51.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-09-16T05:07:51.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 1 -- 172.21.15.114:0/1845599169 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f497c1376e0 msgr2=0x7f497c13bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 1 --2- 172.21.15.114:0/1845599169 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f497c1376e0 0x7f497c13bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 1 -- 172.21.15.114:0/1845599169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f497c1370a0 msgr2=0x7f497c1cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 1 --2- 172.21.15.114:0/1845599169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f497c1370a0 0x7f497c1cf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 10 monclient: _finish_hunting 0 2024-09-16T05:07:51.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 1 monclient: found mon.noname-c 2024-09-16T05:07:51.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:51.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:51.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 1 -- 172.21.15.114:0/1845599169 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497c120060 con 0x7f497c0a5d00 2024-09-16T05:07:51.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f497affd640 1 --2- 172.21.15.114:0/1845599169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f497c0a5d00 0x7f497c1cf2b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f496c00e930 tx=0x7f496c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 10 client.?.objecter ms_handle_connect 0x7f497c0a5d00 2024-09-16T05:07:51.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:51.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 1 -- 172.21.15.114:0/1845599169 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c017070 con 0x7f497c0a5d00 2024-09-16T05:07:51.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:51.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 10 monclient: dump: 2024-09-16T05:07:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.388 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.388 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 10 monclient: _finish_auth 0 2024-09-16T05:07:51.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 10 monclient: _check_auth_tickets 2024-09-16T05:07:51.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:51.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 1 -- 172.21.15.114:0/1845599169 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f496c00ca00 con 0x7f497c0a5d00 2024-09-16T05:07:51.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:51.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 1 -- 172.21.15.114:0/1845599169 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c00cdb0 con 0x7f497c0a5d00 2024-09-16T05:07:51.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:51.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 10 monclient: dump: 2024-09-16T05:07:51.389 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.389 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.389 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.390 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.390 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.390 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.390 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.390 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.390 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.390 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 5 monclient: authenticate success, global_id 4259 2024-09-16T05:07:51.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 10 monclient: _renew_subs 2024-09-16T05:07:51.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7fd78b325640 10 monclient: get_monmap_and_config 2024-09-16T05:07:51.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7fd78b325640 10 monclient: build_initial_monmap 2024-09-16T05:07:51.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7fd78b325640 10 monclient: monmap: 2024-09-16T05:07:51.392 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:51.392 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.392 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:51.371013+0000 2024-09-16T05:07:51.392 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:51.371013+0000 2024-09-16T05:07:51.392 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:51.392 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.392 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:51.392 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:51.393 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:51.393 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:51.393 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:51.393 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:51.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 1 -- 172.21.15.114:0/1845599169 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f497c120340 con 0x7f497c0a5d00 2024-09-16T05:07:51.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f49797fa640 4 set_mon_vals no callback set 2024-09-16T05:07:51.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:51.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 10 monclient: _renew_subs 2024-09-16T05:07:51.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:51.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 1 -- 172.21.15.114:0/1845599169 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f497c13e060 con 0x7f497c0a5d00 2024-09-16T05:07:51.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4981877640 1 librados: init done 2024-09-16T05:07:51.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 1 -- 172.21.15.114:0/1845599169 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f496c01a070 con 0x7f497c0a5d00 2024-09-16T05:07:51.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.366+0000 7f4963fff640 1 --2- 172.21.15.114:0/1845599169 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4968037810 0x7f4968039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.370+0000 7f497a7fc640 1 --2- 172.21.15.114:0/1845599169 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4968037810 0x7f4968039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.370+0000 7f497a7fc640 10 monclient: get_auth_request con 0x7f4968037810 auth_method 0 2024-09-16T05:07:51.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.370+0000 7f4981877640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.370+0000 7f4981877640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.370+0000 7f4981877640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:51.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.370+0000 7f4981877640 1 -- 172.21.15.114:0/1845599169 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f497c13d780 con 0x7f497c0a5d00 2024-09-16T05:07:51.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.370+0000 7f4963fff640 1 -- 172.21.15.114:0/1845599169 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f496c011da0 con 0x7f497c0a5d00 2024-09-16T05:07:51.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.370+0000 7f4963fff640 10 client.4259.objecter ms_dispatch 0x7f497c000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:51.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.370+0000 7f4963fff640 3 client.4259.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:51.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.370+0000 7f4963fff640 3 client.4259.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:51.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.370+0000 7f4963fff640 20 client.4259.objecter dump_active .. 0 homeless 2024-09-16T05:07:51.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.370+0000 7f4963fff640 10 client.4259.objecter ms_handle_connect 0x7f4968037810 2024-09-16T05:07:51.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.370+0000 7f497a7fc640 1 --2- 172.21.15.114:0/1845599169 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4968037810 0x7f4968039cd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f497c13d380 tx=0x7f4970001aa0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78b325640 1 Processor -- start 2024-09-16T05:07:51.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78b325640 1 -- start start 2024-09-16T05:07:51.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78b325640 10 monclient: init 2024-09-16T05:07:51.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78b325640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:51.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78b325640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-16T05:07:51.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78b325640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd77c023cb0 0x7fd77c025370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78b325640 10 monclient(hunting): picked mon.noname-a con 0x7fd77c023cb0 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:51.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78b325640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd77c024090 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:51.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78b325640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd77c0244a0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:51.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78b325640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd77c024930 con 0x7fd77c0244a0 2024-09-16T05:07:51.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78b325640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd77c0bc470 con 0x7fd77c024090 2024-09-16T05:07:51.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78b325640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78b325640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd77c027870 con 0x7fd77c023cb0 2024-09-16T05:07:51.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78b325640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:51.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78b325640 10 monclient(hunting): authenticate will time out at 956.008869s 2024-09-16T05:07:51.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd77c023cb0 0x7fd77c025370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78ab24640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd77c0244a0 0x7fd77c0bbd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38654/0 (socket says 172.21.15.114:38654) 2024-09-16T05:07:51.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78ab24640 1 -- 172.21.15.114:0/944367380 learned_addr learned my addr 172.21.15.114:0/944367380 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:51.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 10 monclient(hunting): get_auth_request con 0x7fd77c023cb0 auth_method 0 2024-09-16T05:07:51.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 10 monclient(hunting): handle_auth_done global_id 4317 payload 274 2024-09-16T05:07:51.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 1 -- 172.21.15.114:0/944367380 >> v1:172.21.15.149:6789/0 conn(0x7fd77c024090 legacy=0x7fd77c0bb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 1 -- 172.21.15.114:0/944367380 >> v1:172.21.15.114:6789/0 conn(0x7fd77c0244a0 legacy=0x7fd77c0bbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 10 monclient: _finish_hunting 0 2024-09-16T05:07:51.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 1 monclient: found mon.noname-a 2024-09-16T05:07:51.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:51.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 1 -- 172.21.15.114:0/944367380 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77c027e30 con 0x7fd77c023cb0 2024-09-16T05:07:51.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7f4963fff640 1 -- 172.21.15.114:0/1845599169 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f496c01a360 con 0x7f497c0a5d00 2024-09-16T05:07:51.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7f4963fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7f4963fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:51.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd78a323640 1 --2- 172.21.15.114:0/944367380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd77c023cb0 0x7fd77c025370 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd7800097c0 tx=0x7fd78001c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceb8ed5ee2ef9eff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd789321640 1 -- 172.21.15.114:0/944367380 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7800b0070 con 0x7fd77c023cb0 2024-09-16T05:07:51.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.378+0000 7fd789321640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd789321640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:51.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd789321640 10 monclient: dump: 2024-09-16T05:07:51.404 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.404 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.404 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.404 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.404 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.404 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.405 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.405 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.405 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.405 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd789321640 10 monclient: _finish_auth 0 2024-09-16T05:07:51.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd789321640 10 monclient: _check_auth_tickets 2024-09-16T05:07:51.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd789321640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:51.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 5 monclient: authenticate success, global_id 4317 2024-09-16T05:07:51.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:51.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd789321640 1 -- 172.21.15.114:0/944367380 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd780004230 con 0x7fd77c023cb0 2024-09-16T05:07:51.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd789321640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:51.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:51.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 4 set_mon_vals no callback set 2024-09-16T05:07:51.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 10 monclient: shutdown 2024-09-16T05:07:51.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:51.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 -- 172.21.15.114:0/944367380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd77c023cb0 msgr2=0x7fd77c025370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 --2- 172.21.15.114:0/944367380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd77c023cb0 0x7fd77c025370 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd7800097c0 tx=0x7fd78001c330 comp rx=0 tx=0).stop 2024-09-16T05:07:51.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 -- 172.21.15.114:0/944367380 shutdown_connections 2024-09-16T05:07:51.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 --2- 172.21.15.114:0/944367380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd77c023cb0 0x7fd77c025370 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 -- 172.21.15.114:0/944367380 >> 172.21.15.114:0/944367380 conn(0x7fd77c006320 msgr2=0x7fd77c006730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:51.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 -- 172.21.15.114:0/944367380 shutdown_connections 2024-09-16T05:07:51.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 -- 172.21.15.114:0/944367380 wait complete. 2024-09-16T05:07:51.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 10 monclient: build_initial_monmap 2024-09-16T05:07:51.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 10 monclient: monmap: 2024-09-16T05:07:51.407 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:51.407 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:51.407 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:51.407 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:51.407 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:51.407 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.407 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:51.408 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:51.408 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:51.408 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 librados: starting msgr at 2024-09-16T05:07:51.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 librados: starting objecter 2024-09-16T05:07:51.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 Processor -- start 2024-09-16T05:07:51.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 -- start start 2024-09-16T05:07:51.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 librados: setting wanted keys 2024-09-16T05:07:51.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 librados: calling monclient init 2024-09-16T05:07:51.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 10 monclient: init 2024-09-16T05:07:51.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:51.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:07:51.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd77c023cb0 0x7fd77c153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 10 monclient(hunting): picked mon.noname-b con 0x7fd77c023cb0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:51.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd77c024090 0x7fd77c0bae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 10 monclient(hunting): picked mon.noname-a con 0x7fd77c024090 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:51.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd77c0244a0 0x7fd77c0bb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 10 monclient(hunting): picked mon.noname-c con 0x7fd77c0244a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:51.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c0c7770 con 0x7fd77c024090 2024-09-16T05:07:51.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c0c75f0 con 0x7fd77c023cb0 2024-09-16T05:07:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd77c0c78f0 con 0x7fd77c0244a0 2024-09-16T05:07:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78b325640 10 monclient(hunting): authenticate will time out at 956.015309s 2024-09-16T05:07:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78ab24640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd77c0244a0 0x7fd77c0bb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78ab24640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd77c0244a0 0x7fd77c0bb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55652/0 (socket says 172.21.15.114:55652) 2024-09-16T05:07:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78ab24640 1 -- 172.21.15.114:0/4083971554 learned_addr learned my addr 172.21.15.114:0/4083971554 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78ab24640 10 monclient(hunting): get_auth_request con 0x7fd77c0244a0 auth_method 0 2024-09-16T05:07:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78ab24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78ab24640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78ab24640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78a323640 1 --2- 172.21.15.114:0/4083971554 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd77c023cb0 0x7fd77c153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78a323640 10 monclient(hunting): get_auth_request con 0x7fd77c023cb0 auth_method 0 2024-09-16T05:07:51.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78a323640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78a323640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78a323640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78a323640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78a323640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd78a323640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd789b22640 1 --2- 172.21.15.114:0/4083971554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd77c024090 0x7fd77c0bae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd789b22640 10 monclient(hunting): get_auth_request con 0x7fd77c024090 auth_method 0 2024-09-16T05:07:51.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd789b22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd789b22640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.382+0000 7fd789b22640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78ab24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78ab24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78ab24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78a323640 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2024-09-16T05:07:51.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78a323640 1 -- 172.21.15.114:0/4083971554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd77c0244a0 msgr2=0x7fd77c0bb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78a323640 1 --2- 172.21.15.114:0/4083971554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd77c0244a0 0x7fd77c0bb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78a323640 1 -- 172.21.15.114:0/4083971554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd77c024090 msgr2=0x7fd77c0bae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78a323640 1 --2- 172.21.15.114:0/4083971554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd77c024090 0x7fd77c0bae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78a323640 10 monclient: _finish_hunting 0 2024-09-16T05:07:51.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78a323640 1 monclient: found mon.noname-b 2024-09-16T05:07:51.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78a323640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:51.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78a323640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.149:3300/0 2024-09-16T05:07:51.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78a323640 1 -- 172.21.15.114:0/4083971554 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77c02c010 con 0x7fd77c023cb0 2024-09-16T05:07:51.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78a323640 1 --2- 172.21.15.114:0/4083971554 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd77c023cb0 0x7fd77c153b80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd780004000 tx=0x7fd7800190d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 10 client.?.objecter ms_handle_connect 0x7fd77c023cb0 2024-09-16T05:07:51.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:51.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 1 -- 172.21.15.114:0/4083971554 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7800b9020 con 0x7fd77c023cb0 2024-09-16T05:07:51.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:51.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 10 monclient: dump: 2024-09-16T05:07:51.414 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.414 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.414 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.414 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.414 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.414 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.414 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.414 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.414 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.414 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 10 monclient: _finish_auth 0 2024-09-16T05:07:51.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 10 monclient: _check_auth_tickets 2024-09-16T05:07:51.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:51.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 1 -- 172.21.15.114:0/4083971554 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7800ab640 con 0x7fd77c023cb0 2024-09-16T05:07:51.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78b325640 5 monclient: authenticate success, global_id 4330 2024-09-16T05:07:51.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:51.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 1 -- 172.21.15.114:0/4083971554 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7800b0040 con 0x7fd77c023cb0 2024-09-16T05:07:51.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:51.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 10 monclient: dump: 2024-09-16T05:07:51.415 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.416 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.416 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.416 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.416 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.416 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.416 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.416 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.416 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.416 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd788b20640 4 set_mon_vals no callback set 2024-09-16T05:07:51.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78b325640 10 monclient: _renew_subs 2024-09-16T05:07:51.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78b325640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:51.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78b325640 1 -- 172.21.15.114:0/4083971554 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd77c0bbf10 con 0x7fd77c023cb0 2024-09-16T05:07:51.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78b325640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:51.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78b325640 10 monclient: _renew_subs 2024-09-16T05:07:51.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78b325640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:51.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78b325640 1 -- 172.21.15.114:0/4083971554 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd77c0c4480 con 0x7fd77c023cb0 2024-09-16T05:07:51.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78b325640 1 librados: init done 2024-09-16T05:07:51.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78b325640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78b325640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78b325640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:51.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd78b325640 1 -- 172.21.15.114:0/4083971554 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd77c0a4580 con 0x7fd77c023cb0 2024-09-16T05:07:51.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 1 -- 172.21.15.114:0/4083971554 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd7800bc020 con 0x7fd77c023cb0 2024-09-16T05:07:51.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 1 --2- 172.21.15.114:0/4083971554 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd768037840 0x7fd768039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 1 -- 172.21.15.114:0/4083971554 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7fd7800e2bb0 con 0x7fd77c023cb0 2024-09-16T05:07:51.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 10 client.4330.objecter ms_dispatch 0x7fd784000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:51.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 3 client.4330.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:51.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 3 client.4330.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:51.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd77affd640 20 client.4330.objecter dump_active .. 0 homeless 2024-09-16T05:07:51.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd789b22640 1 --2- 172.21.15.114:0/4083971554 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd768037840 0x7fd768039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.386+0000 7fd789b22640 10 monclient: get_auth_request con 0x7fd768037840 auth_method 0 2024-09-16T05:07:51.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.390+0000 7fd77affd640 10 client.4330.objecter ms_handle_connect 0x7fd768037840 2024-09-16T05:07:51.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.390+0000 7fd789b22640 1 --2- 172.21.15.114:0/4083971554 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd768037840 0x7fd768039d00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd77c026ff0 tx=0x7fd774009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.402+0000 7fd77affd640 1 -- 172.21.15.114:0/4083971554 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd7800c3030 con 0x7fd77c023cb0 2024-09-16T05:07:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.402+0000 7fd77affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.402+0000 7fd77affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:51.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 10 monclient: get_monmap_and_config 2024-09-16T05:07:51.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 10 monclient: build_initial_monmap 2024-09-16T05:07:51.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 10 monclient: monmap: 2024-09-16T05:07:51.425 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:51.425 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.425 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:51.408443+0000 2024-09-16T05:07:51.425 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:51.408443+0000 2024-09-16T05:07:51.425 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:51.425 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.425 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:51.426 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 1 Processor -- start 2024-09-16T05:07:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 1 -- start start 2024-09-16T05:07:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 10 monclient: init 2024-09-16T05:07:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T05:07:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbca80a4270 0x7fbca806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 10 monclient(hunting): picked mon.noname-b con 0x7fbca80a4270 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fbca80a4650 0x7fbca80a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 10 monclient(hunting): picked mon.noname-c con 0x7fbca80a4650 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbca813b800 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbca8073370 con 0x7fbca813b800 2024-09-16T05:07:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fbca806eab0 con 0x7fbca80a4650 2024-09-16T05:07:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbca80a52f0 con 0x7fbca80a4270 2024-09-16T05:07:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcb005e640 10 monclient(hunting): authenticate will time out at 956.037927s 2024-09-16T05:07:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbca80a4270 0x7fbca806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcae5fb640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fbca813b800 0x7fbca813bc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38666/0 (socket says 172.21.15.114:38666) 2024-09-16T05:07:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcae5fb640 1 -- 172.21.15.114:0/3019877256 learned_addr learned my addr 172.21.15.114:0/3019877256 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 10 monclient(hunting): get_auth_request con 0x7fbca80a4270 auth_method 0 2024-09-16T05:07:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcad5f9640 1 --2- 172.21.15.114:0/3019877256 >> v2:172.21.15.149:3300/0 conn(0x7fbca80a4650 0x7fbca80a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2024-09-16T05:07:51.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 1 -- 172.21.15.114:0/3019877256 >> v2:172.21.15.149:3300/0 conn(0x7fbca80a4650 msgr2=0x7fbca80a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:51.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 1 --2- 172.21.15.114:0/3019877256 >> v2:172.21.15.149:3300/0 conn(0x7fbca80a4650 0x7fbca80a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 1 -- 172.21.15.114:0/3019877256 >> v1:172.21.15.114:6789/0 conn(0x7fbca813b800 legacy=0x7fbca813bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 10 monclient: _finish_hunting 0 2024-09-16T05:07:51.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 1 monclient: found mon.noname-b 2024-09-16T05:07:51.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcaddfa640 1 -- 172.21.15.114:0/3019877256 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca806e180 con 0x7fbca80a4270 2024-09-16T05:07:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcad5f9640 10 monclient: get_auth_request con 0x7fbca80a4650 auth_method 0 2024-09-16T05:07:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcad5f9640 1 --2- 172.21.15.114:0/3019877256 >> v2:172.21.15.149:3300/0 conn(0x7fbca80a4650 0x7fbca80a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:07:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.406+0000 7fbcacdf8640 1 -- 172.21.15.114:0/3019877256 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3458965762 0 0) 0x7fbca8073370 con 0x7fbca813b800 2024-09-16T05:07:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.410+0000 7fbcacdf8640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:07:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.410+0000 7fbcaddfa640 1 --2- 172.21.15.114:0/3019877256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbca80a4270 0x7fbca806ee40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbca40097c0 tx=0x7fbca401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ab6d683c89feb36 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.410+0000 7fbcacdf8640 1 -- 172.21.15.114:0/3019877256 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca40b0070 con 0x7fbca80a4270 2024-09-16T05:07:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.410+0000 7fbcacdf8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.410+0000 7fbcacdf8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.410+0000 7fbcacdf8640 10 monclient: dump: 2024-09-16T05:07:51.430 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.430 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.430 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.430 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.430 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.430 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.430 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.430 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.430 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.430 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.410+0000 7fbcacdf8640 10 monclient: _finish_auth 0 2024-09-16T05:07:51.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.410+0000 7fbcacdf8640 10 monclient: _check_auth_tickets 2024-09-16T05:07:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.410+0000 7fbcacdf8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.410+0000 7fbcb005e640 5 monclient: authenticate success, global_id 4265 2024-09-16T05:07:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.410+0000 7fbcb005e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.410+0000 7fbcacdf8640 1 -- 172.21.15.114:0/3019877256 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca40ab600 con 0x7fbca80a4270 2024-09-16T05:07:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.410+0000 7fbcacdf8640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.410+0000 7fbcb005e640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.410+0000 7fbcb005e640 4 set_mon_vals no callback set 2024-09-16T05:07:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcacdf8640 1 -- 172.21.15.114:0/3019877256 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca40ab9d0 con 0x7fbca80a4270 2024-09-16T05:07:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcacdf8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcacdf8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcacdf8640 10 monclient: dump: 2024-09-16T05:07:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.432 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.432 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.432 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.432 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.432 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.432 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.432 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.432 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.432 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcb005e640 10 monclient: shutdown 2024-09-16T05:07:51.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcb005e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:51.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcb005e640 1 -- 172.21.15.114:0/3019877256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbca80a4270 msgr2=0x7fbca806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcb005e640 1 --2- 172.21.15.114:0/3019877256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbca80a4270 0x7fbca806ee40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbca40097c0 tx=0x7fbca401c360 comp rx=0 tx=0).stop 2024-09-16T05:07:51.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcb005e640 1 -- 172.21.15.114:0/3019877256 shutdown_connections 2024-09-16T05:07:51.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcb005e640 1 --2- 172.21.15.114:0/3019877256 >> v2:172.21.15.149:3300/0 conn(0x7fbca80a4650 0x7fbca80a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcb005e640 1 --2- 172.21.15.114:0/3019877256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbca80a4270 0x7fbca806ee40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcb005e640 1 -- 172.21.15.114:0/3019877256 >> 172.21.15.114:0/3019877256 conn(0x7fbca80a9a40 msgr2=0x7fbca80a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:51.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcb005e640 1 -- 172.21.15.114:0/3019877256 shutdown_connections 2024-09-16T05:07:51.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcb005e640 1 -- 172.21.15.114:0/3019877256 wait complete. 2024-09-16T05:07:51.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcb005e640 10 monclient: build_initial_monmap 2024-09-16T05:07:51.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcb005e640 10 monclient: monmap: 2024-09-16T05:07:51.433 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:51.433 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:51.433 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:51.433 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:51.433 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:51.433 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.433 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:51.434 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:51.434 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:51.434 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcb005e640 1 librados: starting msgr at 2024-09-16T05:07:51.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcb005e640 1 librados: starting objecter 2024-09-16T05:07:51.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.414+0000 7fbcb005e640 1 Processor -- start 2024-09-16T05:07:51.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 1 -- start start 2024-09-16T05:07:51.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 1 librados: setting wanted keys 2024-09-16T05:07:51.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 1 librados: calling monclient init 2024-09-16T05:07:51.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient: init 2024-09-16T05:07:51.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:51.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:07:51.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbca80a4270 0x7fbca809c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient(hunting): picked mon.noname-c con 0x7fbca80a4270 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:51.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbca80a4650 0x7fbca809cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient(hunting): picked mon.noname-b con 0x7fbca80a4650 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:51.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbca813b800 0x7fbca809d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient(hunting): picked mon.noname-a con 0x7fbca813b800 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:51.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbca8144460 con 0x7fbca813b800 2024-09-16T05:07:51.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fbca81442e0 con 0x7fbca80a4650 2024-09-16T05:07:51.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbca81445e0 con 0x7fbca80a4270 2024-09-16T05:07:51.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:51.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient(hunting): authenticate will time out at 956.048721s 2024-09-16T05:07:51.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbca80a4270 0x7fbca809c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbca80a4270 0x7fbca809c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55672/0 (socket says 172.21.15.114:55672) 2024-09-16T05:07:51.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 1 -- 172.21.15.114:0/1594542851 learned_addr learned my addr 172.21.15.114:0/1594542851 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:51.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 10 monclient(hunting): get_auth_request con 0x7fbca80a4270 auth_method 0 2024-09-16T05:07:51.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcae5fb640 1 --2- 172.21.15.114:0/1594542851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbca813b800 0x7fbca809d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcad5f9640 1 --2- 172.21.15.114:0/1594542851 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbca80a4650 0x7fbca809cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcae5fb640 10 monclient(hunting): get_auth_request con 0x7fbca813b800 auth_method 0 2024-09-16T05:07:51.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcae5fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcae5fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcae5fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcad5f9640 10 monclient(hunting): get_auth_request con 0x7fbca80a4650 auth_method 0 2024-09-16T05:07:51.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcad5f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcad5f9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcad5f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcad5f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcad5f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcad5f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcae5fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcae5fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcae5fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2024-09-16T05:07:51.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 1 -- 172.21.15.114:0/1594542851 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbca80a4650 msgr2=0x7fbca809cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 1 --2- 172.21.15.114:0/1594542851 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbca80a4650 0x7fbca809cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 1 -- 172.21.15.114:0/1594542851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbca813b800 msgr2=0x7fbca809d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 1 --2- 172.21.15.114:0/1594542851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbca813b800 0x7fbca809d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 10 monclient: _finish_hunting 0 2024-09-16T05:07:51.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 1 monclient: found mon.noname-c 2024-09-16T05:07:51.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:51.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:51.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 1 -- 172.21.15.114:0/1594542851 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca80a0000 con 0x7fbca80a4270 2024-09-16T05:07:51.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcaddfa640 1 --2- 172.21.15.114:0/1594542851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbca80a4270 0x7fbca809c630 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbca40ae040 tx=0x7fbca4019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 10 client.?.objecter ms_handle_connect 0x7fbca80a4270 2024-09-16T05:07:51.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:51.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 1 -- 172.21.15.114:0/1594542851 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca40b8030 con 0x7fbca80a4270 2024-09-16T05:07:51.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:51.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 10 monclient: dump: 2024-09-16T05:07:51.441 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.441 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.441 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.442 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.442 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.442 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.442 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.442 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.443 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.443 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:07:51.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:07:51.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:51.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 1 -- 172.21.15.114:0/1594542851 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca40b4410 con 0x7fbca80a4270 2024-09-16T05:07:51.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:51.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 1 -- 172.21.15.114:0/1594542851 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca40b0040 con 0x7fbca80a4270 2024-09-16T05:07:51.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:51.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 10 monclient: dump: 2024-09-16T05:07:51.445 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.445 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.445 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.445 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.445 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.446 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.446 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.447 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.447 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.447 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9ffff640 4 set_mon_vals no callback set 2024-09-16T05:07:51.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 5 monclient: authenticate success, global_id 4268 2024-09-16T05:07:51.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient: _renew_subs 2024-09-16T05:07:51.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:51.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 1 -- 172.21.15.114:0/1594542851 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbca809ebb0 con 0x7fbca80a4270 2024-09-16T05:07:51.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:51.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient: _renew_subs 2024-09-16T05:07:51.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:51.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 1 -- 172.21.15.114:0/1594542851 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbca80a02e0 con 0x7fbca80a4270 2024-09-16T05:07:51.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 1 librados: init done 2024-09-16T05:07:51.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:51.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbcb005e640 1 -- 172.21.15.114:0/1594542851 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbca8070c30 con 0x7fbca80a4270 2024-09-16T05:07:51.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.418+0000 7fbc9e7fc640 1 -- 172.21.15.114:0/1594542851 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbca40bb070 con 0x7fbca80a4270 2024-09-16T05:07:51.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.422+0000 7fbc9e7fc640 1 --2- 172.21.15.114:0/1594542851 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbc8c037840 0x7fbc8c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.422+0000 7fbc9e7fc640 1 -- 172.21.15.114:0/1594542851 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7fbca40e1e30 con 0x7fbca80a4270 2024-09-16T05:07:51.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.422+0000 7fbc9e7fc640 10 client.4268.objecter ms_dispatch 0x7fbca8000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:51.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.422+0000 7fbc9e7fc640 3 client.4268.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:51.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.422+0000 7fbc9e7fc640 3 client.4268.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:51.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.422+0000 7fbc9e7fc640 20 client.4268.objecter dump_active .. 0 homeless 2024-09-16T05:07:51.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.422+0000 7fbcad5f9640 1 --2- 172.21.15.114:0/1594542851 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbc8c037840 0x7fbc8c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.422+0000 7fbcad5f9640 10 monclient: get_auth_request con 0x7fbc8c037840 auth_method 0 2024-09-16T05:07:51.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.422+0000 7fbcad5f9640 1 --2- 172.21.15.114:0/1594542851 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbc8c037840 0x7fbc8c039d00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbca80a6c20 tx=0x7fbc98008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.422+0000 7fbc9e7fc640 10 client.4268.objecter ms_handle_connect 0x7fbc8c037840 2024-09-16T05:07:51.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.422+0000 7f03daef7640 10 monclient: get_monmap_and_config 2024-09-16T05:07:51.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.422+0000 7f03daef7640 10 monclient: build_initial_monmap 2024-09-16T05:07:51.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.422+0000 7f03daef7640 10 monclient: monmap: 2024-09-16T05:07:51.455 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:51.455 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.455 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:51.427752+0000 2024-09-16T05:07:51.455 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:51.427752+0000 2024-09-16T05:07:51.455 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:51.456 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.456 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:51.456 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:51.456 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:51.456 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:51.456 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:51.456 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 1 Processor -- start 2024-09-16T05:07:51.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 1 -- start start 2024-09-16T05:07:51.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 10 monclient: init 2024-09-16T05:07:51.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:51.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-16T05:07:51.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f03cc026140 0x7f03cc0220a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 10 monclient(hunting): picked mon.noname-c con 0x7f03cc026140 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:51.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f03cc0225e0 0x7f03cc022a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 10 monclient(hunting): picked mon.noname-a con 0x7f03cc0225e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:51.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03cc0b83c0 0x7f03cc0b8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 10 monclient(hunting): picked mon.noname-b con 0x7f03cc0b83c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:51.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f03cc026520 con 0x7f03cc0225e0 2024-09-16T05:07:51.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f03cc021240 con 0x7f03cc026140 2024-09-16T05:07:51.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f03cc024b90 con 0x7f03cc0b83c0 2024-09-16T05:07:51.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:51.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03daef7640 10 monclient(hunting): authenticate will time out at 956.056987s 2024-09-16T05:07:51.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03cc0b83c0 0x7f03cc0b8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03d9ef5640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f03cc026140 0x7f03cc0220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03cc0b83c0 0x7f03cc0b8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55684/0 (socket says 172.21.15.114:55684) 2024-09-16T05:07:51.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 1 -- 172.21.15.114:0/3537343206 learned_addr learned my addr 172.21.15.114:0/3537343206 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:51.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 10 monclient(hunting): get_auth_request con 0x7f03cc0b83c0 auth_method 0 2024-09-16T05:07:51.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03d9ef5640 10 monclient(hunting): get_auth_request con 0x7f03cc026140 auth_method 0 2024-09-16T05:07:51.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03d9ef5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03d9ef5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03d9ef5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03d9ef5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03d9ef5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03d9ef5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03d96f4640 1 --2- 172.21.15.114:0/3537343206 >> v2:172.21.15.114:3300/0 conn(0x7f03cc0225e0 0x7f03cc022a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-09-16T05:07:51.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 1 -- 172.21.15.114:0/3537343206 >> v2:172.21.15.149:3300/0 conn(0x7f03cc026140 msgr2=0x7f03cc0220a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:51.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 1 --2- 172.21.15.114:0/3537343206 >> v2:172.21.15.149:3300/0 conn(0x7f03cc026140 0x7f03cc0220a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 1 -- 172.21.15.114:0/3537343206 >> v2:172.21.15.114:3300/0 conn(0x7f03cc0225e0 msgr2=0x7f03cc022a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:51.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 1 --2- 172.21.15.114:0/3537343206 >> v2:172.21.15.114:3300/0 conn(0x7f03cc0225e0 0x7f03cc022a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 10 monclient: _finish_hunting 0 2024-09-16T05:07:51.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 1 monclient: found mon.noname-b 2024-09-16T05:07:51.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:51.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:51.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 1 -- 172.21.15.114:0/3537343206 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03cc0242f0 con 0x7f03cc0b83c0 2024-09-16T05:07:51.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03d96f4640 10 monclient: get_auth_request con 0x7f03cc0225e0 auth_method 0 2024-09-16T05:07:51.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03d96f4640 1 --2- 172.21.15.114:0/3537343206 >> v2:172.21.15.114:3300/0 conn(0x7f03cc0225e0 0x7f03cc022a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:07:51.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.426+0000 7f03da6f6640 1 --2- 172.21.15.114:0/3537343206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03cc0b83c0 0x7f03cc0b8850 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f03d4051960 tx=0x7f03d4098600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=658736988a0781f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 10 monclient: get_monmap_and_config 2024-09-16T05:07:51.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 10 monclient: build_initial_monmap 2024-09-16T05:07:51.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 10 monclient: monmap: 2024-09-16T05:07:51.461 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:51.461 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.461 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:51.432961+0000 2024-09-16T05:07:51.462 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:51.432961+0000 2024-09-16T05:07:51.462 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:51.462 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.462 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:51.462 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:51.462 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:51.462 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:51.462 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:51.462 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 1 Processor -- start 2024-09-16T05:07:51.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 1 -- start start 2024-09-16T05:07:51.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 10 monclient: init 2024-09-16T05:07:51.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7fbc9e7fc640 1 -- 172.21.15.114:0/1594542851 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fbca40c2070 con 0x7fbca80a4270 2024-09-16T05:07:51.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7fbc9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7fbc9e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:51.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03d8ef3640 1 -- 172.21.15.114:0/3537343206 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d412c070 con 0x7f03cc0b83c0 2024-09-16T05:07:51.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03d8ef3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03d8ef3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:51.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03d8ef3640 10 monclient: dump: 2024-09-16T05:07:51.464 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.464 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.464 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.464 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.464 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.464 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.464 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.465 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.465 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.465 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03d8ef3640 10 monclient: _finish_auth 0 2024-09-16T05:07:51.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03d8ef3640 10 monclient: _check_auth_tickets 2024-09-16T05:07:51.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03d8ef3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:51.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03daef7640 5 monclient: authenticate success, global_id 4271 2024-09-16T05:07:51.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03daef7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:51.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:51.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-16T05:07:51.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03d8ef3640 1 -- 172.21.15.114:0/3537343206 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03d4127640 con 0x7f03cc0b83c0 2024-09-16T05:07:51.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03d8ef3640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:51.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03daef7640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:51.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03daef7640 4 set_mon_vals no callback set 2024-09-16T05:07:51.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c023cb0 0x7f332c025370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 10 monclient(hunting): picked mon.noname-b con 0x7f332c023cb0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:51.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f332c024090 0x7f332c024470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 10 monclient(hunting): picked mon.noname-c con 0x7f332c024090 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:51.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f332c0bb650 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:51.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f332c027e30 con 0x7f332c0bb650 2024-09-16T05:07:51.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f332c027be0 con 0x7f332c024090 2024-09-16T05:07:51.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f332c027870 con 0x7f332c023cb0 2024-09-16T05:07:51.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:51.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333a38c640 10 monclient(hunting): authenticate will time out at 956.062104s 2024-09-16T05:07:51.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f332c024090 0x7f332c024470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3339b8b640 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f332c0bb650 0x7f332c0bbb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.114:43634/0 (socket says 172.21.15.114:43634) 2024-09-16T05:07:51.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3339b8b640 1 -- 172.21.15.114:0/472833756 learned_addr learned my addr 172.21.15.114:0/472833756 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:51.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 10 monclient(hunting): get_auth_request con 0x7f332c024090 auth_method 0 2024-09-16T05:07:51.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333938a640 1 --2- 172.21.15.114:0/472833756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c023cb0 0x7f332c025370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 10 monclient(hunting): handle_auth_done global_id 4339 payload 274 2024-09-16T05:07:51.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 1 -- 172.21.15.114:0/472833756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c023cb0 msgr2=0x7f332c025370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:51.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03daef7640 10 monclient: shutdown 2024-09-16T05:07:51.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03daef7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:51.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03daef7640 1 -- 172.21.15.114:0/3537343206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03cc0b83c0 msgr2=0x7f03cc0b8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03daef7640 1 --2- 172.21.15.114:0/3537343206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03cc0b83c0 0x7f03cc0b8850 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f03d4051960 tx=0x7f03d4098600 comp rx=0 tx=0).stop 2024-09-16T05:07:51.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 1 --2- 172.21.15.114:0/472833756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c023cb0 0x7f332c025370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 1 -- 172.21.15.114:0/472833756 >> v1:172.21.15.149:6789/0 conn(0x7f332c0bb650 legacy=0x7f332c0bbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 10 monclient: _finish_hunting 0 2024-09-16T05:07:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 1 monclient: found mon.noname-c 2024-09-16T05:07:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:07:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f3338b89640 1 -- 172.21.15.114:0/472833756 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f332c027590 con 0x7f332c024090 2024-09-16T05:07:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f332bfff640 1 -- 172.21.15.114:0/472833756 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1395027558 0 0) 0x7f332c027e30 con 0x7f332c0bb650 2024-09-16T05:07:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f332bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:07:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333938a640 10 monclient: get_auth_request con 0x7f332c023cb0 auth_method 0 2024-09-16T05:07:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f333938a640 1 --2- 172.21.15.114:0/472833756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c023cb0 0x7f332c025370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:07:51.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03daef7640 1 -- 172.21.15.114:0/3537343206 shutdown_connections 2024-09-16T05:07:51.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03daef7640 1 --2- 172.21.15.114:0/3537343206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03cc0b83c0 0x7f03cc0b8850 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03daef7640 1 --2- 172.21.15.114:0/3537343206 >> v2:172.21.15.114:3300/0 conn(0x7f03cc0225e0 0x7f03cc022a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03daef7640 1 --2- 172.21.15.114:0/3537343206 >> v2:172.21.15.149:3300/0 conn(0x7f03cc026140 0x7f03cc0220a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03daef7640 1 -- 172.21.15.114:0/3537343206 >> 172.21.15.114:0/3537343206 conn(0x7f03cc027400 msgr2=0x7f03cc027810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:51.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03daef7640 1 -- 172.21.15.114:0/3537343206 shutdown_connections 2024-09-16T05:07:51.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.430+0000 7f03daef7640 1 -- 172.21.15.114:0/3537343206 wait complete. 2024-09-16T05:07:51.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient: build_initial_monmap 2024-09-16T05:07:51.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient: monmap: 2024-09-16T05:07:51.473 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:51.473 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:51.473 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:51.473 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:51.473 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:51.474 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.474 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:51.474 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:51.474 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:51.474 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 1 librados: starting msgr at 2024-09-16T05:07:51.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 1 librados: starting objecter 2024-09-16T05:07:51.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 1 Processor -- start 2024-09-16T05:07:51.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 1 -- start start 2024-09-16T05:07:51.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 1 librados: setting wanted keys 2024-09-16T05:07:51.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 1 librados: calling monclient init 2024-09-16T05:07:51.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient: init 2024-09-16T05:07:51.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f3338b89640 1 --2- 172.21.15.114:0/472833756 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f332c024090 0x7f332c024470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f33240097c0 tx=0x7f332401c3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c310b9d3f1f04a36 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f332bfff640 1 -- 172.21.15.114:0/472833756 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33240b0070 con 0x7f332c024090 2024-09-16T05:07:51.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f332bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f332bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:51.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f332bfff640 10 monclient: dump: 2024-09-16T05:07:51.476 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.476 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.476 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.476 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.476 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.476 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.476 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.476 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.476 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.476 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f332bfff640 10 monclient: _finish_auth 0 2024-09-16T05:07:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f332bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:07:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f332bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 5 monclient: authenticate success, global_id 4339 2024-09-16T05:07:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f332bfff640 1 -- 172.21.15.114:0/472833756 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33240ab600 con 0x7f332c024090 2024-09-16T05:07:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f332bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f332bfff640 1 -- 172.21.15.114:0/472833756 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33240ab9d0 con 0x7f332c024090 2024-09-16T05:07:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f332bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f332bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f332bfff640 10 monclient: dump: 2024-09-16T05:07:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.478 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 4 set_mon_vals no callback set 2024-09-16T05:07:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:07:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f03cc0225e0 0x7f03cc0b5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient(hunting): picked mon.noname-b con 0x7f03cc0225e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03cc026140 0x7f03cc0b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient(hunting): picked mon.noname-c con 0x7f03cc026140 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03cc0b83c0 0x7f03cc0bb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient(hunting): picked mon.noname-a con 0x7f03cc0b83c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc006dc0 con 0x7f03cc0b83c0 2024-09-16T05:07:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc021670 con 0x7f03cc0225e0 2024-09-16T05:07:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f03cc006f40 con 0x7f03cc026140 2024-09-16T05:07:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient(hunting): authenticate will time out at 956.065430s 2024-09-16T05:07:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d96f4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03cc026140 0x7f03cc0b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f03cc0225e0 0x7f03cc0b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d96f4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03cc026140 0x7f03cc0b5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55700/0 (socket says 172.21.15.114:55700) 2024-09-16T05:07:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d96f4640 1 -- 172.21.15.114:0/2803684399 learned_addr learned my addr 172.21.15.114:0/2803684399 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d96f4640 10 monclient(hunting): get_auth_request con 0x7f03cc026140 auth_method 0 2024-09-16T05:07:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d96f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d96f4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d96f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 10 monclient(hunting): get_auth_request con 0x7f03cc0225e0 auth_method 0 2024-09-16T05:07:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d96f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d96f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d96f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03da6f6640 1 --2- 172.21.15.114:0/2803684399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03cc0b83c0 0x7f03cc0bb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 10 monclient: shutdown 2024-09-16T05:07:51.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:51.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 1 -- 172.21.15.114:0/472833756 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f332c024090 msgr2=0x7f332c024470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 1 --2- 172.21.15.114:0/472833756 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f332c024090 0x7f332c024470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f33240097c0 tx=0x7f332401c3b0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 1 -- 172.21.15.114:0/472833756 shutdown_connections 2024-09-16T05:07:51.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 1 --2- 172.21.15.114:0/472833756 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f332c024090 0x7f332c024470 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 1 --2- 172.21.15.114:0/472833756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c023cb0 0x7f332c025370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 1 -- 172.21.15.114:0/472833756 >> 172.21.15.114:0/472833756 conn(0x7f332c006320 msgr2=0x7f332c006730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:51.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03da6f6640 10 monclient(hunting): get_auth_request con 0x7f03cc0b83c0 auth_method 0 2024-09-16T05:07:51.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03da6f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03da6f6640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03da6f6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03da6f6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03da6f6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03da6f6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2024-09-16T05:07:51.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 1 -- 172.21.15.114:0/2803684399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03cc026140 msgr2=0x7f03cc0b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 1 --2- 172.21.15.114:0/2803684399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03cc026140 0x7f03cc0b5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 1 -- 172.21.15.114:0/2803684399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03cc0b83c0 msgr2=0x7f03cc0bb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 1 --2- 172.21.15.114:0/2803684399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03cc0b83c0 0x7f03cc0bb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 10 monclient: _finish_hunting 0 2024-09-16T05:07:51.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 1 monclient: found mon.noname-b 2024-09-16T05:07:51.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:51.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.149:3300/0 2024-09-16T05:07:51.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 1 -- 172.21.15.114:0/2803684399 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03cc0bd8b0 con 0x7f03cc0225e0 2024-09-16T05:07:51.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 1 -- 172.21.15.114:0/472833756 shutdown_connections 2024-09-16T05:07:51.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 1 -- 172.21.15.114:0/472833756 wait complete. 2024-09-16T05:07:51.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03d9ef5640 1 --2- 172.21.15.114:0/2803684399 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f03cc0225e0 0x7f03cc0b5310 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f03d000df80 tx=0x7f03d0010490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03ca7fc640 10 client.?.objecter ms_handle_connect 0x7f03cc0225e0 2024-09-16T05:07:51.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03ca7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:51.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03ca7fc640 1 -- 172.21.15.114:0/2803684399 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d001f020 con 0x7f03cc0225e0 2024-09-16T05:07:51.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03ca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:51.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03ca7fc640 10 monclient: dump: 2024-09-16T05:07:51.488 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.489 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.489 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.489 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.489 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.489 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.489 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.489 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.489 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.489 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03ca7fc640 10 monclient: _finish_auth 0 2024-09-16T05:07:51.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03ca7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:07:51.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:51.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 5 monclient: authenticate success, global_id 4345 2024-09-16T05:07:51.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient: _renew_subs 2024-09-16T05:07:51.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:51.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 1 -- 172.21.15.114:0/2803684399 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03cc0bdb30 con 0x7f03cc0225e0 2024-09-16T05:07:51.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:51.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient: _renew_subs 2024-09-16T05:07:51.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:51.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 1 -- 172.21.15.114:0/2803684399 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03cc0b7f20 con 0x7f03cc0225e0 2024-09-16T05:07:51.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03daef7640 1 librados: init done 2024-09-16T05:07:51.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03ca7fc640 1 -- 172.21.15.114:0/2803684399 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03d000d280 con 0x7f03cc0225e0 2024-09-16T05:07:51.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:51.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03ca7fc640 1 -- 172.21.15.114:0/2803684399 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d0010da0 con 0x7f03cc0225e0 2024-09-16T05:07:51.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:51.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 10 monclient: build_initial_monmap 2024-09-16T05:07:51.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 10 monclient: monmap: 2024-09-16T05:07:51.491 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:51.491 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:51.491 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:51.491 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:51.491 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:51.492 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.492 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:51.492 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:51.492 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:51.492 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03ca7fc640 10 monclient: dump: 2024-09-16T05:07:51.492 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.493 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.493 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.493 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.493 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.493 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.493 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.493 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.493 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.493 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f03cbfff640 4 set_mon_vals no callback set 2024-09-16T05:07:51.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 1 librados: starting msgr at 2024-09-16T05:07:51.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 1 librados: starting objecter 2024-09-16T05:07:51.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 1 Processor -- start 2024-09-16T05:07:51.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 1 -- start start 2024-09-16T05:07:51.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 1 librados: setting wanted keys 2024-09-16T05:07:51.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 1 librados: calling monclient init 2024-09-16T05:07:51.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.434+0000 7f333a38c640 10 monclient: init 2024-09-16T05:07:51.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f03daef7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:51.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:07:51.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f332c023cb0 0x7f332c0b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient(hunting): picked mon.noname-b con 0x7f332c023cb0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:51.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332c024090 0x7f332c0b7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient(hunting): picked mon.noname-a con 0x7f332c024090 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:51.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c0bb650 0x7f332c0bd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient(hunting): picked mon.noname-c con 0x7f332c0bb650 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:51.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f332c0c6560 con 0x7f332c024090 2024-09-16T05:07:51.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f332c0c63e0 con 0x7f332c023cb0 2024-09-16T05:07:51.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f332c0c66e0 con 0x7f332c0bb650 2024-09-16T05:07:51.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:51.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient(hunting): authenticate will time out at 956.068596s 2024-09-16T05:07:51.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3339b8b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c0bb650 0x7f332c0bd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3339b8b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c0bb650 0x7f332c0bd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55714/0 (socket says 172.21.15.114:55714) 2024-09-16T05:07:51.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3339b8b640 1 -- 172.21.15.114:0/2080038103 learned_addr learned my addr 172.21.15.114:0/2080038103 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:51.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3339b8b640 10 monclient(hunting): get_auth_request con 0x7f332c0bb650 auth_method 0 2024-09-16T05:07:51.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3339b8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3339b8b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3339b8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 1 --2- 172.21.15.114:0/2080038103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332c024090 0x7f332c0b7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333938a640 1 --2- 172.21.15.114:0/2080038103 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f332c023cb0 0x7f332c0b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 10 monclient(hunting): get_auth_request con 0x7f332c024090 auth_method 0 2024-09-16T05:07:51.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333938a640 10 monclient(hunting): get_auth_request con 0x7f332c023cb0 auth_method 0 2024-09-16T05:07:51.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333938a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333938a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333938a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333938a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333938a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333938a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f03ca7fc640 1 -- 172.21.15.114:0/2803684399 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f03d0022020 con 0x7f03cc0225e0 2024-09-16T05:07:51.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f03ca7fc640 1 --2- 172.21.15.114:0/2803684399 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03b8037810 0x7f03b8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f03ca7fc640 1 -- 172.21.15.114:0/2803684399 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f03d0050520 con 0x7f03cc0225e0 2024-09-16T05:07:51.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f03ca7fc640 10 client.4345.objecter ms_dispatch 0x7f03d4000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:51.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f03ca7fc640 3 client.4345.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:51.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f03ca7fc640 3 client.4345.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:51.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f03ca7fc640 20 client.4345.objecter dump_active .. 0 homeless 2024-09-16T05:07:51.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3339b8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3339b8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3339b8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 10 monclient(hunting): handle_auth_done global_id 4335 payload 824 2024-09-16T05:07:51.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 1 -- 172.21.15.114:0/2080038103 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c0bb650 msgr2=0x7f332c0bd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 1 --2- 172.21.15.114:0/2080038103 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c0bb650 0x7f332c0bd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 1 -- 172.21.15.114:0/2080038103 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f332c023cb0 msgr2=0x7f332c0b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 1 --2- 172.21.15.114:0/2080038103 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f332c023cb0 0x7f332c0b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 10 monclient: _finish_hunting 0 2024-09-16T05:07:51.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 1 monclient: found mon.noname-a 2024-09-16T05:07:51.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f03d96f4640 1 --2- 172.21.15.114:0/2803684399 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03b8037810 0x7f03b8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:51.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 1 -- 172.21.15.114:0/2080038103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f332c0bfa30 con 0x7f332c024090 2024-09-16T05:07:51.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333938a640 1 --2- 172.21.15.114:0/2080038103 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f332c023cb0 0x7f332c0b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:51.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f03daef7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f03daef7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:51.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f03daef7640 1 -- 172.21.15.114:0/2803684399 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03bc005090 con 0x7f03cc0225e0 2024-09-16T05:07:51.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f03d96f4640 10 monclient: get_auth_request con 0x7f03b8037810 auth_method 0 2024-09-16T05:07:51.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3338b89640 1 --2- 172.21.15.114:0/2080038103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332c024090 0x7f332c0b7a30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f33240ab000 tx=0x7f3324019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 10 client.?.objecter ms_handle_connect 0x7f332c024090 2024-09-16T05:07:51.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:51.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 1 -- 172.21.15.114:0/2080038103 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33240b0040 con 0x7f332c024090 2024-09-16T05:07:51.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:51.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 10 monclient: dump: 2024-09-16T05:07:51.505 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.505 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.505 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.505 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.505 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.505 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.505 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.505 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.505 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.506 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 10 monclient: _finish_auth 0 2024-09-16T05:07:51.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:07:51.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:51.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 5 monclient: authenticate success, global_id 4335 2024-09-16T05:07:51.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient: _renew_subs 2024-09-16T05:07:51.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 1 -- 172.21.15.114:0/2080038103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f332c0c0ee0 con 0x7f332c024090 2024-09-16T05:07:51.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:51.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient: _renew_subs 2024-09-16T05:07:51.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 1 -- 172.21.15.114:0/2080038103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f332c02d230 con 0x7f332c024090 2024-09-16T05:07:51.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 1 librados: init done 2024-09-16T05:07:51.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f03d96f4640 1 --2- 172.21.15.114:0/2803684399 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03b8037810 0x7f03b8039cd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f03cc022cb0 tx=0x7f03c4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f03ca7fc640 10 client.4345.objecter ms_handle_connect 0x7f03b8037810 2024-09-16T05:07:51.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f333a38c640 1 -- 172.21.15.114:0/2080038103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f332c025370 con 0x7f332c024090 2024-09-16T05:07:51.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 1 -- 172.21.15.114:0/2080038103 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33240b9070 con 0x7f332c024090 2024-09-16T05:07:51.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:51.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 1 -- 172.21.15.114:0/2080038103 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33240b4610 con 0x7f332c024090 2024-09-16T05:07:51.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:51.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 10 monclient: dump: 2024-09-16T05:07:51.510 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.510 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.510 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.510 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.510 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.510 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.510 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.510 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.510 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.511 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.438+0000 7f3329ffb640 1 -- 172.21.15.114:0/2080038103 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f33240b47b0 con 0x7f332c024090 2024-09-16T05:07:51.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.442+0000 7f332affd640 4 set_mon_vals no callback set 2024-09-16T05:07:51.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.442+0000 7f3329ffb640 1 --2- 172.21.15.114:0/2080038103 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3318037810 0x7f3318039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.442+0000 7f3329ffb640 1 -- 172.21.15.114:0/2080038103 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f33240e2fd0 con 0x7f332c024090 2024-09-16T05:07:51.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.442+0000 7f3329ffb640 10 client.4335.objecter ms_dispatch 0x7f3334000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:51.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.442+0000 7f3329ffb640 3 client.4335.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:51.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.442+0000 7f3329ffb640 3 client.4335.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:51.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.442+0000 7f3329ffb640 20 client.4335.objecter dump_active .. 0 homeless 2024-09-16T05:07:51.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.446+0000 7f333938a640 1 --2- 172.21.15.114:0/2080038103 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3318037810 0x7f3318039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.446+0000 7f333938a640 10 monclient: get_auth_request con 0x7f3318037810 auth_method 0 2024-09-16T05:07:51.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.446+0000 7f3329ffb640 10 client.4335.objecter ms_handle_connect 0x7f3318037810 2024-09-16T05:07:51.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.446+0000 7f333938a640 1 --2- 172.21.15.114:0/2080038103 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3318037810 0x7f3318039cd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f333000af30 tx=0x7f3330008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.450+0000 7f3329ffb640 1 -- 172.21.15.114:0/2080038103 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f3324016030 con 0x7f332c024090 2024-09-16T05:07:51.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.450+0000 7f3329ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.450+0000 7f3329ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:51.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.450+0000 7f03ca7fc640 1 -- 172.21.15.114:0/2803684399 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f03d0029070 con 0x7f03cc0225e0 2024-09-16T05:07:51.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.450+0000 7f03ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.450+0000 7f03ca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:51.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.522+0000 7f9f1f464640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T05:07:51.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.522+0000 7f9f1f464640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T05:07:51.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.522+0000 7f9f1f464640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.522+0000 7f9f1f464640 1 -- 172.21.15.114:0/880744628 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f9f1813e490 con 0x7f9f181376e0 2024-09-16T05:07:51.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.522+0000 7f9f0dffb640 1 -- 172.21.15.114:0/880744628 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9f1001a3b0 con 0x7f9f181376e0 2024-09-16T05:07:51.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.522+0000 7f9f0dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T05:07:51.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.522+0000 7f9f0dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:51.524 INFO:teuthology.orchestra.run.smithi114.stdout:51539607557 2024-09-16T05:07:51.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 10 librados: watch_flush enter 2024-09-16T05:07:51.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 10 librados: watch_flush exit 2024-09-16T05:07:51.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 20 client.4308.objecter shutdown clearing up homeless session... 2024-09-16T05:07:51.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 10 client.4308.objecter successfully canceled tick 2024-09-16T05:07:51.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 1 -- 172.21.15.114:0/880744628 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9efc037810 msgr2=0x7f9efc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 1 --2- 172.21.15.114:0/880744628 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9efc037810 0x7f9efc039cd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9f1813d7b0 tx=0x7f9f14008040 comp rx=0 tx=0).stop 2024-09-16T05:07:51.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 10 monclient: shutdown 2024-09-16T05:07:51.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:51.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 1 -- 172.21.15.114:0/880744628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f181376e0 msgr2=0x7f9f1813bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 1 --2- 172.21.15.114:0/880744628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f181376e0 0x7f9f1813bfa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000c730 tx=0x7f9f1000cbf0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 1 -- 172.21.15.114:0/880744628 shutdown_connections 2024-09-16T05:07:51.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 1 --2- 172.21.15.114:0/880744628 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9efc037810 0x7f9efc039cd0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 1 --2- 172.21.15.114:0/880744628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f181376e0 0x7f9f1813bfa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 1 --2- 172.21.15.114:0/880744628 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f181370a0 0x7f9f18136090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 1 --2- 172.21.15.114:0/880744628 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9f180a5d00 0x7f9f181cf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 1 -- 172.21.15.114:0/880744628 >> 172.21.15.114:0/880744628 conn(0x7f9f180a9a40 msgr2=0x7f9f180a7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:51.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 1 -- 172.21.15.114:0/880744628 shutdown_connections 2024-09-16T05:07:51.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 1 -- 172.21.15.114:0/880744628 wait complete. 2024-09-16T05:07:51.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.526+0000 7f9f1f464640 1 librados: shutdown 2024-09-16T05:07:51.541 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.5 2024-09-16T05:07:51.541 DEBUG:teuthology.parallel:result is None 2024-09-16T05:07:51.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T05:07:51.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T05:07:51.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 1 -- 172.21.15.114:0/2080038103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f332c0bf7b0 con 0x7f332c024090 2024-09-16T05:07:51.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f3329ffb640 1 -- 172.21.15.114:0/2080038103 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f33240c7a30 con 0x7f332c024090 2024-09-16T05:07:51.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f3329ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T05:07:51.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f3329ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:51.610 INFO:teuthology.orchestra.run.smithi114.stdout:47244640261 2024-09-16T05:07:51.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 10 librados: watch_flush enter 2024-09-16T05:07:51.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 10 librados: watch_flush exit 2024-09-16T05:07:51.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 20 client.4335.objecter shutdown clearing up homeless session... 2024-09-16T05:07:51.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 10 client.4335.objecter successfully canceled tick 2024-09-16T05:07:51.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 1 -- 172.21.15.114:0/2080038103 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3318037810 msgr2=0x7f3318039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 1 --2- 172.21.15.114:0/2080038103 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3318037810 0x7f3318039cd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f333000af30 tx=0x7f3330008040 comp rx=0 tx=0).stop 2024-09-16T05:07:51.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 10 monclient: shutdown 2024-09-16T05:07:51.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:51.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 1 -- 172.21.15.114:0/2080038103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332c024090 msgr2=0x7f332c0b7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 1 --2- 172.21.15.114:0/2080038103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332c024090 0x7f332c0b7a30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f33240ab000 tx=0x7f3324019150 comp rx=0 tx=0).stop 2024-09-16T05:07:51.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 1 -- 172.21.15.114:0/2080038103 shutdown_connections 2024-09-16T05:07:51.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 1 --2- 172.21.15.114:0/2080038103 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3318037810 0x7f3318039cd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 1 --2- 172.21.15.114:0/2080038103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332c024090 0x7f332c0b7a30 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f33240ab000 tx=0x7f3324019150 comp rx=0 tx=0).stop 2024-09-16T05:07:51.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 1 --2- 172.21.15.114:0/2080038103 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c0bb650 0x7f332c0bd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 1 --2- 172.21.15.114:0/2080038103 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f332c023cb0 0x7f332c0b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 1 -- 172.21.15.114:0/2080038103 >> 172.21.15.114:0/2080038103 conn(0x7f332c006320 msgr2=0x7f332c02b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 1 -- 172.21.15.114:0/2080038103 shutdown_connections 2024-09-16T05:07:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 1 -- 172.21.15.114:0/2080038103 wait complete. 2024-09-16T05:07:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.610+0000 7f333a38c640 1 librados: shutdown 2024-09-16T05:07:51.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.4 2024-09-16T05:07:51.628 DEBUG:teuthology.parallel:result is None 2024-09-16T05:07:51.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.638+0000 7f4981877640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T05:07:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.638+0000 7f4981877640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T05:07:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.638+0000 7f4981877640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.638+0000 7f4981877640 1 -- 172.21.15.114:0/1845599169 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f497c120580 con 0x7f497c0a5d00 2024-09-16T05:07:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.638+0000 7f4963fff640 1 -- 172.21.15.114:0/1845599169 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f496c013470 con 0x7f497c0a5d00 2024-09-16T05:07:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.638+0000 7f4963fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T05:07:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.638+0000 7f4963fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:51.640 INFO:teuthology.orchestra.run.smithi114.stdout:55834574852 2024-09-16T05:07:51.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 10 librados: watch_flush enter 2024-09-16T05:07:51.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 10 librados: watch_flush exit 2024-09-16T05:07:51.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 20 client.4259.objecter shutdown clearing up homeless session... 2024-09-16T05:07:51.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 10 client.4259.objecter successfully canceled tick 2024-09-16T05:07:51.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 1 -- 172.21.15.114:0/1845599169 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4968037810 msgr2=0x7f4968039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 1 --2- 172.21.15.114:0/1845599169 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4968037810 0x7f4968039cd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f497c13d380 tx=0x7f4970001aa0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 10 monclient: shutdown 2024-09-16T05:07:51.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:51.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 1 -- 172.21.15.114:0/1845599169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f497c0a5d00 msgr2=0x7f497c1cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 1 --2- 172.21.15.114:0/1845599169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f497c0a5d00 0x7f497c1cf2b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f496c00e930 tx=0x7f496c00edf0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 1 -- 172.21.15.114:0/1845599169 shutdown_connections 2024-09-16T05:07:51.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 1 --2- 172.21.15.114:0/1845599169 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4968037810 0x7f4968039cd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 1 --2- 172.21.15.114:0/1845599169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f497c1370a0 0x7f497c1cf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 1 --2- 172.21.15.114:0/1845599169 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f497c1376e0 0x7f497c13bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 1 --2- 172.21.15.114:0/1845599169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f497c0a5d00 0x7f497c1cf2b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 1 -- 172.21.15.114:0/1845599169 >> 172.21.15.114:0/1845599169 conn(0x7f497c0a9a40 msgr2=0x7f497c09f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:51.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 1 -- 172.21.15.114:0/1845599169 shutdown_connections 2024-09-16T05:07:51.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 1 -- 172.21.15.114:0/1845599169 wait complete. 2024-09-16T05:07:51.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.642+0000 7f4981877640 1 librados: shutdown 2024-09-16T05:07:51.663 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.0 2024-09-16T05:07:51.664 DEBUG:teuthology.parallel:result is None 2024-09-16T05:07:51.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.662+0000 7fd78b325640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T05:07:51.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.662+0000 7fd78b325640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T05:07:51.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.662+0000 7fd78b325640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:51.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.662+0000 7fd78b325640 1 -- 172.21.15.114:0/4083971554 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd77c025370 con 0x7fd77c023cb0 2024-09-16T05:07:51.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.662+0000 7fd77affd640 1 -- 172.21.15.114:0/4083971554 <== mon.1 v2:172.21.15.149:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd77c02c010 con 0x7fd77c023cb0 2024-09-16T05:07:51.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.662+0000 7fd77affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T05:07:51.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.662+0000 7fd77affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:51.665 INFO:teuthology.orchestra.run.smithi114.stdout:47244640261 2024-09-16T05:07:51.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 10 librados: watch_flush enter 2024-09-16T05:07:51.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 10 librados: watch_flush exit 2024-09-16T05:07:51.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 20 client.4330.objecter shutdown clearing up homeless session... 2024-09-16T05:07:51.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 10 client.4330.objecter successfully canceled tick 2024-09-16T05:07:51.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 1 -- 172.21.15.114:0/4083971554 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd768037840 msgr2=0x7fd768039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 1 --2- 172.21.15.114:0/4083971554 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd768037840 0x7fd768039d00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd77c026ff0 tx=0x7fd774009040 comp rx=0 tx=0).stop 2024-09-16T05:07:51.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 10 monclient: shutdown 2024-09-16T05:07:51.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:51.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 1 -- 172.21.15.114:0/4083971554 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd77c023cb0 msgr2=0x7fd77c153b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 1 --2- 172.21.15.114:0/4083971554 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd77c023cb0 0x7fd77c153b80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd780004000 tx=0x7fd7800190d0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 1 -- 172.21.15.114:0/4083971554 shutdown_connections 2024-09-16T05:07:51.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 1 --2- 172.21.15.114:0/4083971554 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd768037840 0x7fd768039d00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 1 --2- 172.21.15.114:0/4083971554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd77c024090 0x7fd77c0bae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 1 --2- 172.21.15.114:0/4083971554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd77c0244a0 0x7fd77c0bb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 1 --2- 172.21.15.114:0/4083971554 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd77c023cb0 0x7fd77c153b80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 1 -- 172.21.15.114:0/4083971554 >> 172.21.15.114:0/4083971554 conn(0x7fd77c006320 msgr2=0x7fd77c020dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:51.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 1 -- 172.21.15.114:0/4083971554 shutdown_connections 2024-09-16T05:07:51.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 1 -- 172.21.15.114:0/4083971554 wait complete. 2024-09-16T05:07:51.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.666+0000 7fd78b325640 1 librados: shutdown 2024-09-16T05:07:51.680 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.3 2024-09-16T05:07:51.680 DEBUG:teuthology.parallel:result is None 2024-09-16T05:07:51.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.718+0000 7fbcb005e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T05:07:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.718+0000 7fbcb005e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T05:07:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.718+0000 7fbcb005e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.718+0000 7fbcb005e640 1 -- 172.21.15.114:0/1594542851 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fbca806ee40 con 0x7fbca80a4270 2024-09-16T05:07:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.718+0000 7fbc9e7fc640 1 -- 172.21.15.114:0/1594542851 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbca4016070 con 0x7fbca80a4270 2024-09-16T05:07:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.718+0000 7fbc9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T05:07:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.718+0000 7fbc9e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:51.718 INFO:teuthology.orchestra.run.smithi114.stdout:51539607557 2024-09-16T05:07:51.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.718+0000 7fbcb005e640 10 librados: watch_flush enter 2024-09-16T05:07:51.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.718+0000 7fbcb005e640 10 librados: watch_flush exit 2024-09-16T05:07:51.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.718+0000 7fbcb005e640 20 client.4268.objecter shutdown clearing up homeless session... 2024-09-16T05:07:51.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.718+0000 7fbcb005e640 10 client.4268.objecter successfully canceled tick 2024-09-16T05:07:51.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.722+0000 7fbcb005e640 1 -- 172.21.15.114:0/1594542851 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbc8c037840 msgr2=0x7fbc8c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.722+0000 7fbcb005e640 1 --2- 172.21.15.114:0/1594542851 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbc8c037840 0x7fbc8c039d00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbca80a6c20 tx=0x7fbc98008040 comp rx=0 tx=0).stop 2024-09-16T05:07:51.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.722+0000 7fbcb005e640 10 monclient: shutdown 2024-09-16T05:07:51.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.722+0000 7fbcb005e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:51.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.722+0000 7fbcb005e640 1 -- 172.21.15.114:0/1594542851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbca80a4270 msgr2=0x7fbca809c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.722+0000 7fbcb005e640 1 --2- 172.21.15.114:0/1594542851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbca80a4270 0x7fbca809c630 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbca40ae040 tx=0x7fbca4019100 comp rx=0 tx=0).stop 2024-09-16T05:07:51.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.722+0000 7fbcb005e640 1 -- 172.21.15.114:0/1594542851 shutdown_connections 2024-09-16T05:07:51.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.722+0000 7fbcb005e640 1 --2- 172.21.15.114:0/1594542851 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbc8c037840 0x7fbc8c039d00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.722+0000 7fbcb005e640 1 --2- 172.21.15.114:0/1594542851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbca813b800 0x7fbca809d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.722+0000 7fbcb005e640 1 --2- 172.21.15.114:0/1594542851 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbca80a4650 0x7fbca809cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.722+0000 7fbcb005e640 1 --2- 172.21.15.114:0/1594542851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbca80a4270 0x7fbca809c630 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.722+0000 7fbcb005e640 1 -- 172.21.15.114:0/1594542851 >> 172.21.15.114:0/1594542851 conn(0x7fbca80a9a40 msgr2=0x7fbca80a8600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:51.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.722+0000 7fbcb005e640 1 -- 172.21.15.114:0/1594542851 shutdown_connections 2024-09-16T05:07:51.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.722+0000 7fbcb005e640 1 -- 172.21.15.114:0/1594542851 wait complete. 2024-09-16T05:07:51.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.722+0000 7fbcb005e640 1 librados: shutdown 2024-09-16T05:07:51.740 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.1 2024-09-16T05:07:51.740 DEBUG:teuthology.parallel:result is None 2024-09-16T05:07:51.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.742+0000 7f03daef7640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T05:07:51.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.742+0000 7f03daef7640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T05:07:51.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.742+0000 7f03daef7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:07:51.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.742+0000 7f03daef7640 1 -- 172.21.15.114:0/2803684399 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f03bc005380 con 0x7f03cc0225e0 2024-09-16T05:07:51.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03ca7fc640 1 -- 172.21.15.114:0/2803684399 <== mon.1 v2:172.21.15.149:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f03d000b070 con 0x7f03cc0225e0 2024-09-16T05:07:51.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T05:07:51.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03ca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:51.745 INFO:teuthology.orchestra.run.smithi114.stdout:55834574852 2024-09-16T05:07:51.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 10 librados: watch_flush enter 2024-09-16T05:07:51.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 10 librados: watch_flush exit 2024-09-16T05:07:51.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 20 client.4345.objecter shutdown clearing up homeless session... 2024-09-16T05:07:51.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 10 client.4345.objecter successfully canceled tick 2024-09-16T05:07:51.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 1 -- 172.21.15.114:0/2803684399 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03b8037810 msgr2=0x7f03b8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 1 --2- 172.21.15.114:0/2803684399 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03b8037810 0x7f03b8039cd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f03cc022cb0 tx=0x7f03c4008040 comp rx=0 tx=0).stop 2024-09-16T05:07:51.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 10 monclient: shutdown 2024-09-16T05:07:51.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:51.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 1 -- 172.21.15.114:0/2803684399 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f03cc0225e0 msgr2=0x7f03cc0b5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 1 --2- 172.21.15.114:0/2803684399 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f03cc0225e0 0x7f03cc0b5310 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f03d000df80 tx=0x7f03d0010490 comp rx=0 tx=0).stop 2024-09-16T05:07:51.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 1 -- 172.21.15.114:0/2803684399 shutdown_connections 2024-09-16T05:07:51.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 1 --2- 172.21.15.114:0/2803684399 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03b8037810 0x7f03b8039cd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 1 --2- 172.21.15.114:0/2803684399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03cc0b83c0 0x7f03cc0bb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 1 --2- 172.21.15.114:0/2803684399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03cc026140 0x7f03cc0b5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 1 --2- 172.21.15.114:0/2803684399 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f03cc0225e0 0x7f03cc0b5310 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 1 -- 172.21.15.114:0/2803684399 >> 172.21.15.114:0/2803684399 conn(0x7f03cc027400 msgr2=0x7f03cc0b6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:51.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 1 -- 172.21.15.114:0/2803684399 shutdown_connections 2024-09-16T05:07:51.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 1 -- 172.21.15.114:0/2803684399 wait complete. 2024-09-16T05:07:51.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.746+0000 7f03daef7640 1 librados: shutdown 2024-09-16T05:07:51.761 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.2 2024-09-16T05:07:51.761 DEBUG:teuthology.parallel:result is None 2024-09-16T05:07:51.762 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T05:07:51.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:07:51.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 10 monclient: get_monmap_and_config 2024-09-16T05:07:51.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 10 monclient: build_initial_monmap 2024-09-16T05:07:51.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 10 monclient: monmap: 2024-09-16T05:07:51.918 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:51.918 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.918 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:51.920945+0000 2024-09-16T05:07:51.918 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:51.920945+0000 2024-09-16T05:07:51.918 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:51.918 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.918 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:51.918 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:51.918 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:51.918 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:51.918 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:51.918 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 1 Processor -- start 2024-09-16T05:07:51.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 1 -- start start 2024-09-16T05:07:51.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 10 monclient: init 2024-09-16T05:07:51.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:51.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-16T05:07:51.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21900a0ec0 0x7f219009dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 10 monclient(hunting): picked mon.noname-b con 0x7f21900a0ec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:51.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f219009e000 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:51.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f2190139640 0x7f219013ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 10 monclient(hunting): picked mon.noname-c con 0x7f2190139640 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:51.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f219009a760 con 0x7f219009e000 2024-09-16T05:07:51.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f21900a12a0 con 0x7f2190139640 2024-09-16T05:07:51.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f21900994b0 con 0x7f21900a0ec0 2024-09-16T05:07:51.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:51.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f2196085640 10 monclient(hunting): authenticate will time out at 956.550016s 2024-09-16T05:07:51.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218effd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f219009e000 0x7f219009e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38674/0 (socket says 172.21.15.114:38674) 2024-09-16T05:07:51.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218effd640 1 -- 172.21.15.114:0/1389533528 learned_addr learned my addr 172.21.15.114:0/1389533528 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:51.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21900a0ec0 0x7f219009dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 10 monclient(hunting): get_auth_request con 0x7f21900a0ec0 auth_method 0 2024-09-16T05:07:51.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218ffff640 1 --2- 172.21.15.114:0/1389533528 >> v2:172.21.15.149:3300/0 conn(0x7f2190139640 0x7f219013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218e7fc640 1 -- 172.21.15.114:0/1389533528 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 546357355 0 0) 0x7f219009a760 con 0x7f219009e000 2024-09-16T05:07:51.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:07:51.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218e7fc640 10 monclient(hunting): my global_id is 4338 2024-09-16T05:07:51.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218e7fc640 1 -- 172.21.15.114:0/1389533528 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f217c003530 con 0x7f219009e000 2024-09-16T05:07:51.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218ffff640 10 monclient(hunting): get_auth_request con 0x7f2190139640 auth_method 0 2024-09-16T05:07:51.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 10 monclient(hunting): handle_auth_done global_id 4280 payload 274 2024-09-16T05:07:51.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 1 -- 172.21.15.114:0/1389533528 >> v2:172.21.15.149:3300/0 conn(0x7f2190139640 msgr2=0x7f219013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:51.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 1 --2- 172.21.15.114:0/1389533528 >> v2:172.21.15.149:3300/0 conn(0x7f2190139640 0x7f219013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 1 -- 172.21.15.114:0/1389533528 >> v1:172.21.15.114:6789/0 conn(0x7f219009e000 legacy=0x7f219009e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:07:51.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 1 monclient: found mon.noname-b 2024-09-16T05:07:51.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:51.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:51.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.918+0000 7f218f7fe640 1 -- 172.21.15.114:0/1389533528 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21900a1870 con 0x7f21900a0ec0 2024-09-16T05:07:51.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218f7fe640 1 --2- 172.21.15.114:0/1389533528 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21900a0ec0 0x7f219009dac0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2180009d90 tx=0x7f218001c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84edfc2f34d9467b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218e7fc640 1 -- 172.21.15.114:0/1389533528 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21800b2070 con 0x7f21900a0ec0 2024-09-16T05:07:51.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:51.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218e7fc640 10 monclient: dump: 2024-09-16T05:07:51.932 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.932 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.932 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.932 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.932 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.932 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.932 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.932 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.932 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.932 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:07:51.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:07:51.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:51.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 5 monclient: authenticate success, global_id 4280 2024-09-16T05:07:51.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218e7fc640 1 -- 172.21.15.114:0/1389533528 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21800a5890 con 0x7f21900a0ec0 2024-09-16T05:07:51.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:51.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:51.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:51.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 4 set_mon_vals no callback set 2024-09-16T05:07:51.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218e7fc640 1 -- 172.21.15.114:0/1389533528 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21800a5c40 con 0x7f21900a0ec0 2024-09-16T05:07:51.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:51.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218e7fc640 10 monclient: dump: 2024-09-16T05:07:51.935 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.935 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.935 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.935 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.935 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.936 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.936 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.936 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.936 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.936 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 10 monclient: shutdown 2024-09-16T05:07:51.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:51.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 -- 172.21.15.114:0/1389533528 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21900a0ec0 msgr2=0x7f219009dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 --2- 172.21.15.114:0/1389533528 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21900a0ec0 0x7f219009dac0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2180009d90 tx=0x7f218001c8c0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 -- 172.21.15.114:0/1389533528 shutdown_connections 2024-09-16T05:07:51.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 --2- 172.21.15.114:0/1389533528 >> v2:172.21.15.149:3300/0 conn(0x7f2190139640 0x7f219013ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 --2- 172.21.15.114:0/1389533528 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21900a0ec0 0x7f219009dac0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 -- 172.21.15.114:0/1389533528 >> 172.21.15.114:0/1389533528 conn(0x7f21900a1fb0 msgr2=0x7f21900a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:51.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 -- 172.21.15.114:0/1389533528 shutdown_connections 2024-09-16T05:07:51.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 -- 172.21.15.114:0/1389533528 wait complete. 2024-09-16T05:07:51.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 10 monclient: build_initial_monmap 2024-09-16T05:07:51.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 10 monclient: monmap: 2024-09-16T05:07:51.937 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:51.937 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:51.937 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:51.937 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:51.937 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:51.937 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.937 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:51.937 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:51.937 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:51.937 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 librados: starting msgr at 2024-09-16T05:07:51.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 librados: starting objecter 2024-09-16T05:07:51.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 Processor -- start 2024-09-16T05:07:51.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 -- start start 2024-09-16T05:07:51.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 librados: setting wanted keys 2024-09-16T05:07:51.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 librados: calling monclient init 2024-09-16T05:07:51.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 10 monclient: init 2024-09-16T05:07:51.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:51.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:07:51.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f219009e000 0x7f219012a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 10 monclient(hunting): picked mon.noname-b con 0x7f219009e000 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:51.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21900a0ec0 0x7f219012aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 10 monclient(hunting): picked mon.noname-c con 0x7f21900a0ec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:51.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2190139640 0x7f219012aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 10 monclient(hunting): picked mon.noname-a con 0x7f2190139640 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:51.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f219009cb30 con 0x7f2190139640 2024-09-16T05:07:51.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f219009c9b0 con 0x7f219009e000 2024-09-16T05:07:51.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:51.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f219009ccb0 con 0x7f21900a0ec0 2024-09-16T05:07:51.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:51.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2196085640 10 monclient(hunting): authenticate will time out at 956.555358s 2024-09-16T05:07:51.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2190139640 0x7f219012aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2190139640 0x7f219012aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55044/0 (socket says 172.21.15.114:55044) 2024-09-16T05:07:51.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 1 -- 172.21.15.114:0/1699006153 learned_addr learned my addr 172.21.15.114:0/1699006153 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:51.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 10 monclient(hunting): get_auth_request con 0x7f2190139640 auth_method 0 2024-09-16T05:07:51.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218effd640 1 --2- 172.21.15.114:0/1699006153 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21900a0ec0 0x7f219012aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218effd640 10 monclient(hunting): get_auth_request con 0x7f21900a0ec0 auth_method 0 2024-09-16T05:07:51.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:51.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:51.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:51.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:51.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:51.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:51.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-09-16T05:07:51.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 1 -- 172.21.15.114:0/1699006153 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21900a0ec0 msgr2=0x7f219012aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 1 --2- 172.21.15.114:0/1699006153 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21900a0ec0 0x7f219012aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 1 -- 172.21.15.114:0/1699006153 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f219009e000 msgr2=0x7f219012a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:51.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 1 --2- 172.21.15.114:0/1699006153 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f219009e000 0x7f219012a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:51.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:07:51.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 1 monclient: found mon.noname-a 2024-09-16T05:07:51.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:51.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 1 -- 172.21.15.114:0/1699006153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f219012dbc0 con 0x7f2190139640 2024-09-16T05:07:51.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218effd640 1 --2- 172.21.15.114:0/1699006153 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21900a0ec0 0x7f219012aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:51.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f218ffff640 1 --2- 172.21.15.114:0/1699006153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2190139640 0x7f219012aff0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f218800d800 tx=0x7f218800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2187fff640 10 client.?.objecter ms_handle_connect 0x7f2190139640 2024-09-16T05:07:51.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.922+0000 7f2187fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:51.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 1 -- 172.21.15.114:0/1699006153 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f218800b660 con 0x7f2190139640 2024-09-16T05:07:51.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:51.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 10 monclient: dump: 2024-09-16T05:07:51.942 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.942 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.942 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.942 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.942 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.942 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.942 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.942 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.942 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.942 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 10 monclient: _finish_auth 0 2024-09-16T05:07:51.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 10 monclient: _check_auth_tickets 2024-09-16T05:07:51.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:51.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 1 -- 172.21.15.114:0/1699006153 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2188019070 con 0x7f2190139640 2024-09-16T05:07:51.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:51.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 1 -- 172.21.15.114:0/1699006153 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f218800ba40 con 0x7f2190139640 2024-09-16T05:07:51.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:51.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:51.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 10 monclient: dump: 2024-09-16T05:07:51.943 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:51.943 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:51.943 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.943 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:51.943 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:51.943 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:51.944 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:51.944 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:51.944 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:51.944 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:51.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f218dffb640 4 set_mon_vals no callback set 2024-09-16T05:07:51.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2196085640 5 monclient: authenticate success, global_id 4341 2024-09-16T05:07:51.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2196085640 10 monclient: _renew_subs 2024-09-16T05:07:51.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2196085640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2196085640 1 -- 172.21.15.114:0/1699006153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21900a3b80 con 0x7f2190139640 2024-09-16T05:07:51.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2196085640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:51.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2196085640 10 monclient: _renew_subs 2024-09-16T05:07:51.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2196085640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2196085640 1 -- 172.21.15.114:0/1699006153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f219009c6a0 con 0x7f2190139640 2024-09-16T05:07:51.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2196085640 1 librados: init done 2024-09-16T05:07:51.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 1 -- 172.21.15.114:0/1699006153 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f218800bbe0 con 0x7f2190139640 2024-09-16T05:07:51.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2196085640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2196085640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2196085640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:51.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2196085640 1 -- 172.21.15.114:0/1699006153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f219009e3e0 con 0x7f2190139640 2024-09-16T05:07:51.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 1 --2- 172.21.15.114:0/1699006153 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2170037810 0x7f2170039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:51.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 1 -- 172.21.15.114:0/1699006153 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f2188048a60 con 0x7f2190139640 2024-09-16T05:07:51.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 10 client.4341.objecter ms_dispatch 0x7f2190000ba0 osd_map(18..18 src has 1..18) 2024-09-16T05:07:51.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 3 client.4341.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T05:07:51.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 3 client.4341.objecter handle_osd_map decoding full epoch 18 2024-09-16T05:07:51.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 20 client.4341.objecter dump_active .. 0 homeless 2024-09-16T05:07:51.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f218f7fe640 1 --2- 172.21.15.114:0/1699006153 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2170037810 0x7f2170039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:51.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f218f7fe640 10 monclient: get_auth_request con 0x7f2170037810 auth_method 0 2024-09-16T05:07:51.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f2187fff640 10 client.4341.objecter ms_handle_connect 0x7f2170037810 2024-09-16T05:07:51.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.926+0000 7f218f7fe640 1 --2- 172.21.15.114:0/1699006153 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2170037810 0x7f2170039cd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f21800af000 tx=0x7f2180004ac0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:51.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.930+0000 7f2187fff640 1 -- 172.21.15.114:0/1699006153 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f218801e8e0 con 0x7f2190139640 2024-09-16T05:07:51.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.930+0000 7f2187fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:51.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:51.930+0000 7f2187fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:52.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.082+0000 7f2196085640 1 -- 172.21.15.114:0/1699006153 --> v2:172.21.15.149:6812/1185531135 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f219012c430 con 0x7f2170037810 2024-09-16T05:07:52.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2187fff640 1 -- 172.21.15.114:0/1699006153 <== mgr.4108 v2:172.21.15.149:6812/1185531135 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34148 (secure 0 0 0) 0x7f219012c430 con 0x7f2170037810 2024-09-16T05:07:52.087 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:07:52.087 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-16T05:07:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 10 librados: watch_flush enter 2024-09-16T05:07:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 10 librados: watch_flush exit 2024-09-16T05:07:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 20 client.4341.objecter shutdown clearing up homeless session... 2024-09-16T05:07:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 10 client.4341.objecter successfully canceled tick 2024-09-16T05:07:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 1 -- 172.21.15.114:0/1699006153 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2170037810 msgr2=0x7f2170039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 1 --2- 172.21.15.114:0/1699006153 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2170037810 0x7f2170039cd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f21800af000 tx=0x7f2180004ac0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 10 monclient: shutdown 2024-09-16T05:07:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 1 -- 172.21.15.114:0/1699006153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2190139640 msgr2=0x7f219012aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 1 --2- 172.21.15.114:0/1699006153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2190139640 0x7f219012aff0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f218800d800 tx=0x7f218800dcc0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 1 -- 172.21.15.114:0/1699006153 shutdown_connections 2024-09-16T05:07:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 1 --2- 172.21.15.114:0/1699006153 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2170037810 0x7f2170039cd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 1 --2- 172.21.15.114:0/1699006153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2190139640 0x7f219012aff0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 1 --2- 172.21.15.114:0/1699006153 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21900a0ec0 0x7f219012aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 1 --2- 172.21.15.114:0/1699006153 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f219009e000 0x7f219012a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 1 -- 172.21.15.114:0/1699006153 >> 172.21.15.114:0/1699006153 conn(0x7f21900a1fb0 msgr2=0x7f2190138b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.086+0000 7f2196085640 1 -- 172.21.15.114:0/1699006153 shutdown_connections 2024-09-16T05:07:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.090+0000 7f2196085640 1 -- 172.21.15.114:0/1699006153 wait complete. 2024-09-16T05:07:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.090+0000 7f2196085640 1 librados: shutdown 2024-09-16T05:07:52.103 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T05:07:52.071427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164156,"kb_used_data":1960,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066884,"statfs":{"total":579820584960,"available":579652489216,"internally_reserved":0,"allocated":2007040,"data_stored":1350460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.835801"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.267914+0000","last_change":"2024-09-16T05:07:48.267914+0000","last_active":"2024-09-16T05:07:48.267914+0000","last_peered":"2024-09-16T05:07:48.267914+0000","last_clean":"2024-09-16T05:07:48.267914+0000","last_became_active":"2024-09-16T05:07:46.254034+0000","last_became_peered":"2024-09-16T05:07:46.254034+0000","last_unstale":"2024-09-16T05:07:48.267914+0000","last_undegraded":"2024-09-16T05:07:48.267914+0000","last_fullsized":"2024-09-16T05:07:48.267914+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:57:37.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014490219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.275161+0000","last_change":"2024-09-16T05:07:48.275545+0000","last_active":"2024-09-16T05:07:48.275161+0000","last_peered":"2024-09-16T05:07:48.275161+0000","last_clean":"2024-09-16T05:07:48.275161+0000","last_became_active":"2024-09-16T05:07:46.252207+0000","last_became_peered":"2024-09-16T05:07:46.252207+0000","last_unstale":"2024-09-16T05:07:48.275161+0000","last_undegraded":"2024-09-16T05:07:48.275161+0000","last_fullsized":"2024-09-16T05:07:48.275161+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:36:08.378166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073333700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.267372+0000","last_change":"2024-09-16T05:07:48.267372+0000","last_active":"2024-09-16T05:07:48.267372+0000","last_peered":"2024-09-16T05:07:48.267372+0000","last_clean":"2024-09-16T05:07:48.267372+0000","last_became_active":"2024-09-16T05:07:46.243891+0000","last_became_peered":"2024-09-16T05:07:46.243891+0000","last_unstale":"2024-09-16T05:07:48.267372+0000","last_undegraded":"2024-09-16T05:07:48.267372+0000","last_fullsized":"2024-09-16T05:07:48.267372+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:48:25.061723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011829709999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.352028+0000","last_change":"2024-09-16T05:07:48.352028+0000","last_active":"2024-09-16T05:07:48.352028+0000","last_peered":"2024-09-16T05:07:48.352028+0000","last_clean":"2024-09-16T05:07:48.352028+0000","last_became_active":"2024-09-16T05:07:46.255633+0000","last_became_peered":"2024-09-16T05:07:46.255633+0000","last_unstale":"2024-09-16T05:07:48.352028+0000","last_undegraded":"2024-09-16T05:07:48.352028+0000","last_fullsized":"2024-09-16T05:07:48.352028+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:07.079219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000883379,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.275745+0000","last_change":"2024-09-16T05:07:48.274329+0000","last_active":"2024-09-16T05:07:48.275745+0000","last_peered":"2024-09-16T05:07:48.275745+0000","last_clean":"2024-09-16T05:07:48.275745+0000","last_became_active":"2024-09-16T05:07:46.257118+0000","last_became_peered":"2024-09-16T05:07:46.257118+0000","last_unstale":"2024-09-16T05:07:48.275745+0000","last_undegraded":"2024-09-16T05:07:48.275745+0000","last_fullsized":"2024-09-16T05:07:48.275745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:35.932772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020216650000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.263044+0000","last_change":"2024-09-16T05:07:48.263513+0000","last_active":"2024-09-16T05:07:48.263044+0000","last_peered":"2024-09-16T05:07:48.263044+0000","last_clean":"2024-09-16T05:07:48.263044+0000","last_became_active":"2024-09-16T05:07:46.252254+0000","last_became_peered":"2024-09-16T05:07:46.252254+0000","last_unstale":"2024-09-16T05:07:48.263044+0000","last_undegraded":"2024-09-16T05:07:48.263044+0000","last_fullsized":"2024-09-16T05:07:48.263044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:33.736560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081560699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.266504+0000","last_change":"2024-09-16T05:07:48.266504+0000","last_active":"2024-09-16T05:07:48.266504+0000","last_peered":"2024-09-16T05:07:48.266504+0000","last_clean":"2024-09-16T05:07:48.266504+0000","last_became_active":"2024-09-16T05:07:46.254322+0000","last_became_peered":"2024-09-16T05:07:46.254322+0000","last_unstale":"2024-09-16T05:07:48.266504+0000","last_undegraded":"2024-09-16T05:07:48.266504+0000","last_fullsized":"2024-09-16T05:07:48.266504+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:00.175590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093680599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.267201+0000","last_change":"2024-09-16T05:07:48.267201+0000","last_active":"2024-09-16T05:07:48.267201+0000","last_peered":"2024-09-16T05:07:48.267201+0000","last_clean":"2024-09-16T05:07:48.267201+0000","last_became_active":"2024-09-16T05:07:46.246409+0000","last_became_peered":"2024-09-16T05:07:46.246409+0000","last_unstale":"2024-09-16T05:07:48.267201+0000","last_undegraded":"2024-09-16T05:07:48.267201+0000","last_fullsized":"2024-09-16T05:07:48.267201+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:13.808827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010413149999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.265942+0000","last_change":"2024-09-16T05:07:39.580660+0000","last_active":"2024-09-16T05:07:48.265942+0000","last_peered":"2024-09-16T05:07:48.265942+0000","last_clean":"2024-09-16T05:07:48.265942+0000","last_became_active":"2024-09-16T05:07:39.576756+0000","last_became_peered":"2024-09-16T05:07:39.576756+0000","last_unstale":"2024-09-16T05:07:48.265942+0000","last_undegraded":"2024-09-16T05:07:48.265942+0000","last_fullsized":"2024-09-16T05:07:48.265942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_clean_scrub_stamp":"2024-09-16T05:07:38.167879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:57.060906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":48761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":48780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":688128,"data_stored":577689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":688128,"data_stored":577689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":159744,"data_stored":48780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":48761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:07:52.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:07:52.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 10 monclient: get_monmap_and_config 2024-09-16T05:07:52.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 10 monclient: build_initial_monmap 2024-09-16T05:07:52.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 10 monclient: monmap: 2024-09-16T05:07:52.219 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:52.219 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:52.219 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:52.222708+0000 2024-09-16T05:07:52.220 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:52.222708+0000 2024-09-16T05:07:52.220 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:52.220 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.220 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:52.220 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:52.220 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:52.220 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:52.220 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:52.220 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 1 Processor -- start 2024-09-16T05:07:52.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 1 -- start start 2024-09-16T05:07:52.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 10 monclient: init 2024-09-16T05:07:52.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:52.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T05:07:52.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f195809b6c0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19580a15d0 0x7f195813a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 10 monclient(hunting): picked mon.noname-b con 0x7f19580a15d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19580a19e0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f195809d4e0 con 0x7f195809b6c0 2024-09-16T05:07:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f195809f430 con 0x7f19580a19e0 2024-09-16T05:07:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f195809e390 con 0x7f19580a15d0 2024-09-16T05:07:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.218+0000 7f195d456640 10 monclient(hunting): authenticate will time out at 956.851943s 2024-09-16T05:07:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19580a15d0 0x7f195813a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1956ffd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f195809b6c0 0x7f19580a0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38676/0 (socket says 172.21.15.114:38676) 2024-09-16T05:07:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1956ffd640 1 -- 172.21.15.114:0/841376290 learned_addr learned my addr 172.21.15.114:0/841376290 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 10 monclient(hunting): get_auth_request con 0x7f19580a15d0 auth_method 0 2024-09-16T05:07:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 10 monclient(hunting): handle_auth_done global_id 4286 payload 274 2024-09-16T05:07:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1955ffb640 1 -- 172.21.15.114:0/841376290 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1211750008 0 0) 0x7f195809d4e0 con 0x7f195809b6c0 2024-09-16T05:07:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 1 -- 172.21.15.114:0/841376290 >> v1:172.21.15.149:6789/0 conn(0x7f19580a19e0 legacy=0x7f195813cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 1 -- 172.21.15.114:0/841376290 >> v1:172.21.15.114:6789/0 conn(0x7f195809b6c0 legacy=0x7f19580a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 10 monclient: _finish_hunting 0 2024-09-16T05:07:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 1 monclient: found mon.noname-b 2024-09-16T05:07:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:52.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:52.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 1 -- 172.21.15.114:0/841376290 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195809c2f0 con 0x7f19580a15d0 2024-09-16T05:07:52.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1955ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:07:52.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f19567fc640 1 --2- 172.21.15.114:0/841376290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19580a15d0 0x7f195813a650 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1950009e10 tx=0x7f195001c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=571032c5f4ab5f23 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:52.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1955ffb640 1 -- 172.21.15.114:0/841376290 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19500b2070 con 0x7f19580a15d0 2024-09-16T05:07:52.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1955ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:52.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1955ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:52.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1955ffb640 10 monclient: dump: 2024-09-16T05:07:52.227 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:52.227 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:52.227 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.227 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.227 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:52.227 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.227 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:52.227 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:52.227 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:52.228 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1955ffb640 10 monclient: _finish_auth 0 2024-09-16T05:07:52.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1955ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:07:52.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1955ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:52.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1955ffb640 1 -- 172.21.15.114:0/841376290 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19500a5730 con 0x7f19580a15d0 2024-09-16T05:07:52.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1955ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:52.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1955ffb640 1 -- 172.21.15.114:0/841376290 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19500a5b00 con 0x7f19580a15d0 2024-09-16T05:07:52.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1955ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:52.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1955ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:52.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f1955ffb640 10 monclient: dump: 2024-09-16T05:07:52.229 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:52.229 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:52.229 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.229 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.229 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:52.229 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.229 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:52.229 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:52.229 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:52.230 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f195d456640 5 monclient: authenticate success, global_id 4286 2024-09-16T05:07:52.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f195d456640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:52.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.222+0000 7f195d456640 4 set_mon_vals no callback set 2024-09-16T05:07:52.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient: shutdown 2024-09-16T05:07:52.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:52.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 -- 172.21.15.114:0/841376290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19580a15d0 msgr2=0x7f195813a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 --2- 172.21.15.114:0/841376290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19580a15d0 0x7f195813a650 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1950009e10 tx=0x7f195001c870 comp rx=0 tx=0).stop 2024-09-16T05:07:52.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 -- 172.21.15.114:0/841376290 shutdown_connections 2024-09-16T05:07:52.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 --2- 172.21.15.114:0/841376290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19580a15d0 0x7f195813a650 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 -- 172.21.15.114:0/841376290 >> 172.21.15.114:0/841376290 conn(0x7f19580a1fb0 msgr2=0x7f19580a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:52.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 -- 172.21.15.114:0/841376290 shutdown_connections 2024-09-16T05:07:52.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 -- 172.21.15.114:0/841376290 wait complete. 2024-09-16T05:07:52.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient: build_initial_monmap 2024-09-16T05:07:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient: monmap: 2024-09-16T05:07:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:52.231 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 librados: starting msgr at 2024-09-16T05:07:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 librados: starting objecter 2024-09-16T05:07:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 Processor -- start 2024-09-16T05:07:52.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 -- start start 2024-09-16T05:07:52.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 librados: setting wanted keys 2024-09-16T05:07:52.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 librados: calling monclient init 2024-09-16T05:07:52.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient: init 2024-09-16T05:07:52.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:52.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:07:52.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f195809b6c0 0x7f195813dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient(hunting): picked mon.noname-c con 0x7f195809b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:52.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19580a15d0 0x7f1958091830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient(hunting): picked mon.noname-b con 0x7f19580a15d0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:52.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19580a19e0 0x7f1958091d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient(hunting): picked mon.noname-a con 0x7f19580a19e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:52.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:52.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1958140c60 con 0x7f19580a19e0 2024-09-16T05:07:52.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:52.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f1958140ae0 con 0x7f19580a15d0 2024-09-16T05:07:52.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:52.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1958140de0 con 0x7f195809b6c0 2024-09-16T05:07:52.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:52.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient(hunting): authenticate will time out at 956.858629s 2024-09-16T05:07:52.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f195809b6c0 0x7f195813dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:52.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f19577fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19580a19e0 0x7f1958091d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:52.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f195809b6c0 0x7f195813dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55738/0 (socket says 172.21.15.114:55738) 2024-09-16T05:07:52.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 1 -- 172.21.15.114:0/1916121950 learned_addr learned my addr 172.21.15.114:0/1916121950 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:52.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 10 monclient(hunting): get_auth_request con 0x7f195809b6c0 auth_method 0 2024-09-16T05:07:52.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:52.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:52.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:52.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f19577fe640 10 monclient(hunting): get_auth_request con 0x7f19580a19e0 auth_method 0 2024-09-16T05:07:52.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f19577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:52.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f19577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:52.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f19577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:52.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:52.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:52.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:52.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f19577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:52.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f19577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:52.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f19577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:52.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f19567fc640 1 --2- 172.21.15.114:0/1916121950 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19580a15d0 0x7f1958091830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:52.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2024-09-16T05:07:52.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 1 -- 172.21.15.114:0/1916121950 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19580a15d0 msgr2=0x7f1958091830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 1 --2- 172.21.15.114:0/1916121950 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19580a15d0 0x7f1958091830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 1 -- 172.21.15.114:0/1916121950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19580a19e0 msgr2=0x7f1958091d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 1 --2- 172.21.15.114:0/1916121950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19580a19e0 0x7f1958091d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:07:52.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 1 monclient: found mon.noname-c 2024-09-16T05:07:52.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:52.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:52.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 1 -- 172.21.15.114:0/1916121950 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1958093d10 con 0x7f195809b6c0 2024-09-16T05:07:52.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f19567fc640 10 monclient: get_auth_request con 0x7f19580a15d0 auth_method 0 2024-09-16T05:07:52.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f19567fc640 1 --2- 172.21.15.114:0/1916121950 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19580a15d0 0x7f1958091830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:07:52.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f19577fe640 1 --2- 172.21.15.114:0/1916121950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19580a19e0 0x7f1958091d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:52.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1956ffd640 1 --2- 172.21.15.114:0/1916121950 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f195809b6c0 0x7f195813dbc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f194c00c4b0 tx=0x7f194c00c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:52.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f193f7fe640 10 client.?.objecter ms_handle_connect 0x7f195809b6c0 2024-09-16T05:07:52.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f193f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:52.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f193f7fe640 1 -- 172.21.15.114:0/1916121950 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c018070 con 0x7f195809b6c0 2024-09-16T05:07:52.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f193f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:52.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f193f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:52.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f193f7fe640 10 monclient: dump: 2024-09-16T05:07:52.241 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:52.241 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:52.241 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.241 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.241 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:52.241 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.241 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:52.241 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:52.241 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:52.241 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f193f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:07:52.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f193f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:07:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f193f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f193f7fe640 1 -- 172.21.15.114:0/1916121950 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f194c007500 con 0x7f195809b6c0 2024-09-16T05:07:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f193f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f193f7fe640 1 -- 172.21.15.114:0/1916121950 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c004a30 con 0x7f195809b6c0 2024-09-16T05:07:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 5 monclient: authenticate success, global_id 4289 2024-09-16T05:07:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f193f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f193f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f193f7fe640 10 monclient: dump: 2024-09-16T05:07:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:52.243 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient: _renew_subs 2024-09-16T05:07:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f1954ff9640 4 set_mon_vals no callback set 2024-09-16T05:07:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 -- 172.21.15.114:0/1916121950 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1958095440 con 0x7f195809b6c0 2024-09-16T05:07:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient: _renew_subs 2024-09-16T05:07:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 -- 172.21.15.114:0/1916121950 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f195806deb0 con 0x7f195809b6c0 2024-09-16T05:07:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.226+0000 7f195d456640 1 librados: init done 2024-09-16T05:07:52.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.230+0000 7f195d456640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.230+0000 7f195d456640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.230+0000 7f195d456640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.230+0000 7f195d456640 1 -- 172.21.15.114:0/1916121950 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f195809c680 con 0x7f195809b6c0 2024-09-16T05:07:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.230+0000 7f193f7fe640 1 -- 172.21.15.114:0/1916121950 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f194c004bd0 con 0x7f195809b6c0 2024-09-16T05:07:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.230+0000 7f193f7fe640 1 --2- 172.21.15.114:0/1916121950 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1944037810 0x7f1944039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.230+0000 7f193f7fe640 1 -- 172.21.15.114:0/1916121950 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f194c047860 con 0x7f195809b6c0 2024-09-16T05:07:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.230+0000 7f193f7fe640 10 client.4289.objecter ms_dispatch 0x7f1958000ba0 osd_map(19..19 src has 1..19) 2024-09-16T05:07:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.230+0000 7f193f7fe640 3 client.4289.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T05:07:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.230+0000 7f193f7fe640 3 client.4289.objecter handle_osd_map decoding full epoch 19 2024-09-16T05:07:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.230+0000 7f193f7fe640 20 client.4289.objecter dump_active .. 0 homeless 2024-09-16T05:07:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.230+0000 7f19567fc640 1 --2- 172.21.15.114:0/1916121950 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1944037810 0x7f1944039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.230+0000 7f19567fc640 10 monclient: get_auth_request con 0x7f1944037810 auth_method 0 2024-09-16T05:07:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.230+0000 7f193f7fe640 10 client.4289.objecter ms_handle_connect 0x7f1944037810 2024-09-16T05:07:52.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.230+0000 7f19567fc640 1 --2- 172.21.15.114:0/1916121950 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1944037810 0x7f1944039cd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f19500af000 tx=0x7f19500a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:52.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.234+0000 7f193f7fe640 1 -- 172.21.15.114:0/1916121950 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f194c011360 con 0x7f195809b6c0 2024-09-16T05:07:52.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.234+0000 7f193f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:52.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.234+0000 7f193f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:52.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.382+0000 7f195d456640 1 -- 172.21.15.114:0/1916121950 --> v2:172.21.15.149:6812/1185531135 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1958095680 con 0x7f1944037810 2024-09-16T05:07:52.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f193f7fe640 1 -- 172.21.15.114:0/1916121950 <== mgr.4108 v2:172.21.15.149:6812/1185531135 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34148 (secure 0 0 0) 0x7f1958095680 con 0x7f1944037810 2024-09-16T05:07:52.385 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:07:52.386 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-16T05:07:52.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 10 librados: watch_flush enter 2024-09-16T05:07:52.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 10 librados: watch_flush exit 2024-09-16T05:07:52.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 20 client.4289.objecter shutdown clearing up homeless session... 2024-09-16T05:07:52.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 10 client.4289.objecter successfully canceled tick 2024-09-16T05:07:52.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 1 -- 172.21.15.114:0/1916121950 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1944037810 msgr2=0x7f1944039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 1 --2- 172.21.15.114:0/1916121950 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1944037810 0x7f1944039cd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f19500af000 tx=0x7f19500a7070 comp rx=0 tx=0).stop 2024-09-16T05:07:52.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 10 monclient: shutdown 2024-09-16T05:07:52.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:52.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 1 -- 172.21.15.114:0/1916121950 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f195809b6c0 msgr2=0x7f195813dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 1 --2- 172.21.15.114:0/1916121950 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f195809b6c0 0x7f195813dbc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f194c00c4b0 tx=0x7f194c00c970 comp rx=0 tx=0).stop 2024-09-16T05:07:52.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 1 -- 172.21.15.114:0/1916121950 shutdown_connections 2024-09-16T05:07:52.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 1 --2- 172.21.15.114:0/1916121950 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1944037810 0x7f1944039cd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 1 --2- 172.21.15.114:0/1916121950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19580a19e0 0x7f1958091d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 1 --2- 172.21.15.114:0/1916121950 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19580a15d0 0x7f1958091830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 1 --2- 172.21.15.114:0/1916121950 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f195809b6c0 0x7f195813dbc0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 1 -- 172.21.15.114:0/1916121950 >> 172.21.15.114:0/1916121950 conn(0x7f19580a1fb0 msgr2=0x7f1958072340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:52.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 1 -- 172.21.15.114:0/1916121950 shutdown_connections 2024-09-16T05:07:52.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 1 -- 172.21.15.114:0/1916121950 wait complete. 2024-09-16T05:07:52.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.386+0000 7f195d456640 1 librados: shutdown 2024-09-16T05:07:52.402 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T05:07:52.071427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164156,"kb_used_data":1960,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066884,"statfs":{"total":579820584960,"available":579652489216,"internally_reserved":0,"allocated":2007040,"data_stored":1350460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.835801"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.267914+0000","last_change":"2024-09-16T05:07:48.267914+0000","last_active":"2024-09-16T05:07:48.267914+0000","last_peered":"2024-09-16T05:07:48.267914+0000","last_clean":"2024-09-16T05:07:48.267914+0000","last_became_active":"2024-09-16T05:07:46.254034+0000","last_became_peered":"2024-09-16T05:07:46.254034+0000","last_unstale":"2024-09-16T05:07:48.267914+0000","last_undegraded":"2024-09-16T05:07:48.267914+0000","last_fullsized":"2024-09-16T05:07:48.267914+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:57:37.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014490219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.275161+0000","last_change":"2024-09-16T05:07:48.275545+0000","last_active":"2024-09-16T05:07:48.275161+0000","last_peered":"2024-09-16T05:07:48.275161+0000","last_clean":"2024-09-16T05:07:48.275161+0000","last_became_active":"2024-09-16T05:07:46.252207+0000","last_became_peered":"2024-09-16T05:07:46.252207+0000","last_unstale":"2024-09-16T05:07:48.275161+0000","last_undegraded":"2024-09-16T05:07:48.275161+0000","last_fullsized":"2024-09-16T05:07:48.275161+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:36:08.378166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073333700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.267372+0000","last_change":"2024-09-16T05:07:48.267372+0000","last_active":"2024-09-16T05:07:48.267372+0000","last_peered":"2024-09-16T05:07:48.267372+0000","last_clean":"2024-09-16T05:07:48.267372+0000","last_became_active":"2024-09-16T05:07:46.243891+0000","last_became_peered":"2024-09-16T05:07:46.243891+0000","last_unstale":"2024-09-16T05:07:48.267372+0000","last_undegraded":"2024-09-16T05:07:48.267372+0000","last_fullsized":"2024-09-16T05:07:48.267372+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:48:25.061723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011829709999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.352028+0000","last_change":"2024-09-16T05:07:48.352028+0000","last_active":"2024-09-16T05:07:48.352028+0000","last_peered":"2024-09-16T05:07:48.352028+0000","last_clean":"2024-09-16T05:07:48.352028+0000","last_became_active":"2024-09-16T05:07:46.255633+0000","last_became_peered":"2024-09-16T05:07:46.255633+0000","last_unstale":"2024-09-16T05:07:48.352028+0000","last_undegraded":"2024-09-16T05:07:48.352028+0000","last_fullsized":"2024-09-16T05:07:48.352028+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:07.079219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000883379,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.275745+0000","last_change":"2024-09-16T05:07:48.274329+0000","last_active":"2024-09-16T05:07:48.275745+0000","last_peered":"2024-09-16T05:07:48.275745+0000","last_clean":"2024-09-16T05:07:48.275745+0000","last_became_active":"2024-09-16T05:07:46.257118+0000","last_became_peered":"2024-09-16T05:07:46.257118+0000","last_unstale":"2024-09-16T05:07:48.275745+0000","last_undegraded":"2024-09-16T05:07:48.275745+0000","last_fullsized":"2024-09-16T05:07:48.275745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:35.932772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020216650000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.263044+0000","last_change":"2024-09-16T05:07:48.263513+0000","last_active":"2024-09-16T05:07:48.263044+0000","last_peered":"2024-09-16T05:07:48.263044+0000","last_clean":"2024-09-16T05:07:48.263044+0000","last_became_active":"2024-09-16T05:07:46.252254+0000","last_became_peered":"2024-09-16T05:07:46.252254+0000","last_unstale":"2024-09-16T05:07:48.263044+0000","last_undegraded":"2024-09-16T05:07:48.263044+0000","last_fullsized":"2024-09-16T05:07:48.263044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:33.736560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081560699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.266504+0000","last_change":"2024-09-16T05:07:48.266504+0000","last_active":"2024-09-16T05:07:48.266504+0000","last_peered":"2024-09-16T05:07:48.266504+0000","last_clean":"2024-09-16T05:07:48.266504+0000","last_became_active":"2024-09-16T05:07:46.254322+0000","last_became_peered":"2024-09-16T05:07:46.254322+0000","last_unstale":"2024-09-16T05:07:48.266504+0000","last_undegraded":"2024-09-16T05:07:48.266504+0000","last_fullsized":"2024-09-16T05:07:48.266504+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:00.175590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093680599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.267201+0000","last_change":"2024-09-16T05:07:48.267201+0000","last_active":"2024-09-16T05:07:48.267201+0000","last_peered":"2024-09-16T05:07:48.267201+0000","last_clean":"2024-09-16T05:07:48.267201+0000","last_became_active":"2024-09-16T05:07:46.246409+0000","last_became_peered":"2024-09-16T05:07:46.246409+0000","last_unstale":"2024-09-16T05:07:48.267201+0000","last_undegraded":"2024-09-16T05:07:48.267201+0000","last_fullsized":"2024-09-16T05:07:48.267201+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:13.808827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010413149999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.265942+0000","last_change":"2024-09-16T05:07:39.580660+0000","last_active":"2024-09-16T05:07:48.265942+0000","last_peered":"2024-09-16T05:07:48.265942+0000","last_clean":"2024-09-16T05:07:48.265942+0000","last_became_active":"2024-09-16T05:07:39.576756+0000","last_became_peered":"2024-09-16T05:07:39.576756+0000","last_unstale":"2024-09-16T05:07:48.265942+0000","last_undegraded":"2024-09-16T05:07:48.265942+0000","last_fullsized":"2024-09-16T05:07:48.265942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_clean_scrub_stamp":"2024-09-16T05:07:38.167879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:57.060906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":48761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":48780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":688128,"data_stored":577689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":688128,"data_stored":577689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":159744,"data_stored":48780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":48761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:07:52.404 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T05:07:52.404 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T05:07:52.404 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T05:07:52.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T05:07:52.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 10 monclient: get_monmap_and_config 2024-09-16T05:07:52.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 10 monclient: build_initial_monmap 2024-09-16T05:07:52.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 10 monclient: monmap: 2024-09-16T05:07:52.514 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:52.514 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:52.514 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:52.517723+0000 2024-09-16T05:07:52.515 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:52.517723+0000 2024-09-16T05:07:52.515 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:52.515 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.515 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:52.515 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:52.515 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:52.515 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:52.515 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:52.515 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 1 Processor -- start 2024-09-16T05:07:52.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 1 -- start start 2024-09-16T05:07:52.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 10 monclient: init 2024-09-16T05:07:52.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:52.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T05:07:52.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcf3009b6c0 0x7fcf300a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 10 monclient(hunting): picked mon.noname-a con 0x7fcf3009b6c0 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:52.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fcf300a1400 0x7fcf300a1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 10 monclient(hunting): picked mon.noname-c con 0x7fcf300a1400 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:52.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcf30139640 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:52.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf3009d4e0 con 0x7fcf30139640 2024-09-16T05:07:52.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:52.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fcf300a1d50 con 0x7fcf3009b6c0 2024-09-16T05:07:52.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:52.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fcf3009ece0 con 0x7fcf300a1400 2024-09-16T05:07:52.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:52.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf35647640 10 monclient(hunting): authenticate will time out at 957.146775s 2024-09-16T05:07:52.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcf3009b6c0 0x7fcf300a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:52.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcf3009b6c0 0x7fcf300a0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55056/0 (socket says 172.21.15.114:55056) 2024-09-16T05:07:52.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 1 -- 172.21.15.114:0/2477016862 learned_addr learned my addr 172.21.15.114:0/2477016862 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:52.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 10 monclient(hunting): get_auth_request con 0x7fcf3009b6c0 auth_method 0 2024-09-16T05:07:52.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:52.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:52.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:52.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:52.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:52.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:52.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2e7fc640 1 --2- 172.21.15.114:0/2477016862 >> v2:172.21.15.149:3300/0 conn(0x7fcf300a1400 0x7fcf300a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:52.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2e7fc640 10 monclient(hunting): get_auth_request con 0x7fcf300a1400 auth_method 0 2024-09-16T05:07:52.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:52.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:52.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:52.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 10 monclient(hunting): handle_auth_done global_id 4350 payload 274 2024-09-16T05:07:52.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 1 -- 172.21.15.114:0/2477016862 >> v2:172.21.15.149:3300/0 conn(0x7fcf300a1400 msgr2=0x7fcf300a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:52.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 1 --2- 172.21.15.114:0/2477016862 >> v2:172.21.15.149:3300/0 conn(0x7fcf300a1400 0x7fcf300a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 1 -- 172.21.15.114:0/2477016862 >> v1:172.21.15.149:6789/0 conn(0x7fcf30139640 legacy=0x7fcf3013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:52.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 10 monclient: _finish_hunting 0 2024-09-16T05:07:52.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 1 monclient: found mon.noname-a 2024-09-16T05:07:52.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:52.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:52.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.514+0000 7fcf2effd640 1 -- 172.21.15.114:0/2477016862 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf3009f430 con 0x7fcf3009b6c0 2024-09-16T05:07:52.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2effd640 1 --2- 172.21.15.114:0/2477016862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3009b6c0 0x7fcf300a0ec0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcf300a2f70 tx=0x7fcf2801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7071935d9f3f2d1c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:52.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2dffb640 1 -- 172.21.15.114:0/2477016862 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf280b0070 con 0x7fcf3009b6c0 2024-09-16T05:07:52.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:52.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:52.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2dffb640 10 monclient: dump: 2024-09-16T05:07:52.523 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:52.523 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:52.523 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.523 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.523 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:52.523 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.523 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:52.523 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:52.523 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:52.523 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2dffb640 10 monclient: _finish_auth 0 2024-09-16T05:07:52.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:07:52.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:52.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2dffb640 1 -- 172.21.15.114:0/2477016862 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf280ab600 con 0x7fcf3009b6c0 2024-09-16T05:07:52.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 5 monclient: authenticate success, global_id 4350 2024-09-16T05:07:52.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:52.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:52.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2dffb640 1 -- 172.21.15.114:0/2477016862 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf280ab9d0 con 0x7fcf3009b6c0 2024-09-16T05:07:52.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:52.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:52.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2dffb640 10 monclient: dump: 2024-09-16T05:07:52.524 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:52.524 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:52.524 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.524 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.524 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:52.524 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.524 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:52.525 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:52.525 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:52.525 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:52.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 4 set_mon_vals no callback set 2024-09-16T05:07:52.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 10 monclient: shutdown 2024-09-16T05:07:52.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:52.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 -- 172.21.15.114:0/2477016862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3009b6c0 msgr2=0x7fcf300a0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 --2- 172.21.15.114:0/2477016862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3009b6c0 0x7fcf300a0ec0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcf300a2f70 tx=0x7fcf2801c430 comp rx=0 tx=0).stop 2024-09-16T05:07:52.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 -- 172.21.15.114:0/2477016862 shutdown_connections 2024-09-16T05:07:52.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 --2- 172.21.15.114:0/2477016862 >> v2:172.21.15.149:3300/0 conn(0x7fcf300a1400 0x7fcf300a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 --2- 172.21.15.114:0/2477016862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3009b6c0 0x7fcf300a0ec0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 -- 172.21.15.114:0/2477016862 >> 172.21.15.114:0/2477016862 conn(0x7fcf300a1fb0 msgr2=0x7fcf300a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:52.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 -- 172.21.15.114:0/2477016862 shutdown_connections 2024-09-16T05:07:52.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 -- 172.21.15.114:0/2477016862 wait complete. 2024-09-16T05:07:52.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 10 monclient: build_initial_monmap 2024-09-16T05:07:52.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 10 monclient: monmap: 2024-09-16T05:07:52.531 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:52.531 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:52.531 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:52.531 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:52.531 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:52.531 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.531 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:52.531 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:52.531 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:52.532 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 librados: starting msgr at 2024-09-16T05:07:52.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 librados: starting objecter 2024-09-16T05:07:52.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 Processor -- start 2024-09-16T05:07:52.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 -- start start 2024-09-16T05:07:52.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 librados: setting wanted keys 2024-09-16T05:07:52.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 librados: calling monclient init 2024-09-16T05:07:52.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 10 monclient: init 2024-09-16T05:07:52.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:52.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:07:52.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3009b6c0 0x7fcf30138c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 10 monclient(hunting): picked mon.noname-a con 0x7fcf3009b6c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:52.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcf300a1400 0x7fcf30073280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 10 monclient(hunting): picked mon.noname-c con 0x7fcf300a1400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:52.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf30139640 0x7fcf301cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 10 monclient(hunting): picked mon.noname-b con 0x7fcf30139640 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:52.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:52.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3013f970 con 0x7fcf3009b6c0 2024-09-16T05:07:52.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:52.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3009ddd0 con 0x7fcf30139640 2024-09-16T05:07:52.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:52.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fcf3013faf0 con 0x7fcf300a1400 2024-09-16T05:07:52.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:52.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf35647640 10 monclient(hunting): authenticate will time out at 957.151828s 2024-09-16T05:07:52.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3009b6c0 0x7fcf30138c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:52.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3009b6c0 0x7fcf30138c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55060/0 (socket says 172.21.15.114:55060) 2024-09-16T05:07:52.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2effd640 1 -- 172.21.15.114:0/1155318147 learned_addr learned my addr 172.21.15.114:0/1155318147 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:52.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2effd640 10 monclient(hunting): get_auth_request con 0x7fcf3009b6c0 auth_method 0 2024-09-16T05:07:52.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:52.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:52.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.518+0000 7fcf2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:52.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2e7fc640 1 --2- 172.21.15.114:0/1155318147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcf300a1400 0x7fcf30073280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:52.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:52.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:52.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:52.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2e7fc640 10 monclient(hunting): get_auth_request con 0x7fcf300a1400 auth_method 0 2024-09-16T05:07:52.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:52.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:52.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:52.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:52.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:52.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:52.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2effd640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2024-09-16T05:07:52.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2f7fe640 1 --2- 172.21.15.114:0/1155318147 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf30139640 0x7fcf301cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:52.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2effd640 1 -- 172.21.15.114:0/1155318147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcf300a1400 msgr2=0x7fcf30073280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2effd640 1 --2- 172.21.15.114:0/1155318147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcf300a1400 0x7fcf30073280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2effd640 1 -- 172.21.15.114:0/1155318147 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf30139640 msgr2=0x7fcf301cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2effd640 1 --2- 172.21.15.114:0/1155318147 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf30139640 0x7fcf301cb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2effd640 10 monclient: _finish_hunting 0 2024-09-16T05:07:52.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2effd640 1 monclient: found mon.noname-a 2024-09-16T05:07:52.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:52.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:52.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2effd640 1 -- 172.21.15.114:0/1155318147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf301c8b20 con 0x7fcf3009b6c0 2024-09-16T05:07:52.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2effd640 1 --2- 172.21.15.114:0/1155318147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3009b6c0 0x7fcf30138c10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcf280ae040 tx=0x7fcf280191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:52.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 10 client.?.objecter ms_handle_connect 0x7fcf3009b6c0 2024-09-16T05:07:52.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:52.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 1 -- 172.21.15.114:0/1155318147 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf280b0040 con 0x7fcf3009b6c0 2024-09-16T05:07:52.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:52.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:52.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 10 monclient: dump: 2024-09-16T05:07:52.536 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:52.536 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:52.536 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.536 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.536 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:52.537 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.537 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:52.537 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:52.537 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:52.537 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 10 monclient: _finish_auth 0 2024-09-16T05:07:52.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 10 monclient: _check_auth_tickets 2024-09-16T05:07:52.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:52.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 1 -- 172.21.15.114:0/1155318147 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf280b9070 con 0x7fcf3009b6c0 2024-09-16T05:07:52.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf35647640 5 monclient: authenticate success, global_id 4353 2024-09-16T05:07:52.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf35647640 10 monclient: _renew_subs 2024-09-16T05:07:52.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf35647640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:52.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf35647640 1 -- 172.21.15.114:0/1155318147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf30071d50 con 0x7fcf3009b6c0 2024-09-16T05:07:52.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:52.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 1 -- 172.21.15.114:0/1155318147 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf280b46a0 con 0x7fcf3009b6c0 2024-09-16T05:07:52.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:52.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:52.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 10 monclient: dump: 2024-09-16T05:07:52.539 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:52.539 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:52.539 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.539 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.539 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:52.539 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.539 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:52.539 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:52.539 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:52.539 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf35647640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:52.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf35647640 10 monclient: _renew_subs 2024-09-16T05:07:52.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf35647640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:52.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf35647640 1 -- 172.21.15.114:0/1155318147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf301c8140 con 0x7fcf3009b6c0 2024-09-16T05:07:52.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf35647640 1 librados: init done 2024-09-16T05:07:52.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2d7fa640 4 set_mon_vals no callback set 2024-09-16T05:07:52.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf35647640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:52.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf35647640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:52.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 1 -- 172.21.15.114:0/1155318147 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcf280b4840 con 0x7fcf3009b6c0 2024-09-16T05:07:52.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf35647640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:52.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf35647640 1 -- 172.21.15.114:0/1155318147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf30071900 con 0x7fcf3009b6c0 2024-09-16T05:07:52.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 1 --2- 172.21.15.114:0/1155318147 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcf1c037810 0x7fcf1c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 1 -- 172.21.15.114:0/1155318147 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7fcf280e24c0 con 0x7fcf3009b6c0 2024-09-16T05:07:52.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 10 client.4353.objecter ms_dispatch 0x7fcf30000ba0 osd_map(19..19 src has 1..19) 2024-09-16T05:07:52.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 3 client.4353.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T05:07:52.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 3 client.4353.objecter handle_osd_map decoding full epoch 19 2024-09-16T05:07:52.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 20 client.4353.objecter dump_active .. 0 homeless 2024-09-16T05:07:52.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2e7fc640 1 --2- 172.21.15.114:0/1155318147 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcf1c037810 0x7fcf1c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:52.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2e7fc640 10 monclient: get_auth_request con 0x7fcf1c037810 auth_method 0 2024-09-16T05:07:52.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf2e7fc640 1 --2- 172.21.15.114:0/1155318147 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcf1c037810 0x7fcf1c039cd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcf3013cc70 tx=0x7fcf24008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:52.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.522+0000 7fcf177fe640 10 client.4353.objecter ms_handle_connect 0x7fcf1c037810 2024-09-16T05:07:52.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.526+0000 7fcf177fe640 1 -- 172.21.15.114:0/1155318147 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fcf30071900 con 0x7fcf3009b6c0 2024-09-16T05:07:52.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.526+0000 7fcf177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:52.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.526+0000 7fcf177fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:52.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-16T05:07:52.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-16T05:07:52.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:52.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 1 -- 172.21.15.114:0/1155318147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fcf300a0ec0 con 0x7fcf3009b6c0 2024-09-16T05:07:52.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf177fe640 1 -- 172.21.15.114:0/1155318147 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fcf280160b0 con 0x7fcf3009b6c0 2024-09-16T05:07:52.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-16T05:07:52.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf177fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:52.711 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:07:52.711 INFO:teuthology.orchestra.run.smithi114.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T05:07:52.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 10 librados: watch_flush enter 2024-09-16T05:07:52.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 10 librados: watch_flush exit 2024-09-16T05:07:52.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 20 client.4353.objecter shutdown clearing up homeless session... 2024-09-16T05:07:52.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 10 client.4353.objecter successfully canceled tick 2024-09-16T05:07:52.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 1 -- 172.21.15.114:0/1155318147 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcf1c037810 msgr2=0x7fcf1c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 1 --2- 172.21.15.114:0/1155318147 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcf1c037810 0x7fcf1c039cd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcf3013cc70 tx=0x7fcf24008040 comp rx=0 tx=0).stop 2024-09-16T05:07:52.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 10 monclient: shutdown 2024-09-16T05:07:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 1 -- 172.21.15.114:0/1155318147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3009b6c0 msgr2=0x7fcf30138c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 1 --2- 172.21.15.114:0/1155318147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3009b6c0 0x7fcf30138c10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcf280ae040 tx=0x7fcf280191d0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 1 -- 172.21.15.114:0/1155318147 shutdown_connections 2024-09-16T05:07:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 1 --2- 172.21.15.114:0/1155318147 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcf1c037810 0x7fcf1c039cd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 1 --2- 172.21.15.114:0/1155318147 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf30139640 0x7fcf301cb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 1 --2- 172.21.15.114:0/1155318147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcf300a1400 0x7fcf30073280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 1 --2- 172.21.15.114:0/1155318147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3009b6c0 0x7fcf30138c10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 1 -- 172.21.15.114:0/1155318147 >> 172.21.15.114:0/1155318147 conn(0x7fcf300a1fb0 msgr2=0x7fcf3013b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 1 -- 172.21.15.114:0/1155318147 shutdown_connections 2024-09-16T05:07:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 1 -- 172.21.15.114:0/1155318147 wait complete. 2024-09-16T05:07:52.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.710+0000 7fcf35647640 1 librados: shutdown 2024-09-16T05:07:52.727 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T05:07:52.727 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-16T05:07:52.740 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-16T05:07:52.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T05:07:52.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.850+0000 7fac299df640 10 monclient: get_monmap_and_config 2024-09-16T05:07:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.850+0000 7fac299df640 10 monclient: build_initial_monmap 2024-09-16T05:07:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.850+0000 7fac299df640 10 monclient: monmap: 2024-09-16T05:07:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:52.855252+0000 2024-09-16T05:07:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:52.855252+0000 2024-09-16T05:07:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:52.853 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:52.853 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac299df640 1 Processor -- start 2024-09-16T05:07:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac299df640 1 -- start start 2024-09-16T05:07:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac299df640 10 monclient: init 2024-09-16T05:07:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac299df640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac299df640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-16T05:07:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac299df640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac24137730 0x7fac24137b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac299df640 10 monclient(hunting): picked mon.noname-b con 0x7fac24137730 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac299df640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac240a5d00 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac299df640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac240a4980 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac299df640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac240a55e0 con 0x7fac240a5d00 2024-09-16T05:07:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac299df640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac240a3710 con 0x7fac240a4980 2024-09-16T05:07:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac299df640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac299df640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fac2409c450 con 0x7fac24137730 2024-09-16T05:07:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac299df640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac299df640 10 monclient(hunting): authenticate will time out at 957.485151s 2024-09-16T05:07:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac24137730 0x7fac24137b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac24137730 0x7fac24137b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55756/0 (socket says 172.21.15.114:55756) 2024-09-16T05:07:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 1 -- 172.21.15.114:0/1084486658 learned_addr learned my addr 172.21.15.114:0/1084486658 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 10 monclient(hunting): get_auth_request con 0x7fac24137730 auth_method 0 2024-09-16T05:07:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2024-09-16T05:07:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 1 -- 172.21.15.114:0/1084486658 >> v1:172.21.15.149:6789/0 conn(0x7fac240a4980 legacy=0x7fac2413f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 1 -- 172.21.15.114:0/1084486658 >> v1:172.21.15.114:6789/0 conn(0x7fac240a5d00 legacy=0x7fac240a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:07:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 1 monclient: found mon.noname-b 2024-09-16T05:07:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.854+0000 7fac22ffd640 1 -- 172.21.15.114:0/1084486658 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2409be00 con 0x7fac24137730 2024-09-16T05:07:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac22ffd640 1 --2- 172.21.15.114:0/1084486658 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac24137730 0x7fac24137b10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fac140098a0 tx=0x7fac1401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=237612e6abce11cd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac21ffb640 1 -- 172.21.15.114:0/1084486658 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac140b0070 con 0x7fac24137730 2024-09-16T05:07:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac21ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac21ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac21ffb640 10 monclient: dump: 2024-09-16T05:07:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:52.864 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac21ffb640 10 monclient: _finish_auth 0 2024-09-16T05:07:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac21ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:07:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac21ffb640 1 -- 172.21.15.114:0/1084486658 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac140ab600 con 0x7fac24137730 2024-09-16T05:07:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 5 monclient: authenticate success, global_id 4295 2024-09-16T05:07:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac21ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac21ffb640 1 -- 172.21.15.114:0/1084486658 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac140ab9d0 con 0x7fac24137730 2024-09-16T05:07:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac21ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 4 set_mon_vals no callback set 2024-09-16T05:07:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac21ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac21ffb640 10 monclient: dump: 2024-09-16T05:07:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:52.866 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 10 monclient: shutdown 2024-09-16T05:07:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 -- 172.21.15.114:0/1084486658 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac24137730 msgr2=0x7fac24137b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 --2- 172.21.15.114:0/1084486658 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac24137730 0x7fac24137b10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fac140098a0 tx=0x7fac1401c400 comp rx=0 tx=0).stop 2024-09-16T05:07:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 -- 172.21.15.114:0/1084486658 shutdown_connections 2024-09-16T05:07:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 --2- 172.21.15.114:0/1084486658 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac24137730 0x7fac24137b10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 -- 172.21.15.114:0/1084486658 >> 172.21.15.114:0/1084486658 conn(0x7fac240a9c70 msgr2=0x7fac240aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 -- 172.21.15.114:0/1084486658 shutdown_connections 2024-09-16T05:07:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 -- 172.21.15.114:0/1084486658 wait complete. 2024-09-16T05:07:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 10 monclient: build_initial_monmap 2024-09-16T05:07:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 10 monclient: monmap: 2024-09-16T05:07:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:52.868 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 librados: starting msgr at 2024-09-16T05:07:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 librados: starting objecter 2024-09-16T05:07:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 Processor -- start 2024-09-16T05:07:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 -- start start 2024-09-16T05:07:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 librados: setting wanted keys 2024-09-16T05:07:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 librados: calling monclient init 2024-09-16T05:07:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 10 monclient: init 2024-09-16T05:07:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:07:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac240a4980 0x7fac2413f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 10 monclient(hunting): picked mon.noname-c con 0x7fac240a4980 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac240a5d00 0x7fac2413f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 10 monclient(hunting): picked mon.noname-a con 0x7fac240a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac24137730 0x7fac24138290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 10 monclient(hunting): picked mon.noname-b con 0x7fac24137730 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fac24137320 con 0x7fac240a5d00 2024-09-16T05:07:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fac241371a0 con 0x7fac24137730 2024-09-16T05:07:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fac24143420 con 0x7fac240a4980 2024-09-16T05:07:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac299df640 10 monclient(hunting): authenticate will time out at 957.491565s 2024-09-16T05:07:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac227fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac240a5d00 0x7fac2413f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac22ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac240a4980 0x7fac2413f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac227fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac240a5d00 0x7fac2413f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55066/0 (socket says 172.21.15.114:55066) 2024-09-16T05:07:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac227fc640 1 -- 172.21.15.114:0/2315527661 learned_addr learned my addr 172.21.15.114:0/2315527661 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac227fc640 10 monclient(hunting): get_auth_request con 0x7fac240a5d00 auth_method 0 2024-09-16T05:07:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:52.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac22ffd640 10 monclient(hunting): get_auth_request con 0x7fac240a4980 auth_method 0 2024-09-16T05:07:52.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:52.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:52.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:52.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:52.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:52.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:52.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:52.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:52.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:52.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac227fc640 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2024-09-16T05:07:52.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac227fc640 1 -- 172.21.15.114:0/2315527661 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac240a4980 msgr2=0x7fac2413f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac227fc640 1 --2- 172.21.15.114:0/2315527661 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac240a4980 0x7fac2413f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac227fc640 1 -- 172.21.15.114:0/2315527661 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac24137730 msgr2=0x7fac24138290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:52.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.858+0000 7fac227fc640 1 --2- 172.21.15.114:0/2315527661 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac24137730 0x7fac24138290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:52.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac227fc640 10 monclient: _finish_hunting 0 2024-09-16T05:07:52.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac227fc640 1 monclient: found mon.noname-a 2024-09-16T05:07:52.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac227fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac227fc640 1 -- 172.21.15.114:0/2315527661 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2413b8c0 con 0x7fac240a5d00 2024-09-16T05:07:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac227fc640 1 --2- 172.21.15.114:0/2315527661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac240a5d00 0x7fac2413f910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fac1800c400 tx=0x7fac1800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 10 client.?.objecter ms_handle_connect 0x7fac240a5d00 2024-09-16T05:07:52.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:52.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 1 -- 172.21.15.114:0/2315527661 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac180052d0 con 0x7fac240a5d00 2024-09-16T05:07:52.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:52.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:52.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 10 monclient: dump: 2024-09-16T05:07:52.879 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:52.879 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:52.879 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.879 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.879 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:52.879 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.879 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:52.879 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:52.879 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:52.879 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 10 monclient: _finish_auth 0 2024-09-16T05:07:52.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:07:52.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 1 -- 172.21.15.114:0/2315527661 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac18005470 con 0x7fac240a5d00 2024-09-16T05:07:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac299df640 5 monclient: authenticate success, global_id 4359 2024-09-16T05:07:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac299df640 10 monclient: _renew_subs 2024-09-16T05:07:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac299df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac299df640 1 -- 172.21.15.114:0/2315527661 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac2413a470 con 0x7fac240a5d00 2024-09-16T05:07:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 1 -- 172.21.15.114:0/2315527661 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac18007260 con 0x7fac240a5d00 2024-09-16T05:07:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 10 monclient: dump: 2024-09-16T05:07:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.880 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:52.881 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:52.881 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:52.881 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:52.881 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:52.881 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:52.881 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:52.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac217fa640 4 set_mon_vals no callback set 2024-09-16T05:07:52.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac299df640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:52.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac299df640 10 monclient: _renew_subs 2024-09-16T05:07:52.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac299df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:52.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac299df640 1 -- 172.21.15.114:0/2315527661 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac2409cbc0 con 0x7fac240a5d00 2024-09-16T05:07:52.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac299df640 1 librados: init done 2024-09-16T05:07:52.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac299df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:52.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac299df640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:52.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac299df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:52.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 1 -- 172.21.15.114:0/2315527661 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fac18007400 con 0x7fac240a5d00 2024-09-16T05:07:52.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac299df640 1 -- 172.21.15.114:0/2315527661 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac24137b10 con 0x7fac240a5d00 2024-09-16T05:07:52.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 1 --2- 172.21.15.114:0/2315527661 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac10037810 0x7fac10039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:52.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 1 -- 172.21.15.114:0/2315527661 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7fac18047be0 con 0x7fac240a5d00 2024-09-16T05:07:52.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 10 client.4359.objecter ms_dispatch 0x7fac24000ba0 osd_map(19..19 src has 1..19) 2024-09-16T05:07:52.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 3 client.4359.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T05:07:52.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 3 client.4359.objecter handle_osd_map decoding full epoch 19 2024-09-16T05:07:52.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 20 client.4359.objecter dump_active .. 0 homeless 2024-09-16T05:07:52.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac22ffd640 1 --2- 172.21.15.114:0/2315527661 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac10037810 0x7fac10039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:52.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac22ffd640 10 monclient: get_auth_request con 0x7fac10037810 auth_method 0 2024-09-16T05:07:52.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac0bfff640 10 client.4359.objecter ms_handle_connect 0x7fac10037810 2024-09-16T05:07:52.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.862+0000 7fac22ffd640 1 --2- 172.21.15.114:0/2315527661 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac10037810 0x7fac10039cd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fac140ad000 tx=0x7fac140a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:52.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.866+0000 7fac0bfff640 1 -- 172.21.15.114:0/2315527661 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fac18012d20 con 0x7fac240a5d00 2024-09-16T05:07:52.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.866+0000 7fac0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:52.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:52.866+0000 7fac0bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:53.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.026+0000 7fac299df640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:53.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.026+0000 7fac299df640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:53.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.026+0000 7fac299df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:53.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.026+0000 7fac299df640 1 -- 172.21.15.114:0/2315527661 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac24068650 con 0x7fac240a5d00 2024-09-16T05:07:53.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.026+0000 7fac0bfff640 1 -- 172.21.15.114:0/2315527661 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10936 (secure 0 0 0) 0x7fac18012a60 con 0x7fac240a5d00 2024-09-16T05:07:53.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.026+0000 7fac0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:07:53.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.026+0000 7fac0bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:53.029 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:07:53.029 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":19,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","created":"2024-09-16T05:07:27.305861+0000","modified":"2024-09-16T05:07:52.020041+0000","last_up_change":"2024-09-16T05:07:39.180058+0000","last_in_change":"2024-09-16T05:07:29.028596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:07:38.142989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T05:07:44.273942+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6ffdab41-0191-4764-ad62-922ec1091569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":269070361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":269070361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6811","nonce":269070361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":269070361}]},"public_addr":"172.21.15.114:6808/269070361","cluster_addr":"172.21.15.114:6809/269070361","heartbeat_back_addr":"172.21.15.114:6811/269070361","heartbeat_front_addr":"172.21.15.114:6810/269070361","state":["exists","up"]},{"osd":1,"uuid":"7de579ee-87a3-4463-87a5-6980b56cd113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2903720358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":2903720358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6803","nonce":2903720358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2903720358}]},"public_addr":"172.21.15.114:6800/2903720358","cluster_addr":"172.21.15.114:6801/2903720358","heartbeat_back_addr":"172.21.15.114:6803/2903720358","heartbeat_front_addr":"172.21.15.114:6802/2903720358","state":["exists","up"]},{"osd":2,"uuid":"258c8bf4-58c5-4c13-a0e5-ef9931efd698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1932657732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":1932657732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6807","nonce":1932657732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1932657732}]},"public_addr":"172.21.15.114:6804/1932657732","cluster_addr":"172.21.15.114:6805/1932657732","heartbeat_back_addr":"172.21.15.114:6807/1932657732","heartbeat_front_addr":"172.21.15.114:6806/1932657732","state":["exists","up"]},{"osd":3,"uuid":"acc1e03f-8a6c-4c85-9525-88707597d732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6805","nonce":1362772130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6807","nonce":1362772130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6811","nonce":1362772130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6809","nonce":1362772130}]},"public_addr":"172.21.15.149:6805/1362772130","cluster_addr":"172.21.15.149:6807/1362772130","heartbeat_back_addr":"172.21.15.149:6811/1362772130","heartbeat_front_addr":"172.21.15.149:6809/1362772130","state":["exists","up"]},{"osd":4,"uuid":"92cca4cf-19e9-44a1-9588-7c301a2c2065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":1092505261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6801","nonce":1092505261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6803","nonce":1092505261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":1092505261}]},"public_addr":"172.21.15.149:6800/1092505261","cluster_addr":"172.21.15.149:6801/1092505261","heartbeat_back_addr":"172.21.15.149:6803/1092505261","heartbeat_front_addr":"172.21.15.149:6802/1092505261","state":["exists","up"]},{"osd":5,"uuid":"ac717f4e-83fc-46ea-9816-7e35ef75a674","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":1244703827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":1244703827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":1244703827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":1244703827}]},"public_addr":"172.21.15.149:6804/1244703827","cluster_addr":"172.21.15.149:6806/1244703827","heartbeat_back_addr":"172.21.15.149:6810/1244703827","heartbeat_front_addr":"172.21.15.149:6808/1244703827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.554731+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.453254+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.446266+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:35.709335+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.069934+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.433872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:07:53.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 10 librados: watch_flush enter 2024-09-16T05:07:53.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 10 librados: watch_flush exit 2024-09-16T05:07:53.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 20 client.4359.objecter shutdown clearing up homeless session... 2024-09-16T05:07:53.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 10 client.4359.objecter successfully canceled tick 2024-09-16T05:07:53.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 1 -- 172.21.15.114:0/2315527661 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac10037810 msgr2=0x7fac10039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 1 --2- 172.21.15.114:0/2315527661 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac10037810 0x7fac10039cd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fac140ad000 tx=0x7fac140a6000 comp rx=0 tx=0).stop 2024-09-16T05:07:53.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 10 monclient: shutdown 2024-09-16T05:07:53.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:53.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 1 -- 172.21.15.114:0/2315527661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac240a5d00 msgr2=0x7fac2413f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 1 --2- 172.21.15.114:0/2315527661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac240a5d00 0x7fac2413f910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fac1800c400 tx=0x7fac1800c8c0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 1 -- 172.21.15.114:0/2315527661 shutdown_connections 2024-09-16T05:07:53.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 1 --2- 172.21.15.114:0/2315527661 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac10037810 0x7fac10039cd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 1 --2- 172.21.15.114:0/2315527661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac240a5d00 0x7fac2413f910 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 1 --2- 172.21.15.114:0/2315527661 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac24137730 0x7fac24138290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 1 --2- 172.21.15.114:0/2315527661 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac240a4980 0x7fac2413f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 1 -- 172.21.15.114:0/2315527661 >> 172.21.15.114:0/2315527661 conn(0x7fac240a9c70 msgr2=0x7fac240a8600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:53.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 1 -- 172.21.15.114:0/2315527661 shutdown_connections 2024-09-16T05:07:53.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 1 -- 172.21.15.114:0/2315527661 wait complete. 2024-09-16T05:07:53.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.030+0000 7fac299df640 1 librados: shutdown 2024-09-16T05:07:53.046 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T05:07:38.142989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T05:07:44.273942+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 18, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T05:07:53.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T05:07:53.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.162+0000 7f2108e25640 10 monclient: get_monmap_and_config 2024-09-16T05:07:53.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.162+0000 7f2108e25640 10 monclient: build_initial_monmap 2024-09-16T05:07:53.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.162+0000 7f2108e25640 10 monclient: monmap: 2024-09-16T05:07:53.164 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:53.164 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:53.164 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:53.167212+0000 2024-09-16T05:07:53.164 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:53.167212+0000 2024-09-16T05:07:53.164 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:53.164 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.164 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:53.164 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:53.164 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:53.164 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:53.164 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:53.165 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 1 Processor -- start 2024-09-16T05:07:53.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 1 -- start start 2024-09-16T05:07:53.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 10 monclient: init 2024-09-16T05:07:53.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:53.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T05:07:53.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f210412d7b0 0x7f21040a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 10 monclient(hunting): picked mon.noname-c con 0x7f210412d7b0 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:53.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f210412db90 0x7f2104072540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 10 monclient(hunting): picked mon.noname-b con 0x7f210412db90 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:53.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f210412dfd0 0x7f2104072a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 10 monclient(hunting): picked mon.noname-a con 0x7f210412dfd0 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:53.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:53.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f21040a13a0 con 0x7f210412dfd0 2024-09-16T05:07:53.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:53.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f21040a1680 con 0x7f210412d7b0 2024-09-16T05:07:53.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:53.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f210409cab0 con 0x7f210412db90 2024-09-16T05:07:53.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:53.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 10 monclient(hunting): authenticate will time out at 957.797049s 2024-09-16T05:07:53.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f210412db90 0x7f2104072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2102d9d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f210412dfd0 0x7f2104072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f210412db90 0x7f2104072540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55776/0 (socket says 172.21.15.114:55776) 2024-09-16T05:07:53.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2102d9d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f210412dfd0 0x7f2104072a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55076/0 (socket says 172.21.15.114:55076) 2024-09-16T05:07:53.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 1 -- 172.21.15.114:0/200722818 learned_addr learned my addr 172.21.15.114:0/200722818 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:53.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 10 monclient(hunting): get_auth_request con 0x7f210412db90 auth_method 0 2024-09-16T05:07:53.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:53.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:53.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:53.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2102d9d640 10 monclient(hunting): get_auth_request con 0x7f210412dfd0 auth_method 0 2024-09-16T05:07:53.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2102d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:53.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2102d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:53.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2102d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:53.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:53.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:53.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:53.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2102d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:53.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2102d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:53.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2102d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:53.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2024-09-16T05:07:53.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 1 -- 172.21.15.114:0/200722818 >> v2:172.21.15.149:3300/0 conn(0x7f210412d7b0 msgr2=0x7f21040a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:53.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 1 --2- 172.21.15.114:0/200722818 >> v2:172.21.15.149:3300/0 conn(0x7f210412d7b0 0x7f21040a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 1 -- 172.21.15.114:0/200722818 >> v2:172.21.15.114:3300/0 conn(0x7f210412dfd0 msgr2=0x7f2104072a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:53.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 1 --2- 172.21.15.114:0/200722818 >> v2:172.21.15.114:3300/0 conn(0x7f210412dfd0 0x7f2104072a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:07:53.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 1 monclient: found mon.noname-b 2024-09-16T05:07:53.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:53.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:53.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 1 -- 172.21.15.114:0/200722818 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21040684b0 con 0x7f210412db90 2024-09-16T05:07:53.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2102d9d640 1 --2- 172.21.15.114:0/200722818 >> v2:172.21.15.114:3300/0 conn(0x7f210412dfd0 0x7f2104072a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:53.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2101d9b640 1 --2- 172.21.15.114:0/200722818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f210412db90 0x7f2104072540 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f210406e0e0 tx=0x7f20f801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae9754e36831b48a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:53.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f210159a640 1 -- 172.21.15.114:0/200722818 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f80b0070 con 0x7f210412db90 2024-09-16T05:07:53.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f210159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:53.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f210159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:53.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f210159a640 10 monclient: dump: 2024-09-16T05:07:53.178 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:53.178 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:53.178 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.178 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.178 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:53.178 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.178 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:53.178 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:53.178 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:53.179 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f210159a640 10 monclient: _finish_auth 0 2024-09-16T05:07:53.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f210159a640 10 monclient: _check_auth_tickets 2024-09-16T05:07:53.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f210159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:53.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f210159a640 1 -- 172.21.15.114:0/200722818 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20f80ab600 con 0x7f210412db90 2024-09-16T05:07:53.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f210159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:53.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f210159a640 1 -- 172.21.15.114:0/200722818 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f80ab9d0 con 0x7f210412db90 2024-09-16T05:07:53.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 5 monclient: authenticate success, global_id 4301 2024-09-16T05:07:53.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:53.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f2108e25640 4 set_mon_vals no callback set 2024-09-16T05:07:53.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f210159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:53.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f210159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:53.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.166+0000 7f210159a640 10 monclient: dump: 2024-09-16T05:07:53.180 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:53.180 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:53.180 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.180 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.180 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:53.180 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.180 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:53.180 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:53.180 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:53.180 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient: shutdown 2024-09-16T05:07:53.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:53.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 -- 172.21.15.114:0/200722818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f210412db90 msgr2=0x7f2104072540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 --2- 172.21.15.114:0/200722818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f210412db90 0x7f2104072540 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f210406e0e0 tx=0x7f20f801c3e0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 -- 172.21.15.114:0/200722818 shutdown_connections 2024-09-16T05:07:53.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 --2- 172.21.15.114:0/200722818 >> v2:172.21.15.114:3300/0 conn(0x7f210412dfd0 0x7f2104072a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 --2- 172.21.15.114:0/200722818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f210412db90 0x7f2104072540 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 --2- 172.21.15.114:0/200722818 >> v2:172.21.15.149:3300/0 conn(0x7f210412d7b0 0x7f21040a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 -- 172.21.15.114:0/200722818 >> 172.21.15.114:0/200722818 conn(0x7f21040a6380 msgr2=0x7f21040a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:53.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 -- 172.21.15.114:0/200722818 shutdown_connections 2024-09-16T05:07:53.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 -- 172.21.15.114:0/200722818 wait complete. 2024-09-16T05:07:53.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient: build_initial_monmap 2024-09-16T05:07:53.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient: monmap: 2024-09-16T05:07:53.181 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:53.181 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:53.182 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:53.182 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:53.182 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:53.182 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.182 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:53.182 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:53.182 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:53.182 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 librados: starting msgr at 2024-09-16T05:07:53.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 librados: starting objecter 2024-09-16T05:07:53.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 Processor -- start 2024-09-16T05:07:53.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 -- start start 2024-09-16T05:07:53.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 librados: setting wanted keys 2024-09-16T05:07:53.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 librados: calling monclient init 2024-09-16T05:07:53.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient: init 2024-09-16T05:07:53.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:53.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:07:53.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f210412d7b0 0x7f210412e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient(hunting): picked mon.noname-c con 0x7f210412d7b0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:53.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f210412db90 0x7f210412edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient(hunting): picked mon.noname-a con 0x7f210412db90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:53.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f210412dfd0 0x7f210412f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient(hunting): picked mon.noname-b con 0x7f210412dfd0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:53.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:53.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2104140d60 con 0x7f210412db90 2024-09-16T05:07:53.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:53.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f21040733d0 con 0x7f210412dfd0 2024-09-16T05:07:53.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:53.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2104140ee0 con 0x7f210412d7b0 2024-09-16T05:07:53.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:53.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient(hunting): authenticate will time out at 957.801926s 2024-09-16T05:07:53.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f210412d7b0 0x7f210412e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2101d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f210412db90 0x7f210412edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f210412d7b0 0x7f210412e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55782/0 (socket says 172.21.15.114:55782) 2024-09-16T05:07:53.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 1 -- 172.21.15.114:0/1597547083 learned_addr learned my addr 172.21.15.114:0/1597547083 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:53.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 10 monclient(hunting): get_auth_request con 0x7f210412d7b0 auth_method 0 2024-09-16T05:07:53.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:53.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:53.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:53.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2101d9b640 10 monclient(hunting): get_auth_request con 0x7f210412db90 auth_method 0 2024-09-16T05:07:53.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2101d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:53.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2101d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:53.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2101d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:53.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:53.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:53.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:53.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2101d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:53.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2101d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:53.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2101d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:53.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2102d9d640 1 --2- 172.21.15.114:0/1597547083 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f210412dfd0 0x7f210412f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-09-16T05:07:53.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 1 -- 172.21.15.114:0/1597547083 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f210412dfd0 msgr2=0x7f210412f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 1 --2- 172.21.15.114:0/1597547083 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f210412dfd0 0x7f210412f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 1 -- 172.21.15.114:0/1597547083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f210412db90 msgr2=0x7f210412edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 1 --2- 172.21.15.114:0/1597547083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f210412db90 0x7f210412edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 10 monclient: _finish_hunting 0 2024-09-16T05:07:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 1 monclient: found mon.noname-c 2024-09-16T05:07:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 1 -- 172.21.15.114:0/1597547083 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2104138f30 con 0x7f210412d7b0 2024-09-16T05:07:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2102d9d640 10 monclient: get_auth_request con 0x7f210412dfd0 auth_method 0 2024-09-16T05:07:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2102d9d640 1 --2- 172.21.15.114:0/1597547083 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f210412dfd0 0x7f210412f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:07:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2101d9b640 1 --2- 172.21.15.114:0/1597547083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f210412db90 0x7f210412edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f210259c640 1 --2- 172.21.15.114:0/1597547083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f210412d7b0 0x7f210412e8b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f20f400b550 tx=0x7f20f400ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 10 client.?.objecter ms_handle_connect 0x7f210412d7b0 2024-09-16T05:07:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:53.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 1 -- 172.21.15.114:0/1597547083 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f4014070 con 0x7f210412d7b0 2024-09-16T05:07:53.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:53.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:53.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 10 monclient: dump: 2024-09-16T05:07:53.187 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:53.187 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:53.187 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.187 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.187 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:53.187 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.187 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:53.187 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:53.187 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:53.187 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:07:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:07:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 1 -- 172.21.15.114:0/1597547083 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20f4010610 con 0x7f210412d7b0 2024-09-16T05:07:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 1 -- 172.21.15.114:0/1597547083 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f40109e0 con 0x7f210412d7b0 2024-09-16T05:07:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 5 monclient: authenticate success, global_id 4304 2024-09-16T05:07:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient: _renew_subs 2024-09-16T05:07:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2100d99640 4 set_mon_vals no callback set 2024-09-16T05:07:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 -- 172.21.15.114:0/1597547083 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f210413e0e0 con 0x7f210412d7b0 2024-09-16T05:07:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 10 monclient: dump: 2024-09-16T05:07:53.188 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:53.189 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient: _renew_subs 2024-09-16T05:07:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:53.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 -- 172.21.15.114:0/1597547083 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2104138c50 con 0x7f210412d7b0 2024-09-16T05:07:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 librados: init done 2024-09-16T05:07:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 1 -- 172.21.15.114:0/1597547083 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f20f4017070 con 0x7f210412d7b0 2024-09-16T05:07:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f2108e25640 1 -- 172.21.15.114:0/1597547083 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21040a1e60 con 0x7f210412d7b0 2024-09-16T05:07:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 1 --2- 172.21.15.114:0/1597547083 >> v2:172.21.15.149:6812/1185531135 conn(0x7f20e4037810 0x7f20e4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 1 -- 172.21.15.114:0/1597547083 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f20f4048100 con 0x7f210412d7b0 2024-09-16T05:07:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 10 client.4304.objecter ms_dispatch 0x7f2104000ba0 osd_map(19..19 src has 1..19) 2024-09-16T05:07:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 3 client.4304.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T05:07:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 3 client.4304.objecter handle_osd_map decoding full epoch 19 2024-09-16T05:07:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.170+0000 7f20f2ffd640 20 client.4304.objecter dump_active .. 0 homeless 2024-09-16T05:07:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.174+0000 7f2101d9b640 1 --2- 172.21.15.114:0/1597547083 >> v2:172.21.15.149:6812/1185531135 conn(0x7f20e4037810 0x7f20e4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.174+0000 7f2101d9b640 10 monclient: get_auth_request con 0x7f20e4037810 auth_method 0 2024-09-16T05:07:53.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.174+0000 7f20f2ffd640 10 client.4304.objecter ms_handle_connect 0x7f20e4037810 2024-09-16T05:07:53.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.174+0000 7f2101d9b640 1 --2- 172.21.15.114:0/1597547083 >> v2:172.21.15.149:6812/1185531135 conn(0x7f20e4037810 0x7f20e4039cd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f20f80ad000 tx=0x7f20f80a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:53.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.178+0000 7f20f2ffd640 1 -- 172.21.15.114:0/1597547083 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f20f40179a0 con 0x7f210412d7b0 2024-09-16T05:07:53.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.178+0000 7f20f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:53.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.178+0000 7f20f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:53.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.326+0000 7f2108e25640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T05:07:53.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.326+0000 7f2108e25640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T05:07:53.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.326+0000 7f2108e25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:53.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.326+0000 7f2108e25640 1 -- 172.21.15.114:0/1597547083 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f210406d190 con 0x7f210412d7b0 2024-09-16T05:07:53.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.326+0000 7f20f2ffd640 1 -- 172.21.15.114:0/1597547083 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) ==== 93+0+10 (secure 0 0 0) 0x7f20f4010b80 con 0x7f210412d7b0 2024-09-16T05:07:53.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.326+0000 7f20f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T05:07:53.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.326+0000 7f20f2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:53.330 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-09-16T05:07:53.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 10 librados: watch_flush enter 2024-09-16T05:07:53.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 10 librados: watch_flush exit 2024-09-16T05:07:53.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 20 client.4304.objecter shutdown clearing up homeless session... 2024-09-16T05:07:53.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 10 client.4304.objecter successfully canceled tick 2024-09-16T05:07:53.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 1 -- 172.21.15.114:0/1597547083 >> v2:172.21.15.149:6812/1185531135 conn(0x7f20e4037810 msgr2=0x7f20e4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 1 --2- 172.21.15.114:0/1597547083 >> v2:172.21.15.149:6812/1185531135 conn(0x7f20e4037810 0x7f20e4039cd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f20f80ad000 tx=0x7f20f80a6000 comp rx=0 tx=0).stop 2024-09-16T05:07:53.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 10 monclient: shutdown 2024-09-16T05:07:53.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:53.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 1 -- 172.21.15.114:0/1597547083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f210412d7b0 msgr2=0x7f210412e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 1 --2- 172.21.15.114:0/1597547083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f210412d7b0 0x7f210412e8b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f20f400b550 tx=0x7f20f400ba10 comp rx=0 tx=0).stop 2024-09-16T05:07:53.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 1 -- 172.21.15.114:0/1597547083 shutdown_connections 2024-09-16T05:07:53.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 1 --2- 172.21.15.114:0/1597547083 >> v2:172.21.15.149:6812/1185531135 conn(0x7f20e4037810 0x7f20e4039cd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 1 --2- 172.21.15.114:0/1597547083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f210412db90 0x7f210412edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 1 --2- 172.21.15.114:0/1597547083 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f210412dfd0 0x7f210412f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 1 --2- 172.21.15.114:0/1597547083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f210412d7b0 0x7f210412e8b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 1 -- 172.21.15.114:0/1597547083 >> 172.21.15.114:0/1597547083 conn(0x7f21040a6380 msgr2=0x7f210409f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:53.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 1 -- 172.21.15.114:0/1597547083 shutdown_connections 2024-09-16T05:07:53.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 1 -- 172.21.15.114:0/1597547083 wait complete. 2024-09-16T05:07:53.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.330+0000 7f2108e25640 1 librados: shutdown 2024-09-16T05:07:53.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-16T05:07:53.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.462+0000 7f0c14eb8640 10 monclient: get_monmap_and_config 2024-09-16T05:07:53.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.462+0000 7f0c14eb8640 10 monclient: build_initial_monmap 2024-09-16T05:07:53.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.462+0000 7f0c14eb8640 10 monclient: monmap: 2024-09-16T05:07:53.465 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:53.465 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:53.465 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:53.468220+0000 2024-09-16T05:07:53.465 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:53.468220+0000 2024-09-16T05:07:53.465 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:53.465 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.465 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:53.466 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:53.466 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:53.466 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:53.466 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:53.466 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c14eb8640 1 Processor -- start 2024-09-16T05:07:53.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c14eb8640 1 -- start start 2024-09-16T05:07:53.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c14eb8640 10 monclient: init 2024-09-16T05:07:53.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c14eb8640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:53.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c14eb8640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T05:07:53.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c14eb8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c101376e0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:53.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c14eb8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c101370a0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:53.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c14eb8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0c100a5d00 0x7f0c10137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c14eb8640 10 monclient(hunting): picked mon.noname-a con 0x7f0c100a5d00 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:53.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c14eb8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c100a5090 con 0x7f0c101376e0 2024-09-16T05:07:53.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c14eb8640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c1009bd20 con 0x7f0c101370a0 2024-09-16T05:07:53.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c14eb8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:53.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c14eb8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f0c10138000 con 0x7f0c100a5d00 2024-09-16T05:07:53.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c14eb8640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:53.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c14eb8640 10 monclient(hunting): authenticate will time out at 958.097537s 2024-09-16T05:07:53.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0c100a5d00 0x7f0c10137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0c100a5d00 0x7f0c10137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55108/0 (socket says 172.21.15.114:55108) 2024-09-16T05:07:53.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0e59c640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0c101376e0 0x7f0c100a4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38698/0 (socket says 172.21.15.114:38698) 2024-09-16T05:07:53.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 1 -- 172.21.15.114:0/1700946435 learned_addr learned my addr 172.21.15.114:0/1700946435 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:53.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 10 monclient(hunting): get_auth_request con 0x7f0c100a5d00 auth_method 0 2024-09-16T05:07:53.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:53.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:53.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:53.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:53.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:53.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:53.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 10 monclient(hunting): handle_auth_done global_id 4368 payload 274 2024-09-16T05:07:53.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 1 -- 172.21.15.114:0/1700946435 >> v1:172.21.15.149:6789/0 conn(0x7f0c101370a0 legacy=0x7f0c100a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:53.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 1 -- 172.21.15.114:0/1700946435 >> v1:172.21.15.114:6789/0 conn(0x7f0c101376e0 legacy=0x7f0c100a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 10 monclient: _finish_hunting 0 2024-09-16T05:07:53.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 1 monclient: found mon.noname-a 2024-09-16T05:07:53.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0d59a640 1 -- 172.21.15.114:0/1700946435 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 166491243 0 0) 0x7f0c100a5090 con 0x7f0c101376e0 2024-09-16T05:07:53.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:53.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:53.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0ed9d640 1 -- 172.21.15.114:0/1700946435 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c1009c000 con 0x7f0c100a5d00 2024-09-16T05:07:53.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.466+0000 7f0c0d59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:07:53.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 1 --2- 172.21.15.114:0/1700946435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c100a5d00 0x7f0c10137ac0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0c08009e50 tx=0x7f0c0801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b64a4029b04ed251 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0d59a640 1 -- 172.21.15.114:0/1700946435 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c080a5570 con 0x7f0c100a5d00 2024-09-16T05:07:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0d59a640 10 monclient: dump: 2024-09-16T05:07:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:53.478 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0d59a640 10 monclient: _finish_auth 0 2024-09-16T05:07:53.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0d59a640 10 monclient: _check_auth_tickets 2024-09-16T05:07:53.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:53.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 5 monclient: authenticate success, global_id 4368 2024-09-16T05:07:53.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:53.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0d59a640 1 -- 172.21.15.114:0/1700946435 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c080a5710 con 0x7f0c100a5d00 2024-09-16T05:07:53.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:53.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0d59a640 1 -- 172.21.15.114:0/1700946435 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c080a5ae0 con 0x7f0c100a5d00 2024-09-16T05:07:53.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:53.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:53.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0d59a640 10 monclient: dump: 2024-09-16T05:07:53.478 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:53.479 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:53.479 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.479 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.479 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:53.479 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.479 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:53.479 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:53.479 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:53.479 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:53.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 4 set_mon_vals no callback set 2024-09-16T05:07:53.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient: shutdown 2024-09-16T05:07:53.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:53.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 -- 172.21.15.114:0/1700946435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c100a5d00 msgr2=0x7f0c10137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 --2- 172.21.15.114:0/1700946435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c100a5d00 0x7f0c10137ac0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0c08009e50 tx=0x7f0c0801c8b0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 -- 172.21.15.114:0/1700946435 shutdown_connections 2024-09-16T05:07:53.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 --2- 172.21.15.114:0/1700946435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c100a5d00 0x7f0c10137ac0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 -- 172.21.15.114:0/1700946435 >> 172.21.15.114:0/1700946435 conn(0x7f0c100a9a40 msgr2=0x7f0c100a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:53.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 -- 172.21.15.114:0/1700946435 shutdown_connections 2024-09-16T05:07:53.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 -- 172.21.15.114:0/1700946435 wait complete. 2024-09-16T05:07:53.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient: build_initial_monmap 2024-09-16T05:07:53.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient: monmap: 2024-09-16T05:07:53.480 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:53.480 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:53.480 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:53.480 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:53.480 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:53.481 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.481 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:53.481 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:53.481 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:53.481 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 librados: starting msgr at 2024-09-16T05:07:53.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 librados: starting objecter 2024-09-16T05:07:53.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 Processor -- start 2024-09-16T05:07:53.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 -- start start 2024-09-16T05:07:53.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 librados: setting wanted keys 2024-09-16T05:07:53.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 librados: calling monclient init 2024-09-16T05:07:53.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient: init 2024-09-16T05:07:53.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:53.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:07:53.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0c100a5d00 0x7f0c101359e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient(hunting): picked mon.noname-b con 0x7f0c100a5d00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:53.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c101370a0 0x7f0c10135f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient(hunting): picked mon.noname-c con 0x7f0c101370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:53.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c101376e0 0x7f0c1013be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient(hunting): picked mon.noname-a con 0x7f0c101376e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:53.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:53.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0c10142fa0 con 0x7f0c101376e0 2024-09-16T05:07:53.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:53.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f0c10142e20 con 0x7f0c100a5d00 2024-09-16T05:07:53.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:53.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f0c10143120 con 0x7f0c101370a0 2024-09-16T05:07:53.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:53.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient(hunting): authenticate will time out at 958.102709s 2024-09-16T05:07:53.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c101376e0 0x7f0c1013be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c101376e0 0x7f0c1013be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55120/0 (socket says 172.21.15.114:55120) 2024-09-16T05:07:53.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 1 -- 172.21.15.114:0/2792992001 learned_addr learned my addr 172.21.15.114:0/2792992001 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:53.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 10 monclient(hunting): get_auth_request con 0x7f0c101376e0 auth_method 0 2024-09-16T05:07:53.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:53.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:53.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:53.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0dd9b640 1 --2- 172.21.15.114:0/2792992001 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c101370a0 0x7f0c10135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0dd9b640 10 monclient(hunting): get_auth_request con 0x7f0c101370a0 auth_method 0 2024-09-16T05:07:53.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:53.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:53.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:53.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:53.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:53.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:53.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:53.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:53.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:53.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2024-09-16T05:07:53.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 1 -- 172.21.15.114:0/2792992001 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c101370a0 msgr2=0x7f0c10135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 1 --2- 172.21.15.114:0/2792992001 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c101370a0 0x7f0c10135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0e59c640 1 --2- 172.21.15.114:0/2792992001 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0c100a5d00 0x7f0c101359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 1 -- 172.21.15.114:0/2792992001 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0c100a5d00 msgr2=0x7f0c101359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 1 --2- 172.21.15.114:0/2792992001 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0c100a5d00 0x7f0c101359e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 10 monclient: _finish_hunting 0 2024-09-16T05:07:53.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 1 monclient: found mon.noname-a 2024-09-16T05:07:53.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:53.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:53.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 1 -- 172.21.15.114:0/2792992001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c1013f750 con 0x7f0c101376e0 2024-09-16T05:07:53.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c0ed9d640 1 --2- 172.21.15.114:0/2792992001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c101376e0 0x7f0c1013be60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0c08009e50 tx=0x7f0c08019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:53.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfeffd640 10 client.?.objecter ms_handle_connect 0x7f0c101376e0 2024-09-16T05:07:53.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfeffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:53.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfeffd640 1 -- 172.21.15.114:0/2792992001 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c080bc030 con 0x7f0c101376e0 2024-09-16T05:07:53.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:53.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:53.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfeffd640 10 monclient: dump: 2024-09-16T05:07:53.486 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:53.486 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:53.486 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.486 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.486 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:53.486 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.486 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:53.486 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:53.486 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:53.486 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfeffd640 10 monclient: _finish_auth 0 2024-09-16T05:07:53.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfeffd640 10 monclient: _check_auth_tickets 2024-09-16T05:07:53.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:53.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfeffd640 1 -- 172.21.15.114:0/2792992001 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c080b8470 con 0x7f0c101376e0 2024-09-16T05:07:53.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 5 monclient: authenticate success, global_id 4374 2024-09-16T05:07:53.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient: _renew_subs 2024-09-16T05:07:53.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:53.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 -- 172.21.15.114:0/2792992001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c1013e580 con 0x7f0c101376e0 2024-09-16T05:07:53.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:53.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfeffd640 1 -- 172.21.15.114:0/2792992001 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c080b8820 con 0x7f0c101376e0 2024-09-16T05:07:53.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:53.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:53.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfeffd640 10 monclient: dump: 2024-09-16T05:07:53.488 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:53.488 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:53.488 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.488 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.488 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:53.488 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.488 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:53.488 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:53.488 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:53.489 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:53.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient: _renew_subs 2024-09-16T05:07:53.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:53.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 -- 172.21.15.114:0/2792992001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c100a2010 con 0x7f0c101376e0 2024-09-16T05:07:53.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0c14eb8640 1 librados: init done 2024-09-16T05:07:53.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.470+0000 7f0bfffff640 4 set_mon_vals no callback set 2024-09-16T05:07:53.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.474+0000 7f0c14eb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:53.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.474+0000 7f0c14eb8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:53.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.474+0000 7f0c14eb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:53.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.474+0000 7f0c14eb8640 1 -- 172.21.15.114:0/2792992001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c10137ac0 con 0x7f0c101376e0 2024-09-16T05:07:53.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.474+0000 7f0bfeffd640 1 -- 172.21.15.114:0/2792992001 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0c080bf070 con 0x7f0c101376e0 2024-09-16T05:07:53.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.474+0000 7f0bfeffd640 1 --2- 172.21.15.114:0/2792992001 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0bf0037810 0x7f0bf0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.474+0000 7f0bfeffd640 1 -- 172.21.15.114:0/2792992001 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f0c080e3610 con 0x7f0c101376e0 2024-09-16T05:07:53.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.474+0000 7f0bfeffd640 10 client.4374.objecter ms_dispatch 0x7f0c10000ba0 osd_map(19..19 src has 1..19) 2024-09-16T05:07:53.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.474+0000 7f0bfeffd640 3 client.4374.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T05:07:53.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.474+0000 7f0bfeffd640 3 client.4374.objecter handle_osd_map decoding full epoch 19 2024-09-16T05:07:53.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.474+0000 7f0bfeffd640 20 client.4374.objecter dump_active .. 0 homeless 2024-09-16T05:07:53.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.474+0000 7f0c0e59c640 1 --2- 172.21.15.114:0/2792992001 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0bf0037810 0x7f0bf0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.474+0000 7f0c0e59c640 10 monclient: get_auth_request con 0x7f0bf0037810 auth_method 0 2024-09-16T05:07:53.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.474+0000 7f0c0e59c640 1 --2- 172.21.15.114:0/2792992001 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0bf0037810 0x7f0bf0039cd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c00000f70 tx=0x7f0c00009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:53.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.474+0000 7f0bfeffd640 10 client.4374.objecter ms_handle_connect 0x7f0bf0037810 2024-09-16T05:07:53.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.478+0000 7f0bfeffd640 1 -- 172.21.15.114:0/2792992001 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f0c080165c0 con 0x7f0c101376e0 2024-09-16T05:07:53.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.478+0000 7f0bfeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:53.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.478+0000 7f0bfeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:53.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T05:07:53.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T05:07:53.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:53.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 1 -- 172.21.15.114:0/2792992001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f0c1013e300 con 0x7f0c101376e0 2024-09-16T05:07:53.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0bfeffd640 1 -- 172.21.15.114:0/2792992001 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) ==== 92+0+10 (secure 0 0 0) 0x7f0c080b8a70 con 0x7f0c101376e0 2024-09-16T05:07:53.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0bfeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T05:07:53.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0bfeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:53.630 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 8 2024-09-16T05:07:53.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 10 librados: watch_flush enter 2024-09-16T05:07:53.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 10 librados: watch_flush exit 2024-09-16T05:07:53.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 20 client.4374.objecter shutdown clearing up homeless session... 2024-09-16T05:07:53.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 10 client.4374.objecter successfully canceled tick 2024-09-16T05:07:53.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 1 -- 172.21.15.114:0/2792992001 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0bf0037810 msgr2=0x7f0bf0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 1 --2- 172.21.15.114:0/2792992001 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0bf0037810 0x7f0bf0039cd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c00000f70 tx=0x7f0c00009040 comp rx=0 tx=0).stop 2024-09-16T05:07:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 10 monclient: shutdown 2024-09-16T05:07:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 1 -- 172.21.15.114:0/2792992001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c101376e0 msgr2=0x7f0c1013be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 1 --2- 172.21.15.114:0/2792992001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c101376e0 0x7f0c1013be60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0c08009e50 tx=0x7f0c08019650 comp rx=0 tx=0).stop 2024-09-16T05:07:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 1 -- 172.21.15.114:0/2792992001 shutdown_connections 2024-09-16T05:07:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 1 --2- 172.21.15.114:0/2792992001 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0bf0037810 0x7f0bf0039cd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 1 --2- 172.21.15.114:0/2792992001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c101376e0 0x7f0c1013be60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 1 --2- 172.21.15.114:0/2792992001 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c101370a0 0x7f0c10135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 1 --2- 172.21.15.114:0/2792992001 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0c100a5d00 0x7f0c101359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 1 -- 172.21.15.114:0/2792992001 >> 172.21.15.114:0/2792992001 conn(0x7f0c100a9a40 msgr2=0x7f0c1009f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 1 -- 172.21.15.114:0/2792992001 shutdown_connections 2024-09-16T05:07:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 1 -- 172.21.15.114:0/2792992001 wait complete. 2024-09-16T05:07:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.630+0000 7f0c14eb8640 1 librados: shutdown 2024-09-16T05:07:53.648 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-16T05:07:53.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:07:53.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.758+0000 7f2995ef4640 10 monclient: get_monmap_and_config 2024-09-16T05:07:53.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.758+0000 7f2995ef4640 10 monclient: build_initial_monmap 2024-09-16T05:07:53.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.758+0000 7f2995ef4640 10 monclient: monmap: 2024-09-16T05:07:53.760 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:53.760 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:53.760 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:53.762889+0000 2024-09-16T05:07:53.760 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:53.762889+0000 2024-09-16T05:07:53.760 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:53.760 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.760 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:53.761 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:53.761 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:53.761 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:53.761 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:53.761 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.758+0000 7f2995ef4640 1 Processor -- start 2024-09-16T05:07:53.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.758+0000 7f2995ef4640 1 -- start start 2024-09-16T05:07:53.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.758+0000 7f2995ef4640 10 monclient: init 2024-09-16T05:07:53.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.758+0000 7f2995ef4640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:53.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.758+0000 7f2995ef4640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-16T05:07:53.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.758+0000 7f2995ef4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f299009b6c0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:53.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.758+0000 7f2995ef4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f29900a15d0 0x7f299013a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.758+0000 7f2995ef4640 10 monclient(hunting): picked mon.noname-a con 0x7f29900a15d0 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:53.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29900a19e0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:53.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f299009d4e0 con 0x7f29900a19e0 2024-09-16T05:07:53.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f299009f430 con 0x7f299009b6c0 2024-09-16T05:07:53.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:53.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f299009e390 con 0x7f29900a15d0 2024-09-16T05:07:53.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:53.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 10 monclient(hunting): authenticate will time out at 958.392080s 2024-09-16T05:07:53.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f29900a15d0 0x7f299013a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f29900a15d0 0x7f299013a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55130/0 (socket says 172.21.15.114:55130) 2024-09-16T05:07:53.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298ffff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f29900a19e0 0x7f299013cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38712/0 (socket says 172.21.15.114:38712) 2024-09-16T05:07:53.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 1 -- 172.21.15.114:0/3385569067 learned_addr learned my addr 172.21.15.114:0/3385569067 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:53.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 10 monclient(hunting): get_auth_request con 0x7f29900a15d0 auth_method 0 2024-09-16T05:07:53.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:53.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:53.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:53.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:53.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:53.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:53.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 10 monclient(hunting): handle_auth_done global_id 4377 payload 274 2024-09-16T05:07:53.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 1 -- 172.21.15.114:0/3385569067 >> v1:172.21.15.149:6789/0 conn(0x7f299009b6c0 legacy=0x7f29900a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:53.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 1 -- 172.21.15.114:0/3385569067 >> v1:172.21.15.114:6789/0 conn(0x7f29900a19e0 legacy=0x7f299013cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298e7fc640 1 -- 172.21.15.114:0/3385569067 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2672393986 0 0) 0x7f299009d4e0 con 0x7f29900a19e0 2024-09-16T05:07:53.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 10 monclient: _finish_hunting 0 2024-09-16T05:07:53.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 1 monclient: found mon.noname-a 2024-09-16T05:07:53.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:53.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:53.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 1 -- 172.21.15.114:0/3385569067 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299009c2f0 con 0x7f29900a15d0 2024-09-16T05:07:53.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298e7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:07:53.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298effd640 1 --2- 172.21.15.114:0/3385569067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29900a15d0 0x7f299013a650 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2980006fd0 tx=0x7f298001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba398172c1e01074 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:53.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298e7fc640 1 -- 172.21.15.114:0/3385569067 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800b0070 con 0x7f29900a15d0 2024-09-16T05:07:53.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:53.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:53.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298e7fc640 10 monclient: dump: 2024-09-16T05:07:53.776 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:53.776 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:53.776 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.776 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.776 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:53.776 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.777 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:53.777 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:53.777 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:53.777 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:07:53.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:07:53.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:53.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298e7fc640 1 -- 172.21.15.114:0/3385569067 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29800ab600 con 0x7f29900a15d0 2024-09-16T05:07:53.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:53.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298e7fc640 1 -- 172.21.15.114:0/3385569067 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800ab9d0 con 0x7f29900a15d0 2024-09-16T05:07:53.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 5 monclient: authenticate success, global_id 4377 2024-09-16T05:07:53.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:53.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 4 set_mon_vals no callback set 2024-09-16T05:07:53.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:53.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:53.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f298e7fc640 10 monclient: dump: 2024-09-16T05:07:53.778 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:53.778 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:53.778 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.778 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.779 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:53.779 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.779 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:53.779 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:53.779 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:53.779 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 10 monclient: shutdown 2024-09-16T05:07:53.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:53.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 1 -- 172.21.15.114:0/3385569067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29900a15d0 msgr2=0x7f299013a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 1 --2- 172.21.15.114:0/3385569067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29900a15d0 0x7f299013a650 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2980006fd0 tx=0x7f298001c490 comp rx=0 tx=0).stop 2024-09-16T05:07:53.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 1 -- 172.21.15.114:0/3385569067 shutdown_connections 2024-09-16T05:07:53.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 1 --2- 172.21.15.114:0/3385569067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29900a15d0 0x7f299013a650 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 1 -- 172.21.15.114:0/3385569067 >> 172.21.15.114:0/3385569067 conn(0x7f29900a1fb0 msgr2=0x7f29900a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:53.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 1 -- 172.21.15.114:0/3385569067 shutdown_connections 2024-09-16T05:07:53.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.762+0000 7f2995ef4640 1 -- 172.21.15.114:0/3385569067 wait complete. 2024-09-16T05:07:53.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient: build_initial_monmap 2024-09-16T05:07:53.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient: monmap: 2024-09-16T05:07:53.780 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:53.780 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:53.780 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:53.780 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:53.780 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:53.780 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.780 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:53.781 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:53.781 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:53.781 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 librados: starting msgr at 2024-09-16T05:07:53.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 librados: starting objecter 2024-09-16T05:07:53.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 Processor -- start 2024-09-16T05:07:53.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 -- start start 2024-09-16T05:07:53.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 librados: setting wanted keys 2024-09-16T05:07:53.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 librados: calling monclient init 2024-09-16T05:07:53.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient: init 2024-09-16T05:07:53.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:53.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:07:53.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f299009b6c0 0x7f299013d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient(hunting): picked mon.noname-b con 0x7f299009b6c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:53.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29900a15d0 0x7f29900994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient(hunting): picked mon.noname-a con 0x7f29900a15d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:53.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f29900a19e0 0x7f29900999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient(hunting): picked mon.noname-c con 0x7f29900a19e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:53.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:53.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2990140c60 con 0x7f29900a15d0 2024-09-16T05:07:53.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:53.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f2990140ae0 con 0x7f299009b6c0 2024-09-16T05:07:53.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:53.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2990140de0 con 0x7f29900a19e0 2024-09-16T05:07:53.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:53.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient(hunting): authenticate will time out at 958.396982s 2024-09-16T05:07:53.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29900a15d0 0x7f29900994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29900a15d0 0x7f29900994b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55134/0 (socket says 172.21.15.114:55134) 2024-09-16T05:07:53.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 1 -- 172.21.15.114:0/2899088336 learned_addr learned my addr 172.21.15.114:0/2899088336 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:53.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 10 monclient(hunting): get_auth_request con 0x7f29900a15d0 auth_method 0 2024-09-16T05:07:53.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:53.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:53.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:53.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298ffff640 1 --2- 172.21.15.114:0/2899088336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f29900a19e0 0x7f29900999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298ffff640 10 monclient(hunting): get_auth_request con 0x7f29900a19e0 auth_method 0 2024-09-16T05:07:53.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:53.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:53.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:53.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:53.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:53.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:53.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:53.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:53.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:53.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298f7fe640 1 --2- 172.21.15.114:0/2899088336 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f299009b6c0 0x7f299013d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298f7fe640 10 monclient(hunting): get_auth_request con 0x7f299009b6c0 auth_method 0 2024-09-16T05:07:53.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:53.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:53.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:53.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 10 monclient(hunting): handle_auth_done global_id 4383 payload 824 2024-09-16T05:07:53.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 1 -- 172.21.15.114:0/2899088336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f29900a19e0 msgr2=0x7f29900999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 1 --2- 172.21.15.114:0/2899088336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f29900a19e0 0x7f29900999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 1 -- 172.21.15.114:0/2899088336 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f299009b6c0 msgr2=0x7f299013d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 1 --2- 172.21.15.114:0/2899088336 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f299009b6c0 0x7f299013d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 10 monclient: _finish_hunting 0 2024-09-16T05:07:53.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 1 monclient: found mon.noname-a 2024-09-16T05:07:53.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:53.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:53.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 1 -- 172.21.15.114:0/2899088336 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299013c6b0 con 0x7f29900a15d0 2024-09-16T05:07:53.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298effd640 1 --2- 172.21.15.114:0/2899088336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29900a15d0 0x7f29900994b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f298001a0c0 tx=0x7f2980019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:53.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 10 client.?.objecter ms_handle_connect 0x7f29900a15d0 2024-09-16T05:07:53.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:53.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 1 -- 172.21.15.114:0/2899088336 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800b0040 con 0x7f29900a15d0 2024-09-16T05:07:53.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:53.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:53.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 10 monclient: dump: 2024-09-16T05:07:53.786 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:53.786 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:53.787 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.787 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.787 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:53.787 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.787 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:53.787 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:53.787 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:53.788 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 10 monclient: _finish_auth 0 2024-09-16T05:07:53.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 10 monclient: _check_auth_tickets 2024-09-16T05:07:53.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:53.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 1 -- 172.21.15.114:0/2899088336 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2980005ba0 con 0x7f29900a15d0 2024-09-16T05:07:53.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:53.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 1 -- 172.21.15.114:0/2899088336 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29800ab7b0 con 0x7f29900a15d0 2024-09-16T05:07:53.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 5 monclient: authenticate success, global_id 4383 2024-09-16T05:07:53.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient: _renew_subs 2024-09-16T05:07:53.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:53.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 -- 172.21.15.114:0/2899088336 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299009b490 con 0x7f29900a15d0 2024-09-16T05:07:53.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:53.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:53.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 10 monclient: dump: 2024-09-16T05:07:53.789 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:53.789 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:53.789 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.789 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:53.789 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:53.789 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:53.789 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:53.789 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:53.790 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:53.790 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:53.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298dffb640 4 set_mon_vals no callback set 2024-09-16T05:07:53.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:53.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient: _renew_subs 2024-09-16T05:07:53.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:53.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 -- 172.21.15.114:0/2899088336 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2990072a30 con 0x7f29900a15d0 2024-09-16T05:07:53.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 librados: init done 2024-09-16T05:07:53.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 1 -- 172.21.15.114:0/2899088336 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f29800ab950 con 0x7f29900a15d0 2024-09-16T05:07:53.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:53.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:53.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:53.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2995ef4640 1 -- 172.21.15.114:0/2899088336 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f299006de70 con 0x7f29900a15d0 2024-09-16T05:07:53.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 1 --2- 172.21.15.114:0/2899088336 >> v2:172.21.15.149:6812/1185531135 conn(0x7f297c037810 0x7f297c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:53.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 1 -- 172.21.15.114:0/2899088336 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f29800e2110 con 0x7f29900a15d0 2024-09-16T05:07:53.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 10 client.4383.objecter ms_dispatch 0x7f2990000ba0 osd_map(19..19 src has 1..19) 2024-09-16T05:07:53.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 3 client.4383.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T05:07:53.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 3 client.4383.objecter handle_osd_map decoding full epoch 19 2024-09-16T05:07:53.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 20 client.4383.objecter dump_active .. 0 homeless 2024-09-16T05:07:53.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298f7fe640 1 --2- 172.21.15.114:0/2899088336 >> v2:172.21.15.149:6812/1185531135 conn(0x7f297c037810 0x7f297c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:53.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298f7fe640 10 monclient: get_auth_request con 0x7f297c037810 auth_method 0 2024-09-16T05:07:53.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f298f7fe640 1 --2- 172.21.15.114:0/2899088336 >> v2:172.21.15.149:6812/1185531135 conn(0x7f297c037810 0x7f297c039cd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2984000f70 tx=0x7f2984009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:53.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.766+0000 7f2977fff640 10 client.4383.objecter ms_handle_connect 0x7f297c037810 2024-09-16T05:07:53.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.770+0000 7f2977fff640 1 -- 172.21.15.114:0/2899088336 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2980016110 con 0x7f29900a15d0 2024-09-16T05:07:53.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.770+0000 7f2977fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:53.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.770+0000 7f2977fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:53.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.922+0000 7f2995ef4640 1 -- 172.21.15.114:0/2899088336 --> v2:172.21.15.149:6812/1185531135 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f29900a0ec0 con 0x7f297c037810 2024-09-16T05:07:53.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.926+0000 7f2977fff640 1 -- 172.21.15.114:0/2899088336 <== mgr.4108 v2:172.21.15.149:6812/1185531135 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34148 (secure 0 0 0) 0x7f29900a0ec0 con 0x7f297c037810 2024-09-16T05:07:53.928 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:07:53.928 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-16T05:07:53.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 10 librados: watch_flush enter 2024-09-16T05:07:53.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 10 librados: watch_flush exit 2024-09-16T05:07:53.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 20 client.4383.objecter shutdown clearing up homeless session... 2024-09-16T05:07:53.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 10 client.4383.objecter successfully canceled tick 2024-09-16T05:07:53.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 1 -- 172.21.15.114:0/2899088336 >> v2:172.21.15.149:6812/1185531135 conn(0x7f297c037810 msgr2=0x7f297c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 1 --2- 172.21.15.114:0/2899088336 >> v2:172.21.15.149:6812/1185531135 conn(0x7f297c037810 0x7f297c039cd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2984000f70 tx=0x7f2984009040 comp rx=0 tx=0).stop 2024-09-16T05:07:53.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 10 monclient: shutdown 2024-09-16T05:07:53.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:53.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 1 -- 172.21.15.114:0/2899088336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29900a15d0 msgr2=0x7f29900994b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:53.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 1 --2- 172.21.15.114:0/2899088336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29900a15d0 0x7f29900994b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f298001a0c0 tx=0x7f2980019230 comp rx=0 tx=0).stop 2024-09-16T05:07:53.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 1 -- 172.21.15.114:0/2899088336 shutdown_connections 2024-09-16T05:07:53.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 1 --2- 172.21.15.114:0/2899088336 >> v2:172.21.15.149:6812/1185531135 conn(0x7f297c037810 0x7f297c039cd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 1 --2- 172.21.15.114:0/2899088336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29900a15d0 0x7f29900994b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 1 --2- 172.21.15.114:0/2899088336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f29900a19e0 0x7f29900999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 1 --2- 172.21.15.114:0/2899088336 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f299009b6c0 0x7f299013d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:53.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 1 -- 172.21.15.114:0/2899088336 >> 172.21.15.114:0/2899088336 conn(0x7f29900a1fb0 msgr2=0x7f29900a4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:53.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 1 -- 172.21.15.114:0/2899088336 shutdown_connections 2024-09-16T05:07:53.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 1 -- 172.21.15.114:0/2899088336 wait complete. 2024-09-16T05:07:53.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:53.930+0000 7f2995ef4640 1 librados: shutdown 2024-09-16T05:07:53.945 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T05:07:52.071427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164156,"kb_used_data":1960,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066884,"statfs":{"total":579820584960,"available":579652489216,"internally_reserved":0,"allocated":2007040,"data_stored":1350460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.835801"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.267914+0000","last_change":"2024-09-16T05:07:48.267914+0000","last_active":"2024-09-16T05:07:48.267914+0000","last_peered":"2024-09-16T05:07:48.267914+0000","last_clean":"2024-09-16T05:07:48.267914+0000","last_became_active":"2024-09-16T05:07:46.254034+0000","last_became_peered":"2024-09-16T05:07:46.254034+0000","last_unstale":"2024-09-16T05:07:48.267914+0000","last_undegraded":"2024-09-16T05:07:48.267914+0000","last_fullsized":"2024-09-16T05:07:48.267914+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:57:37.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014490219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.275161+0000","last_change":"2024-09-16T05:07:48.275545+0000","last_active":"2024-09-16T05:07:48.275161+0000","last_peered":"2024-09-16T05:07:48.275161+0000","last_clean":"2024-09-16T05:07:48.275161+0000","last_became_active":"2024-09-16T05:07:46.252207+0000","last_became_peered":"2024-09-16T05:07:46.252207+0000","last_unstale":"2024-09-16T05:07:48.275161+0000","last_undegraded":"2024-09-16T05:07:48.275161+0000","last_fullsized":"2024-09-16T05:07:48.275161+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:36:08.378166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073333700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.267372+0000","last_change":"2024-09-16T05:07:48.267372+0000","last_active":"2024-09-16T05:07:48.267372+0000","last_peered":"2024-09-16T05:07:48.267372+0000","last_clean":"2024-09-16T05:07:48.267372+0000","last_became_active":"2024-09-16T05:07:46.243891+0000","last_became_peered":"2024-09-16T05:07:46.243891+0000","last_unstale":"2024-09-16T05:07:48.267372+0000","last_undegraded":"2024-09-16T05:07:48.267372+0000","last_fullsized":"2024-09-16T05:07:48.267372+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:48:25.061723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011829709999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.352028+0000","last_change":"2024-09-16T05:07:48.352028+0000","last_active":"2024-09-16T05:07:48.352028+0000","last_peered":"2024-09-16T05:07:48.352028+0000","last_clean":"2024-09-16T05:07:48.352028+0000","last_became_active":"2024-09-16T05:07:46.255633+0000","last_became_peered":"2024-09-16T05:07:46.255633+0000","last_unstale":"2024-09-16T05:07:48.352028+0000","last_undegraded":"2024-09-16T05:07:48.352028+0000","last_fullsized":"2024-09-16T05:07:48.352028+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:07.079219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000883379,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.275745+0000","last_change":"2024-09-16T05:07:48.274329+0000","last_active":"2024-09-16T05:07:48.275745+0000","last_peered":"2024-09-16T05:07:48.275745+0000","last_clean":"2024-09-16T05:07:48.275745+0000","last_became_active":"2024-09-16T05:07:46.257118+0000","last_became_peered":"2024-09-16T05:07:46.257118+0000","last_unstale":"2024-09-16T05:07:48.275745+0000","last_undegraded":"2024-09-16T05:07:48.275745+0000","last_fullsized":"2024-09-16T05:07:48.275745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:35.932772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020216650000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.263044+0000","last_change":"2024-09-16T05:07:48.263513+0000","last_active":"2024-09-16T05:07:48.263044+0000","last_peered":"2024-09-16T05:07:48.263044+0000","last_clean":"2024-09-16T05:07:48.263044+0000","last_became_active":"2024-09-16T05:07:46.252254+0000","last_became_peered":"2024-09-16T05:07:46.252254+0000","last_unstale":"2024-09-16T05:07:48.263044+0000","last_undegraded":"2024-09-16T05:07:48.263044+0000","last_fullsized":"2024-09-16T05:07:48.263044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:33.736560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081560699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.266504+0000","last_change":"2024-09-16T05:07:48.266504+0000","last_active":"2024-09-16T05:07:48.266504+0000","last_peered":"2024-09-16T05:07:48.266504+0000","last_clean":"2024-09-16T05:07:48.266504+0000","last_became_active":"2024-09-16T05:07:46.254322+0000","last_became_peered":"2024-09-16T05:07:46.254322+0000","last_unstale":"2024-09-16T05:07:48.266504+0000","last_undegraded":"2024-09-16T05:07:48.266504+0000","last_fullsized":"2024-09-16T05:07:48.266504+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:00.175590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093680599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.267201+0000","last_change":"2024-09-16T05:07:48.267201+0000","last_active":"2024-09-16T05:07:48.267201+0000","last_peered":"2024-09-16T05:07:48.267201+0000","last_clean":"2024-09-16T05:07:48.267201+0000","last_became_active":"2024-09-16T05:07:46.246409+0000","last_became_peered":"2024-09-16T05:07:46.246409+0000","last_unstale":"2024-09-16T05:07:48.267201+0000","last_undegraded":"2024-09-16T05:07:48.267201+0000","last_fullsized":"2024-09-16T05:07:48.267201+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:13.808827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010413149999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.265942+0000","last_change":"2024-09-16T05:07:39.580660+0000","last_active":"2024-09-16T05:07:48.265942+0000","last_peered":"2024-09-16T05:07:48.265942+0000","last_clean":"2024-09-16T05:07:48.265942+0000","last_became_active":"2024-09-16T05:07:39.576756+0000","last_became_peered":"2024-09-16T05:07:39.576756+0000","last_unstale":"2024-09-16T05:07:48.265942+0000","last_undegraded":"2024-09-16T05:07:48.265942+0000","last_fullsized":"2024-09-16T05:07:48.265942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_clean_scrub_stamp":"2024-09-16T05:07:38.167879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:57.060906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":48761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":48780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":688128,"data_stored":577689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":688128,"data_stored":577689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":159744,"data_stored":48780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":48761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:07:53.947 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:07:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.062+0000 7fa17d4ea640 10 monclient: get_monmap_and_config 2024-09-16T05:07:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.062+0000 7fa17d4ea640 10 monclient: build_initial_monmap 2024-09-16T05:07:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.062+0000 7fa17d4ea640 10 monclient: monmap: 2024-09-16T05:07:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:54.067279+0000 2024-09-16T05:07:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:54.067279+0000 2024-09-16T05:07:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:54.066 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:54.066 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:54.066 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:54.066 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:54.066 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 1 Processor -- start 2024-09-16T05:07:54.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 1 -- start start 2024-09-16T05:07:54.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 10 monclient: init 2024-09-16T05:07:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T05:07:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fa17809b6c0 0x7fa1780a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 10 monclient(hunting): picked mon.noname-c con 0x7fa17809b6c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa1780a1400 0x7fa1780a1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 10 monclient(hunting): picked mon.noname-a con 0x7fa1780a1400 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa178139640 0x7fa17813ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 10 monclient(hunting): picked mon.noname-b con 0x7fa178139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:54.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa17809e390 con 0x7fa1780a1400 2024-09-16T05:07:54.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fa1780a1d50 con 0x7fa17809b6c0 2024-09-16T05:07:54.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa17809ece0 con 0x7fa178139640 2024-09-16T05:07:54.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:54.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 10 monclient(hunting): authenticate will time out at 958.696870s 2024-09-16T05:07:54.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa178139640 0x7fa17813ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1767fc640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa1780a1400 0x7fa1780a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa178139640 0x7fa17813ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55812/0 (socket says 172.21.15.114:55812) 2024-09-16T05:07:54.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1767fc640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa1780a1400 0x7fa1780a1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55140/0 (socket says 172.21.15.114:55140) 2024-09-16T05:07:54.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 1 -- 172.21.15.114:0/141014843 learned_addr learned my addr 172.21.15.114:0/141014843 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:54.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 10 monclient(hunting): get_auth_request con 0x7fa178139640 auth_method 0 2024-09-16T05:07:54.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1767fc640 10 monclient(hunting): get_auth_request con 0x7fa1780a1400 auth_method 0 2024-09-16T05:07:54.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa176ffd640 1 --2- 172.21.15.114:0/141014843 >> v2:172.21.15.149:3300/0 conn(0x7fa17809b6c0 0x7fa1780a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa176ffd640 10 monclient(hunting): get_auth_request con 0x7fa17809b6c0 auth_method 0 2024-09-16T05:07:54.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa176ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa176ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa176ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 10 monclient(hunting): handle_auth_done global_id 4313 payload 274 2024-09-16T05:07:54.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 1 -- 172.21.15.114:0/141014843 >> v2:172.21.15.149:3300/0 conn(0x7fa17809b6c0 msgr2=0x7fa1780a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:54.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 1 --2- 172.21.15.114:0/141014843 >> v2:172.21.15.149:3300/0 conn(0x7fa17809b6c0 0x7fa1780a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 1 -- 172.21.15.114:0/141014843 >> v2:172.21.15.114:3300/0 conn(0x7fa1780a1400 msgr2=0x7fa1780a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:54.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 1 --2- 172.21.15.114:0/141014843 >> v2:172.21.15.114:3300/0 conn(0x7fa1780a1400 0x7fa1780a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 10 monclient: _finish_hunting 0 2024-09-16T05:07:54.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 1 monclient: found mon.noname-b 2024-09-16T05:07:54.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:54.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:54.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 1 -- 172.21.15.114:0/141014843 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa17809d4e0 con 0x7fa178139640 2024-09-16T05:07:54.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1767fc640 1 --2- 172.21.15.114:0/141014843 >> v2:172.21.15.114:3300/0 conn(0x7fa1780a1400 0x7fa1780a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:54.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa1777fe640 1 --2- 172.21.15.114:0/141014843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa178139640 0x7fa17813ba30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa1780a2f70 tx=0x7fa17001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de91a9455fbcf361 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa175ffb640 1 -- 172.21.15.114:0/141014843 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1700a5530 con 0x7fa178139640 2024-09-16T05:07:54.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa175ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa175ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:54.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa175ffb640 10 monclient: dump: 2024-09-16T05:07:54.086 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.086 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.086 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.087 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.087 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.087 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.087 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.087 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.087 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.087 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa175ffb640 10 monclient: _finish_auth 0 2024-09-16T05:07:54.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa175ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:07:54.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa175ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:54.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa175ffb640 1 -- 172.21.15.114:0/141014843 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1700a56d0 con 0x7fa178139640 2024-09-16T05:07:54.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 5 monclient: authenticate success, global_id 4313 2024-09-16T05:07:54.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:54.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa175ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:54.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:54.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa175ffb640 1 -- 172.21.15.114:0/141014843 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1700a5aa0 con 0x7fa178139640 2024-09-16T05:07:54.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa175ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa17d4ea640 4 set_mon_vals no callback set 2024-09-16T05:07:54.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa175ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:54.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.066+0000 7fa175ffb640 10 monclient: dump: 2024-09-16T05:07:54.088 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.088 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.088 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.088 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.088 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.088 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.088 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.088 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.089 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.089 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient: shutdown 2024-09-16T05:07:54.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:54.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 -- 172.21.15.114:0/141014843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa178139640 msgr2=0x7fa17813ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 --2- 172.21.15.114:0/141014843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa178139640 0x7fa17813ba30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa1780a2f70 tx=0x7fa17001c8b0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 -- 172.21.15.114:0/141014843 shutdown_connections 2024-09-16T05:07:54.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 --2- 172.21.15.114:0/141014843 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa178139640 0x7fa17813ba30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 --2- 172.21.15.114:0/141014843 >> v2:172.21.15.114:3300/0 conn(0x7fa1780a1400 0x7fa1780a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 --2- 172.21.15.114:0/141014843 >> v2:172.21.15.149:3300/0 conn(0x7fa17809b6c0 0x7fa1780a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 -- 172.21.15.114:0/141014843 >> 172.21.15.114:0/141014843 conn(0x7fa1780a1fb0 msgr2=0x7fa1780a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:54.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 -- 172.21.15.114:0/141014843 shutdown_connections 2024-09-16T05:07:54.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 -- 172.21.15.114:0/141014843 wait complete. 2024-09-16T05:07:54.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient: build_initial_monmap 2024-09-16T05:07:54.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient: monmap: 2024-09-16T05:07:54.090 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:54.090 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:54.090 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:54.091 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:54.091 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:54.091 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.091 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:54.091 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:54.091 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:54.091 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 librados: starting msgr at 2024-09-16T05:07:54.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 librados: starting objecter 2024-09-16T05:07:54.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 Processor -- start 2024-09-16T05:07:54.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 -- start start 2024-09-16T05:07:54.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 librados: setting wanted keys 2024-09-16T05:07:54.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 librados: calling monclient init 2024-09-16T05:07:54.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient: init 2024-09-16T05:07:54.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:54.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:07:54.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa17809b6c0 0x7fa1781ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient(hunting): picked mon.noname-b con 0x7fa17809b6c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:54.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1780a1400 0x7fa1780984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient(hunting): picked mon.noname-c con 0x7fa1780a1400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:54.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa178139640 0x7fa178098a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient(hunting): picked mon.noname-a con 0x7fa178139640 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:54.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa17813e790 con 0x7fa178139640 2024-09-16T05:07:54.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fa17809cad0 con 0x7fa17809b6c0 2024-09-16T05:07:54.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa17813e910 con 0x7fa1780a1400 2024-09-16T05:07:54.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:54.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient(hunting): authenticate will time out at 958.701643s 2024-09-16T05:07:54.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1780a1400 0x7fa1780984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1780a1400 0x7fa1780984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55826/0 (socket says 172.21.15.114:55826) 2024-09-16T05:07:54.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 1 -- 172.21.15.114:0/2316810866 learned_addr learned my addr 172.21.15.114:0/2316810866 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:54.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 10 monclient(hunting): get_auth_request con 0x7fa1780a1400 auth_method 0 2024-09-16T05:07:54.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1777fe640 1 --2- 172.21.15.114:0/2316810866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa178139640 0x7fa178098a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1777fe640 10 monclient(hunting): get_auth_request con 0x7fa178139640 auth_method 0 2024-09-16T05:07:54.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-09-16T05:07:54.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa176ffd640 1 --2- 172.21.15.114:0/2316810866 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa17809b6c0 0x7fa1781ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 1 -- 172.21.15.114:0/2316810866 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa17809b6c0 msgr2=0x7fa1781ccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 1 --2- 172.21.15.114:0/2316810866 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa17809b6c0 0x7fa1781ccc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 1 -- 172.21.15.114:0/2316810866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa178139640 msgr2=0x7fa178098a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 1 --2- 172.21.15.114:0/2316810866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa178139640 0x7fa178098a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 10 monclient: _finish_hunting 0 2024-09-16T05:07:54.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 1 monclient: found mon.noname-c 2024-09-16T05:07:54.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:54.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:54.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 1 -- 172.21.15.114:0/2316810866 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1780906a0 con 0x7fa1780a1400 2024-09-16T05:07:54.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1777fe640 1 --2- 172.21.15.114:0/2316810866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa178139640 0x7fa178098a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:54.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1767fc640 1 --2- 172.21.15.114:0/2316810866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1780a1400 0x7fa1780984d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00ebe0 tx=0x7fa16c00c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 10 client.?.objecter ms_handle_connect 0x7fa1780a1400 2024-09-16T05:07:54.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:54.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 1 -- 172.21.15.114:0/2316810866 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c00cc30 con 0x7fa1780a1400 2024-09-16T05:07:54.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:54.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 10 monclient: dump: 2024-09-16T05:07:54.096 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.096 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.096 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.096 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.096 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.096 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.096 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.096 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.096 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.097 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:07:54.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:07:54.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:54.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 1 -- 172.21.15.114:0/2316810866 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa16c00cdd0 con 0x7fa1780a1400 2024-09-16T05:07:54.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 5 monclient: authenticate success, global_id 4316 2024-09-16T05:07:54.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient: _renew_subs 2024-09-16T05:07:54.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:54.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 -- 172.21.15.114:0/2316810866 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa178090980 con 0x7fa1780a1400 2024-09-16T05:07:54.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:54.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 1 -- 172.21.15.114:0/2316810866 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c005bc0 con 0x7fa1780a1400 2024-09-16T05:07:54.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:54.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 10 monclient: dump: 2024-09-16T05:07:54.098 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.098 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.099 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient: _renew_subs 2024-09-16T05:07:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 -- 172.21.15.114:0/2316810866 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa17808eda0 con 0x7fa1780a1400 2024-09-16T05:07:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 librados: init done 2024-09-16T05:07:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa1757fa640 4 set_mon_vals no callback set 2024-09-16T05:07:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 1 -- 172.21.15.114:0/2316810866 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa16c006030 con 0x7fa1780a1400 2024-09-16T05:07:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa17d4ea640 1 -- 172.21.15.114:0/2316810866 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1780008e0 con 0x7fa1780a1400 2024-09-16T05:07:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 1 --2- 172.21.15.114:0/2316810866 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa164037810 0x7fa164039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 1 -- 172.21.15.114:0/2316810866 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7fa16c048c20 con 0x7fa1780a1400 2024-09-16T05:07:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 10 client.4316.objecter ms_dispatch 0x7fa178000ba0 osd_map(19..19 src has 1..19) 2024-09-16T05:07:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 3 client.4316.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T05:07:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 3 client.4316.objecter handle_osd_map decoding full epoch 19 2024-09-16T05:07:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa15f7fe640 20 client.4316.objecter dump_active .. 0 homeless 2024-09-16T05:07:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa176ffd640 1 --2- 172.21.15.114:0/2316810866 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa164037810 0x7fa164039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.070+0000 7fa176ffd640 10 monclient: get_auth_request con 0x7fa164037810 auth_method 0 2024-09-16T05:07:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.074+0000 7fa15f7fe640 10 client.4316.objecter ms_handle_connect 0x7fa164037810 2024-09-16T05:07:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.074+0000 7fa176ffd640 1 --2- 172.21.15.114:0/2316810866 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa164037810 0x7fa164039cd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa1680096f0 tx=0x7fa168008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.074+0000 7fa15f7fe640 1 -- 172.21.15.114:0/2316810866 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fa16c017020 con 0x7fa1780a1400 2024-09-16T05:07:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.074+0000 7fa15f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.074+0000 7fa15f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:54.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.222+0000 7fa17d4ea640 1 -- 172.21.15.114:0/2316810866 --> v2:172.21.15.149:6812/1185531135 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa178090d80 con 0x7fa164037810 2024-09-16T05:07:54.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.226+0000 7fa15f7fe640 1 -- 172.21.15.114:0/2316810866 <== mgr.4108 v2:172.21.15.149:6812/1185531135 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34096 (secure 0 0 0) 0x7fa16800e4c0 con 0x7fa164037810 2024-09-16T05:07:54.228 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:07:54.228 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-16T05:07:54.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 10 librados: watch_flush enter 2024-09-16T05:07:54.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 10 librados: watch_flush exit 2024-09-16T05:07:54.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 20 client.4316.objecter shutdown clearing up homeless session... 2024-09-16T05:07:54.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 10 client.4316.objecter successfully canceled tick 2024-09-16T05:07:54.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 1 -- 172.21.15.114:0/2316810866 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa164037810 msgr2=0x7fa164039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 1 --2- 172.21.15.114:0/2316810866 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa164037810 0x7fa164039cd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa1680096f0 tx=0x7fa168008040 comp rx=0 tx=0).stop 2024-09-16T05:07:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 10 monclient: shutdown 2024-09-16T05:07:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 1 -- 172.21.15.114:0/2316810866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1780a1400 msgr2=0x7fa1780984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 1 --2- 172.21.15.114:0/2316810866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1780a1400 0x7fa1780984d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00ebe0 tx=0x7fa16c00c520 comp rx=0 tx=0).stop 2024-09-16T05:07:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 1 -- 172.21.15.114:0/2316810866 shutdown_connections 2024-09-16T05:07:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 1 --2- 172.21.15.114:0/2316810866 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa164037810 0x7fa164039cd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 1 --2- 172.21.15.114:0/2316810866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa178139640 0x7fa178098a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 1 --2- 172.21.15.114:0/2316810866 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1780a1400 0x7fa1780984d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 1 --2- 172.21.15.114:0/2316810866 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa17809b6c0 0x7fa1781ccc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 1 -- 172.21.15.114:0/2316810866 >> 172.21.15.114:0/2316810866 conn(0x7fa1780a1fb0 msgr2=0x7fa178137850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 1 -- 172.21.15.114:0/2316810866 shutdown_connections 2024-09-16T05:07:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 1 -- 172.21.15.114:0/2316810866 wait complete. 2024-09-16T05:07:54.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.230+0000 7fa17d4ea640 1 librados: shutdown 2024-09-16T05:07:54.244 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T05:07:54.071948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164160,"kb_used_data":1976,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066880,"statfs":{"total":579820584960,"available":579652485120,"internally_reserved":0,"allocated":2023424,"data_stored":1357984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.834053"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.267914+0000","last_change":"2024-09-16T05:07:48.267914+0000","last_active":"2024-09-16T05:07:48.267914+0000","last_peered":"2024-09-16T05:07:48.267914+0000","last_clean":"2024-09-16T05:07:48.267914+0000","last_became_active":"2024-09-16T05:07:46.254034+0000","last_became_peered":"2024-09-16T05:07:46.254034+0000","last_unstale":"2024-09-16T05:07:48.267914+0000","last_undegraded":"2024-09-16T05:07:48.267914+0000","last_fullsized":"2024-09-16T05:07:48.267914+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:57:37.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014490219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:07:52.243323+0000","last_change":"2024-09-16T05:07:48.275545+0000","last_active":"2024-09-16T05:07:52.243323+0000","last_peered":"2024-09-16T05:07:52.243323+0000","last_clean":"2024-09-16T05:07:52.243323+0000","last_became_active":"2024-09-16T05:07:46.252207+0000","last_became_peered":"2024-09-16T05:07:46.252207+0000","last_unstale":"2024-09-16T05:07:52.243323+0000","last_undegraded":"2024-09-16T05:07:52.243323+0000","last_fullsized":"2024-09-16T05:07:52.243323+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:36:08.378166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073333700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.267372+0000","last_change":"2024-09-16T05:07:48.267372+0000","last_active":"2024-09-16T05:07:48.267372+0000","last_peered":"2024-09-16T05:07:48.267372+0000","last_clean":"2024-09-16T05:07:48.267372+0000","last_became_active":"2024-09-16T05:07:46.243891+0000","last_became_peered":"2024-09-16T05:07:46.243891+0000","last_unstale":"2024-09-16T05:07:48.267372+0000","last_undegraded":"2024-09-16T05:07:48.267372+0000","last_fullsized":"2024-09-16T05:07:48.267372+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:48:25.061723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011829709999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.352028+0000","last_change":"2024-09-16T05:07:48.352028+0000","last_active":"2024-09-16T05:07:48.352028+0000","last_peered":"2024-09-16T05:07:48.352028+0000","last_clean":"2024-09-16T05:07:48.352028+0000","last_became_active":"2024-09-16T05:07:46.255633+0000","last_became_peered":"2024-09-16T05:07:46.255633+0000","last_unstale":"2024-09-16T05:07:48.352028+0000","last_undegraded":"2024-09-16T05:07:48.352028+0000","last_fullsized":"2024-09-16T05:07:48.352028+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:07.079219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000883379,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.275745+0000","last_change":"2024-09-16T05:07:48.274329+0000","last_active":"2024-09-16T05:07:48.275745+0000","last_peered":"2024-09-16T05:07:48.275745+0000","last_clean":"2024-09-16T05:07:48.275745+0000","last_became_active":"2024-09-16T05:07:46.257118+0000","last_became_peered":"2024-09-16T05:07:46.257118+0000","last_unstale":"2024-09-16T05:07:48.275745+0000","last_undegraded":"2024-09-16T05:07:48.275745+0000","last_fullsized":"2024-09-16T05:07:48.275745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:35.932772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020216650000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:07:52.024202+0000","last_change":"2024-09-16T05:07:48.263513+0000","last_active":"2024-09-16T05:07:52.024202+0000","last_peered":"2024-09-16T05:07:52.024202+0000","last_clean":"2024-09-16T05:07:52.024202+0000","last_became_active":"2024-09-16T05:07:46.252254+0000","last_became_peered":"2024-09-16T05:07:46.252254+0000","last_unstale":"2024-09-16T05:07:52.024202+0000","last_undegraded":"2024-09-16T05:07:52.024202+0000","last_fullsized":"2024-09-16T05:07:52.024202+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:33.736560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081560699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.266504+0000","last_change":"2024-09-16T05:07:48.266504+0000","last_active":"2024-09-16T05:07:48.266504+0000","last_peered":"2024-09-16T05:07:48.266504+0000","last_clean":"2024-09-16T05:07:48.266504+0000","last_became_active":"2024-09-16T05:07:46.254322+0000","last_became_peered":"2024-09-16T05:07:46.254322+0000","last_unstale":"2024-09-16T05:07:48.266504+0000","last_undegraded":"2024-09-16T05:07:48.266504+0000","last_fullsized":"2024-09-16T05:07:48.266504+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:00.175590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093680599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.267201+0000","last_change":"2024-09-16T05:07:48.267201+0000","last_active":"2024-09-16T05:07:48.267201+0000","last_peered":"2024-09-16T05:07:48.267201+0000","last_clean":"2024-09-16T05:07:48.267201+0000","last_became_active":"2024-09-16T05:07:46.246409+0000","last_became_peered":"2024-09-16T05:07:46.246409+0000","last_unstale":"2024-09-16T05:07:48.267201+0000","last_undegraded":"2024-09-16T05:07:48.267201+0000","last_fullsized":"2024-09-16T05:07:48.267201+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:13.808827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010413149999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T05:07:48.265942+0000","last_change":"2024-09-16T05:07:39.580660+0000","last_active":"2024-09-16T05:07:48.265942+0000","last_peered":"2024-09-16T05:07:48.265942+0000","last_clean":"2024-09-16T05:07:48.265942+0000","last_became_active":"2024-09-16T05:07:39.576756+0000","last_became_peered":"2024-09-16T05:07:39.576756+0000","last_unstale":"2024-09-16T05:07:48.265942+0000","last_undegraded":"2024-09-16T05:07:48.265942+0000","last_fullsized":"2024-09-16T05:07:48.265942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_clean_scrub_stamp":"2024-09-16T05:07:38.167879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:57.060906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":48761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":48780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":688128,"data_stored":577689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":688128,"data_stored":577689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":52542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":52523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:07:54.245 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-16T05:07:54.246 DEBUG:tasks.mon_thrash:Yielding 2024-09-16T05:07:54.246 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T05:07:54.259 INFO:tasks.workunit:Pulling workunits from ref ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T05:07:54.260 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T05:07:54.260 DEBUG:teuthology.orchestra.run.smithi149:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T05:07:54.261 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-16T05:07:54.261 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726463274, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-16T05:07:54.261 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:07:54.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:07:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:07:54.267 INFO:teuthology.orchestra.run.smithi149.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T05:07:54.268 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T05:07:54.315 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T05:07:54.315 DEBUG:teuthology.orchestra.run.smithi149:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T05:07:54.366 INFO:tasks.workunit:timeout=3h 2024-09-16T05:07:54.366 INFO:tasks.workunit:cleanup=True 2024-09-16T05:07:54.367 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T05:07:54.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.378+0000 7fe34f782640 10 monclient: get_monmap_and_config 2024-09-16T05:07:54.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.378+0000 7fe34f782640 10 monclient: build_initial_monmap 2024-09-16T05:07:54.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.378+0000 7fe34f782640 10 monclient: monmap: 2024-09-16T05:07:54.380 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:54.380 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.380 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:54.383235+0000 2024-09-16T05:07:54.380 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:54.383235+0000 2024-09-16T05:07:54.380 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:54.380 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.380 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:54.380 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:54.380 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:54.380 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:54.381 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:54.381 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.378+0000 7fe34f782640 1 Processor -- start 2024-09-16T05:07:54.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.378+0000 7fe34f782640 1 -- start start 2024-09-16T05:07:54.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.378+0000 7fe34f782640 10 monclient: init 2024-09-16T05:07:54.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34f782640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:54.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34f782640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T05:07:54.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34f782640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe348071da0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:54.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34f782640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe3480a1ab0 0x7fe348072890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34f782640 10 monclient(hunting): picked mon.noname-c con 0x7fe3480a1ab0 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:54.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34f782640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe348099cc0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:54.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34f782640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe348099910 con 0x7fe348071da0 2024-09-16T05:07:54.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34f782640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3480a1870 con 0x7fe348099cc0 2024-09-16T05:07:54.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34f782640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34f782640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fe348073e20 con 0x7fe3480a1ab0 2024-09-16T05:07:54.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34f782640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:54.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34f782640 10 monclient(hunting): authenticate will time out at 959.012319s 2024-09-16T05:07:54.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34d51e640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe348071da0 0x7fe348072180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38722/0 (socket says 172.21.15.114:38722) 2024-09-16T05:07:54.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34d51e640 1 -- 172.21.15.114:0/2735564451 learned_addr learned my addr 172.21.15.114:0/2735564451 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:54.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 1 --2- 172.21.15.114:0/2735564451 >> v2:172.21.15.149:3300/0 conn(0x7fe3480a1ab0 0x7fe348072890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 -- 172.21.15.114:0/2735564451 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2153162111 0 0) 0x7fe348099910 con 0x7fe348071da0 2024-09-16T05:07:54.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:07:54.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient(hunting): my global_id is 4392 2024-09-16T05:07:54.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 -- 172.21.15.114:0/2735564451 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe334003530 con 0x7fe348071da0 2024-09-16T05:07:54.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient(hunting): get_auth_request con 0x7fe3480a1ab0 auth_method 0 2024-09-16T05:07:54.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 -- 172.21.15.114:0/2735564451 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4028154327 0 0) 0x7fe334003530 con 0x7fe348071da0 2024-09-16T05:07:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 -- 172.21.15.114:0/2735564451 >> v2:172.21.15.149:3300/0 conn(0x7fe3480a1ab0 msgr2=0x7fe348072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 --2- 172.21.15.114:0/2735564451 >> v2:172.21.15.149:3300/0 conn(0x7fe3480a1ab0 0x7fe348072890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 -- 172.21.15.114:0/2735564451 >> v1:172.21.15.149:6789/0 conn(0x7fe348099cc0 legacy=0x7fe34813e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:07:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:07:54.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:54.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:07:54.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 -- 172.21.15.114:0/2735564451 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3480a1120 con 0x7fe348071da0 2024-09-16T05:07:54.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient: _finish_auth 0 2024-09-16T05:07:54.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:07:54.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:54.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 -- 172.21.15.114:0/2735564451 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7fe344001680 con 0x7fe348071da0 2024-09-16T05:07:54.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:07:54.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient: dump: 2024-09-16T05:07:54.390 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.390 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.390 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.390 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.390 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.390 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.390 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.390 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.390 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.390 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:07:54.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:54.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 -- 172.21.15.114:0/2735564451 >> v1:172.21.15.114:6789/0 conn(0x7fe348071da0 legacy=0x7fe348072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:07:54.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 --2- 172.21.15.114:0/2735564451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3340039c0 0x7fe334003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient(hunting): picked mon.a con 0x7fe3340039c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:54.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 --2- 172.21.15.114:0/2735564451 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe334004810 0x7fe334004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient(hunting): picked mon.c con 0x7fe334004810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:54.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 --2- 172.21.15.114:0/2735564451 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe3340052f0 0x7fe33400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient(hunting): picked mon.b con 0x7fe3340052f0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:54.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 -- 172.21.15.114:0/2735564451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe348073e20 con 0x7fe3340039c0 2024-09-16T05:07:54.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 -- 172.21.15.114:0/2735564451 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fe33400c1c0 con 0x7fe3340052f0 2024-09-16T05:07:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 -- 172.21.15.114:0/2735564451 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe33400c3c0 con 0x7fe334004810 2024-09-16T05:07:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:07:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 1 --2- 172.21.15.114:0/2735564451 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe334004810 0x7fe334004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient(hunting): get_auth_request con 0x7fe334004810 auth_method 0 2024-09-16T05:07:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34d51e640 1 --2- 172.21.15.114:0/2735564451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3340039c0 0x7fe334003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34d51e640 10 monclient(hunting): get_auth_request con 0x7fe3340039c0 auth_method 0 2024-09-16T05:07:54.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34d51e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34d51e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34d51e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:54.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34dd1f640 1 --2- 172.21.15.114:0/2735564451 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe3340052f0 0x7fe33400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:07:54.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34dd1f640 10 monclient(hunting): get_auth_request con 0x7fe3340052f0 auth_method 0 2024-09-16T05:07:54.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34dd1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34dd1f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34dd1f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:07:54.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34d51e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34d51e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34d51e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:07:54.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34dd1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34dd1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34dd1f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:07:54.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient(hunting): handle_auth_done global_id 4392 payload 293 2024-09-16T05:07:54.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 1 -- 172.21.15.114:0/2735564451 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe3340052f0 msgr2=0x7fe33400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:54.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 1 --2- 172.21.15.114:0/2735564451 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe3340052f0 0x7fe33400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 1 -- 172.21.15.114:0/2735564451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3340039c0 msgr2=0x7fe334003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:54.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 1 --2- 172.21.15.114:0/2735564451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3340039c0 0x7fe334003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient: _finish_hunting 0 2024-09-16T05:07:54.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 1 monclient: found mon.c 2024-09-16T05:07:54.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:54.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:54.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 1 -- 172.21.15.114:0/2735564451 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe334003530 con 0x7fe334004810 2024-09-16T05:07:54.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34dd1f640 1 -- 172.21.15.114:0/2735564451 reap_dead start 2024-09-16T05:07:54.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient: _finish_auth 0 2024-09-16T05:07:54.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 10 monclient: _check_auth_tickets 2024-09-16T05:07:54.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:54.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34f782640 5 monclient: authenticate success, global_id 4392 2024-09-16T05:07:54.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34f782640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:54.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34d51e640 1 --2- 172.21.15.114:0/2735564451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3340039c0 0x7fe334003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:54.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe34cd1d640 1 --2- 172.21.15.114:0/2735564451 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe334004810 0x7fe334004c90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe338009a90 tx=0x7fe33801ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dba54bfb13ad1afe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 -- 172.21.15.114:0/2735564451 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe338009690 con 0x7fe334004810 2024-09-16T05:07:54.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:54.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 10 monclient: dump: 2024-09-16T05:07:54.395 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.395 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.395 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.396 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.396 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.396 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.396 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.396 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.396 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.396 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.382+0000 7fe33ffff640 1 -- 172.21.15.114:0/2735564451 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3380059a0 con 0x7fe334004810 2024-09-16T05:07:54.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:54.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:54.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:54.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 4 set_mon_vals no callback set 2024-09-16T05:07:54.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33ffff640 1 -- 172.21.15.114:0/2735564451 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe338005d90 con 0x7fe334004810 2024-09-16T05:07:54.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:54.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33ffff640 10 monclient: dump: 2024-09-16T05:07:54.397 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.397 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.397 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.397 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.397 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.397 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.397 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.397 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.397 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.398 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient: shutdown 2024-09-16T05:07:54.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:54.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 -- 172.21.15.114:0/2735564451 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe334004810 msgr2=0x7fe334004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 --2- 172.21.15.114:0/2735564451 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe334004810 0x7fe334004c90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe338009a90 tx=0x7fe33801ca10 comp rx=0 tx=0).stop 2024-09-16T05:07:54.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 -- 172.21.15.114:0/2735564451 shutdown_connections 2024-09-16T05:07:54.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 --2- 172.21.15.114:0/2735564451 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe334004810 0x7fe334004c90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 -- 172.21.15.114:0/2735564451 >> 172.21.15.114:0/2735564451 conn(0x7fe3480a1fb0 msgr2=0x7fe3480a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:54.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 -- 172.21.15.114:0/2735564451 shutdown_connections 2024-09-16T05:07:54.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 -- 172.21.15.114:0/2735564451 wait complete. 2024-09-16T05:07:54.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient: build_initial_monmap 2024-09-16T05:07:54.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient: monmap: 2024-09-16T05:07:54.403 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:54.403 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:54.403 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:54.404 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:54.404 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:54.404 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.404 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:54.404 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:54.404 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:54.404 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 librados: starting msgr at 2024-09-16T05:07:54.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 librados: starting objecter 2024-09-16T05:07:54.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 Processor -- start 2024-09-16T05:07:54.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 -- start start 2024-09-16T05:07:54.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 librados: setting wanted keys 2024-09-16T05:07:54.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 librados: calling monclient init 2024-09-16T05:07:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient: init 2024-09-16T05:07:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:07:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe334004810 0x7fe3481ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient(hunting): picked mon.noname-b con 0x7fe334004810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe348098610 0x7fe348098a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient(hunting): picked mon.noname-c con 0x7fe348098610 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe34808f330 0x7fe34808f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient(hunting): picked mon.noname-a con 0x7fe34808f330 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe348137640 con 0x7fe34808f330 2024-09-16T05:07:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fe348072ba0 con 0x7fe334004810 2024-09-16T05:07:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe3481377c0 con 0x7fe348098610 2024-09-16T05:07:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient(hunting): authenticate will time out at 959.018217s 2024-09-16T05:07:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe34808f330 0x7fe34808f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34cd1d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe348098610 0x7fe348098a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe34808f330 0x7fe34808f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55172/0 (socket says 172.21.15.114:55172) 2024-09-16T05:07:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34cd1d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe348098610 0x7fe348098a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55836/0 (socket says 172.21.15.114:55836) 2024-09-16T05:07:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 1 -- 172.21.15.114:0/1824292200 learned_addr learned my addr 172.21.15.114:0/1824292200 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 10 monclient(hunting): get_auth_request con 0x7fe34808f330 auth_method 0 2024-09-16T05:07:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34cd1d640 10 monclient(hunting): get_auth_request con 0x7fe348098610 auth_method 0 2024-09-16T05:07:54.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34cd1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34cd1d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34cd1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34cd1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34cd1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34cd1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 10 monclient(hunting): handle_auth_done global_id 4395 payload 824 2024-09-16T05:07:54.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34d51e640 1 --2- 172.21.15.114:0/1824292200 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe334004810 0x7fe3481ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 1 -- 172.21.15.114:0/1824292200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe348098610 msgr2=0x7fe348098a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 1 --2- 172.21.15.114:0/1824292200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe348098610 0x7fe348098a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 1 -- 172.21.15.114:0/1824292200 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe334004810 msgr2=0x7fe3481ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 1 --2- 172.21.15.114:0/1824292200 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe334004810 0x7fe3481ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 10 monclient: _finish_hunting 0 2024-09-16T05:07:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 1 monclient: found mon.noname-a 2024-09-16T05:07:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 1 -- 172.21.15.114:0/1824292200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3481cf430 con 0x7fe34808f330 2024-09-16T05:07:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34dd1f640 1 --2- 172.21.15.114:0/1824292200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe34808f330 0x7fe34808f7b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe340007fd0 tx=0x7fe34000fd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 10 client.?.objecter ms_handle_connect 0x7fe34808f330 2024-09-16T05:07:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 1 -- 172.21.15.114:0/1824292200 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe340003bb0 con 0x7fe34808f330 2024-09-16T05:07:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 10 monclient: dump: 2024-09-16T05:07:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.409 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.410 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 10 monclient: _finish_auth 0 2024-09-16T05:07:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:07:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 1 -- 172.21.15.114:0/1824292200 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe340018070 con 0x7fe34808f330 2024-09-16T05:07:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 5 monclient: authenticate success, global_id 4395 2024-09-16T05:07:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient: _renew_subs 2024-09-16T05:07:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 -- 172.21.15.114:0/1824292200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe34809c6a0 con 0x7fe34808f330 2024-09-16T05:07:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 1 -- 172.21.15.114:0/1824292200 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3400050d0 con 0x7fe34808f330 2024-09-16T05:07:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:54.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 10 monclient: dump: 2024-09-16T05:07:54.411 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.411 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.411 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.411 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.411 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.411 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.411 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.411 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.412 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33f7fe640 4 set_mon_vals no callback set 2024-09-16T05:07:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient: _renew_subs 2024-09-16T05:07:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 -- 172.21.15.114:0/1824292200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe348098f70 con 0x7fe34808f330 2024-09-16T05:07:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 librados: init done 2024-09-16T05:07:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe33dffb640 1 -- 172.21.15.114:0/1824292200 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe340005270 con 0x7fe34808f330 2024-09-16T05:07:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.386+0000 7fe34f782640 1 -- 172.21.15.114:0/1824292200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe348137b50 con 0x7fe34808f330 2024-09-16T05:07:54.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.390+0000 7fe33dffb640 1 --2- 172.21.15.114:0/1824292200 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe32c037840 0x7fe32c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.390+0000 7fe33dffb640 1 -- 172.21.15.114:0/1824292200 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7fe340048620 con 0x7fe34808f330 2024-09-16T05:07:54.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.390+0000 7fe33dffb640 10 client.4395.objecter ms_dispatch 0x7fe348000ba0 osd_map(19..19 src has 1..19) 2024-09-16T05:07:54.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.390+0000 7fe33dffb640 3 client.4395.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T05:07:54.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.390+0000 7fe33dffb640 3 client.4395.objecter handle_osd_map decoding full epoch 19 2024-09-16T05:07:54.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.390+0000 7fe33dffb640 20 client.4395.objecter dump_active .. 0 homeless 2024-09-16T05:07:54.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.390+0000 7fe34d51e640 1 --2- 172.21.15.114:0/1824292200 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe32c037840 0x7fe32c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.390+0000 7fe34d51e640 10 monclient: get_auth_request con 0x7fe32c037840 auth_method 0 2024-09-16T05:07:54.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.390+0000 7fe33dffb640 10 client.4395.objecter ms_handle_connect 0x7fe32c037840 2024-09-16T05:07:54.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.390+0000 7fe34d51e640 1 --2- 172.21.15.114:0/1824292200 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe32c037840 0x7fe32c039d00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe348069fb0 tx=0x7fe344009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.394+0000 7fe33dffb640 1 -- 172.21.15.114:0/1824292200 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fe34000d680 con 0x7fe34808f330 2024-09-16T05:07:54.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.394+0000 7fe33dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:54.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.394+0000 7fe33dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:54.421 INFO:tasks.workunit.client.0.smithi149.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T05:07:54.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.598+0000 7fe34f782640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:07:54.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.598+0000 7fe34f782640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:07:54.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.598+0000 7fe34f782640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:54.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.598+0000 7fe34f782640 1 -- 172.21.15.114:0/1824292200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe348090e30 con 0x7fe34808f330 2024-09-16T05:07:54.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.598+0000 7fe33dffb640 1 -- 172.21.15.114:0/1824292200 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fe340012350 con 0x7fe34808f330 2024-09-16T05:07:54.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.598+0000 7fe33dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:07:54.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.598+0000 7fe33dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:54.600 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:07:54.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 10 librados: watch_flush enter 2024-09-16T05:07:54.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 10 librados: watch_flush exit 2024-09-16T05:07:54.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 20 client.4395.objecter shutdown clearing up homeless session... 2024-09-16T05:07:54.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 10 client.4395.objecter successfully canceled tick 2024-09-16T05:07:54.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 1 -- 172.21.15.114:0/1824292200 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe32c037840 msgr2=0x7fe32c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 1 --2- 172.21.15.114:0/1824292200 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe32c037840 0x7fe32c039d00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe348069fb0 tx=0x7fe344009040 comp rx=0 tx=0).stop 2024-09-16T05:07:54.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 10 monclient: shutdown 2024-09-16T05:07:54.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:54.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 1 -- 172.21.15.114:0/1824292200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe34808f330 msgr2=0x7fe34808f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 1 --2- 172.21.15.114:0/1824292200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe34808f330 0x7fe34808f7b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe340007fd0 tx=0x7fe34000fd20 comp rx=0 tx=0).stop 2024-09-16T05:07:54.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 1 -- 172.21.15.114:0/1824292200 shutdown_connections 2024-09-16T05:07:54.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 1 --2- 172.21.15.114:0/1824292200 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe32c037840 0x7fe32c039d00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 1 --2- 172.21.15.114:0/1824292200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe34808f330 0x7fe34808f7b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 1 --2- 172.21.15.114:0/1824292200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe348098610 0x7fe348098a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 1 --2- 172.21.15.114:0/1824292200 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe334004810 0x7fe3481ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 1 -- 172.21.15.114:0/1824292200 >> 172.21.15.114:0/1824292200 conn(0x7fe3480a1fb0 msgr2=0x7fe34809a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:54.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 1 -- 172.21.15.114:0/1824292200 shutdown_connections 2024-09-16T05:07:54.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 1 -- 172.21.15.114:0/1824292200 wait complete. 2024-09-16T05:07:54.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.602+0000 7fe34f782640 1 librados: shutdown 2024-09-16T05:07:54.618 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:07:54.618 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:07:54.618 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:07:54.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:07:54.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 10 monclient: get_monmap_and_config 2024-09-16T05:07:54.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 10 monclient: build_initial_monmap 2024-09-16T05:07:54.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 10 monclient: monmap: 2024-09-16T05:07:54.722 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:54.722 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.722 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:54.725549+0000 2024-09-16T05:07:54.722 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:54.725549+0000 2024-09-16T05:07:54.722 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:54.722 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.722 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:54.722 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:54.722 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:54.722 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:54.723 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:54.723 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 1 Processor -- start 2024-09-16T05:07:54.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 1 -- start start 2024-09-16T05:07:54.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 10 monclient: init 2024-09-16T05:07:54.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:54.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T05:07:54.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f521809f6f0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:54.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f521809fad0 0x7f521809feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 10 monclient(hunting): picked mon.noname-b con 0x7f521809fad0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:54.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f5218138850 0x7f521813ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 10 monclient(hunting): picked mon.noname-c con 0x7f5218138850 addr v2:172.21.15.149:3300/0 2024-09-16T05:07:54.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52180a1df0 con 0x7f521809f6f0 2024-09-16T05:07:54.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f52181344a0 con 0x7f5218138850 2024-09-16T05:07:54.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f52180a1c20 con 0x7f521809fad0 2024-09-16T05:07:54.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:54.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f521e016640 10 monclient(hunting): authenticate will time out at 959.355210s 2024-09-16T05:07:54.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f521809fad0 0x7f521809feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f521809fad0 0x7f521809feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55848/0 (socket says 172.21.15.114:55848) 2024-09-16T05:07:54.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 1 -- 172.21.15.114:0/2147345498 learned_addr learned my addr 172.21.15.114:0/2147345498 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:54.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 10 monclient(hunting): get_auth_request con 0x7f521809fad0 auth_method 0 2024-09-16T05:07:54.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-09-16T05:07:54.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 1 -- 172.21.15.114:0/2147345498 >> v2:172.21.15.149:3300/0 conn(0x7f5218138850 msgr2=0x7f521813ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:54.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 1 --2- 172.21.15.114:0/2147345498 >> v2:172.21.15.149:3300/0 conn(0x7f5218138850 0x7f521813ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 1 -- 172.21.15.114:0/2147345498 >> v1:172.21.15.149:6789/0 conn(0x7f521809f6f0 legacy=0x7f5218133530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:54.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:07:54.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 1 monclient: found mon.noname-b 2024-09-16T05:07:54.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:54.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:54.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.722+0000 7f5216ffd640 1 -- 172.21.15.114:0/2147345498 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f521809d920 con 0x7f521809fad0 2024-09-16T05:07:54.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f5216ffd640 1 --2- 172.21.15.114:0/2147345498 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f521809fad0 0x7f521809feb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f520c009870 tx=0x7f520c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=319efdfe33ef1b9e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f52167fc640 1 -- 172.21.15.114:0/2147345498 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520c0b0070 con 0x7f521809fad0 2024-09-16T05:07:54.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f52167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f52167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:54.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f52167fc640 10 monclient: dump: 2024-09-16T05:07:54.736 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.737 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.737 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.737 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.738 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.739 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.739 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.739 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.739 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.739 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f52167fc640 10 monclient: _finish_auth 0 2024-09-16T05:07:54.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f52167fc640 10 monclient: _check_auth_tickets 2024-09-16T05:07:54.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f52167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:54.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f52167fc640 1 -- 172.21.15.114:0/2147345498 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f520c0ab600 con 0x7f521809fad0 2024-09-16T05:07:54.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 5 monclient: authenticate success, global_id 4322 2024-09-16T05:07:54.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:54.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f52167fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:54.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f52167fc640 1 -- 172.21.15.114:0/2147345498 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520c0ab9d0 con 0x7f521809fad0 2024-09-16T05:07:54.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:54.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 4 set_mon_vals no callback set 2024-09-16T05:07:54.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f52167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f52167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:54.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f52167fc640 10 monclient: dump: 2024-09-16T05:07:54.740 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.740 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.740 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.740 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.741 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.741 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.741 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.741 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.741 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.741 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 10 monclient: shutdown 2024-09-16T05:07:54.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:54.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 1 -- 172.21.15.114:0/2147345498 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f521809fad0 msgr2=0x7f521809feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 1 --2- 172.21.15.114:0/2147345498 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f521809fad0 0x7f521809feb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f520c009870 tx=0x7f520c01c3e0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 1 -- 172.21.15.114:0/2147345498 shutdown_connections 2024-09-16T05:07:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 1 --2- 172.21.15.114:0/2147345498 >> v2:172.21.15.149:3300/0 conn(0x7f5218138850 0x7f521813ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 1 --2- 172.21.15.114:0/2147345498 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f521809fad0 0x7f521809feb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 1 -- 172.21.15.114:0/2147345498 >> 172.21.15.114:0/2147345498 conn(0x7f52180a6380 msgr2=0x7f52180a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 1 -- 172.21.15.114:0/2147345498 shutdown_connections 2024-09-16T05:07:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 1 -- 172.21.15.114:0/2147345498 wait complete. 2024-09-16T05:07:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 10 monclient: build_initial_monmap 2024-09-16T05:07:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.726+0000 7f521e016640 10 monclient: monmap: 2024-09-16T05:07:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:54.743 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:54.743 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:54.743 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 librados: starting msgr at 2024-09-16T05:07:54.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 librados: starting objecter 2024-09-16T05:07:54.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 Processor -- start 2024-09-16T05:07:54.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 -- start start 2024-09-16T05:07:54.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 librados: setting wanted keys 2024-09-16T05:07:54.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 librados: calling monclient init 2024-09-16T05:07:54.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient: init 2024-09-16T05:07:54.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:54.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:07:54.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f521809f6f0 0x7f52181cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient(hunting): picked mon.noname-b con 0x7f521809f6f0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:54.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f521809fad0 0x7f5218133060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient(hunting): picked mon.noname-a con 0x7f521809fad0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:54.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5218138850 0x7f5218070a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient(hunting): picked mon.noname-c con 0x7f5218138850 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:54.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5218140050 con 0x7f521809fad0 2024-09-16T05:07:54.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f521813fed0 con 0x7f521809f6f0 2024-09-16T05:07:54.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f52181401d0 con 0x7f5218138850 2024-09-16T05:07:54.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:54.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient(hunting): authenticate will time out at 959.360669s 2024-09-16T05:07:54.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5217fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5218138850 0x7f5218070a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f521809fad0 0x7f5218133060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f521809fad0 0x7f5218133060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55180/0 (socket says 172.21.15.114:55180) 2024-09-16T05:07:54.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 1 -- 172.21.15.114:0/3593565361 learned_addr learned my addr 172.21.15.114:0/3593565361 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:54.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5217fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5218138850 0x7f5218070a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55852/0 (socket says 172.21.15.114:55852) 2024-09-16T05:07:54.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 10 monclient(hunting): get_auth_request con 0x7f521809fad0 auth_method 0 2024-09-16T05:07:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5217fff640 10 monclient(hunting): get_auth_request con 0x7f5218138850 auth_method 0 2024-09-16T05:07:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5217fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5217fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5217fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5217fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5217fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5217fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 1 --2- 172.21.15.114:0/3593565361 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f521809f6f0 0x7f52181cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 10 monclient(hunting): get_auth_request con 0x7f521809f6f0 auth_method 0 2024-09-16T05:07:54.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 10 monclient(hunting): handle_auth_done global_id 4398 payload 824 2024-09-16T05:07:54.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 1 -- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5218138850 msgr2=0x7f5218070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 1 --2- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5218138850 0x7f5218070a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 1 -- 172.21.15.114:0/3593565361 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f521809f6f0 msgr2=0x7f52181cd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 1 --2- 172.21.15.114:0/3593565361 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f521809f6f0 0x7f52181cd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:07:54.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 1 monclient: found mon.noname-a 2024-09-16T05:07:54.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:54.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:07:54.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 1 -- 172.21.15.114:0/3593565361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52180a4640 con 0x7f521809fad0 2024-09-16T05:07:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5216ffd640 1 --2- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f521809fad0 0x7f5218133060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f520c0ae040 tx=0x7f520c01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 client.?.objecter ms_handle_connect 0x7f521809fad0 2024-09-16T05:07:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 1 -- 172.21.15.114:0/3593565361 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520c0ba020 con 0x7f521809fad0 2024-09-16T05:07:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:07:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 monclient: dump: 2024-09-16T05:07:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.749 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 monclient: _finish_auth 0 2024-09-16T05:07:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 monclient: _check_auth_tickets 2024-09-16T05:07:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 1 -- 172.21.15.114:0/3593565361 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f520c0aa740 con 0x7f521809fad0 2024-09-16T05:07:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 1 -- 172.21.15.114:0/3593565361 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520c0b0040 con 0x7f521809fad0 2024-09-16T05:07:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 monclient: dump: 2024-09-16T05:07:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.750 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.751 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5215ffb640 4 set_mon_vals no callback set 2024-09-16T05:07:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 5 monclient: authenticate success, global_id 4398 2024-09-16T05:07:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient: _renew_subs 2024-09-16T05:07:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 -- 172.21.15.114:0/3593565361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f521811d710 con 0x7f521809fad0 2024-09-16T05:07:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient: _renew_subs 2024-09-16T05:07:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:07:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 -- 172.21.15.114:0/3593565361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5218072b60 con 0x7f521809fad0 2024-09-16T05:07:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 librados: init done 2024-09-16T05:07:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:54.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 --2- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52181335a0 0x7f5218070f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f521e016640 1 -- 172.21.15.114:0/3593565361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5218072720 con 0x7f52181335a0 2024-09-16T05:07:54.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 1 --2- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52181335a0 0x7f5218070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 10 monclient: get_auth_request con 0x7f52181335a0 auth_method 0 2024-09-16T05:07:54.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 1 -- 172.21.15.114:0/3593565361 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f520c0bd020 con 0x7f521809fad0 2024-09-16T05:07:54.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 10 monclient(hunting): handle_auth_done global_id 4401 payload 824 2024-09-16T05:07:54.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 1 -- 172.21.15.114:0/3593565361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f521809feb0 con 0x7f52181335a0 2024-09-16T05:07:54.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 1 --2- 172.21.15.114:0/3593565361 >> v2:172.21.15.149:6812/1185531135 conn(0x7f5204037810 0x7f5204039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 1 -- 172.21.15.114:0/3593565361 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f520c0e2bc0 con 0x7f521809fad0 2024-09-16T05:07:54.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 client.4398.objecter ms_dispatch 0x7f5218000ba0 osd_map(19..19 src has 1..19) 2024-09-16T05:07:54.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 3 client.4398.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T05:07:54.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 3 client.4398.objecter handle_osd_map decoding full epoch 19 2024-09-16T05:07:54.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f52177fe640 1 --2- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52181335a0 0x7f5218070f60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f520800e7c0 tx=0x7f520800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 20 client.4398.objecter dump_active .. 0 homeless 2024-09-16T05:07:54.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 client.4398.objecter ms_handle_connect 0x7f52181335a0 2024-09-16T05:07:54.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 client.4398.objecter resend_mon_ops 2024-09-16T05:07:54.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 1 -- 172.21.15.114:0/3593565361 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5208003b90 con 0x7f52181335a0 2024-09-16T05:07:54.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 monclient: dump: 2024-09-16T05:07:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.754 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5217fff640 1 --2- 172.21.15.114:0/3593565361 >> v2:172.21.15.149:6812/1185531135 conn(0x7f5204037810 0x7f5204039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5217fff640 10 monclient: get_auth_request con 0x7f5204037810 auth_method 0 2024-09-16T05:07:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 1 -- 172.21.15.114:0/3593565361 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5208003b90 con 0x7f52181335a0 2024-09-16T05:07:54.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:54.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:54.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 1 -- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52181335a0 msgr2=0x7f5218070f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 1 --2- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52181335a0 0x7f5218070f60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f520800e7c0 tx=0x7f520800ec80 comp rx=0 tx=0).stop 2024-09-16T05:07:54.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 1 -- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52181335a0 msgr2=0x7f5218070f60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:07:54.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 1 --2- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52181335a0 0x7f5218070f60 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f520800e7c0 tx=0x7f520800ec80 comp rx=0 tx=0).stop 2024-09-16T05:07:54.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f51fffff640 10 client.4398.objecter ms_handle_connect 0x7f5204037810 2024-09-16T05:07:54.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.730+0000 7f5217fff640 1 --2- 172.21.15.114:0/3593565361 >> v2:172.21.15.149:6812/1185531135 conn(0x7f5204037810 0x7f5204039cd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5210006fd0 tx=0x7f52100090e0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:07:54.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 1 --2- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52180714a0 0x7f5218136630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 1 -- 172.21.15.114:0/3593565361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f52180718b0 con 0x7f52180714a0 2024-09-16T05:07:54.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f52177fe640 1 --2- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52180714a0 0x7f5218136630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f52177fe640 10 monclient: get_auth_request con 0x7f52180714a0 auth_method 0 2024-09-16T05:07:54.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f52177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f52177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f52177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f52177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f52177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f52177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f52177fe640 10 monclient(hunting): handle_auth_done global_id 4404 payload 824 2024-09-16T05:07:54.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f52177fe640 1 -- 172.21.15.114:0/3593565361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f521813a160 con 0x7f52180714a0 2024-09-16T05:07:54.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f52177fe640 1 --2- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52180714a0 0x7f5218136630 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f520800e7c0 tx=0x7f52080013b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f51fffff640 10 client.4398.objecter ms_handle_connect 0x7f52180714a0 2024-09-16T05:07:54.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f51fffff640 10 client.4398.objecter resend_mon_ops 2024-09-16T05:07:54.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f51fffff640 1 -- 172.21.15.114:0/3593565361 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52080050d0 con 0x7f52180714a0 2024-09-16T05:07:54.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f51fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f51fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:07:54.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f51fffff640 10 monclient: dump: 2024-09-16T05:07:54.759 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.759 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.759 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.759 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.759 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.759 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.759 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.759 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.759 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.759 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f51fffff640 1 -- 172.21.15.114:0/3593565361 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7f52080050d0 con 0x7f52180714a0 2024-09-16T05:07:54.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f51fffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:07:54.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f51fffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:54.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f51fffff640 1 -- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52180714a0 msgr2=0x7f5218136630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f51fffff640 1 --2- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52180714a0 0x7f5218136630 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f520800e7c0 tx=0x7f52080013b0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f51fffff640 1 -- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52180714a0 msgr2=0x7f5218136630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:07:54.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f51fffff640 1 --2- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52180714a0 0x7f5218136630 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.760 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:07:54.760 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:07:54.760 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:07:54.760 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:07:54.760 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 8, 2024-09-16T05:07:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 32727, 2024-09-16T05:07:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:07:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:07:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:07:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:07:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 27, 2024-09-16T05:07:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:07:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:07:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:07:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:07:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:07:54.761 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:07:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:07:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:07:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:07:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:07:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:07:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:07:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:07:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:07:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:07:54.765 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:07:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:07:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:07:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:07:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:07:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:07:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:07:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:07:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:07:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:07:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:07:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:07:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.766 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:07:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:07:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:07:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:07:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:07:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:07:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:07:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:07:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:07:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:07:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:07:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:07:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:07:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:07:54.767 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:07:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:07:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:07:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:07:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:07:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:07:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:07:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:07:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:07:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:07:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:07:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:07:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:07:54.768 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:07:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:07:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:07:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:07:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:07:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:07:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:07:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:07:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:07:54.769 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:07:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:07:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:07:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:07:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:07:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:07:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:07:54.770 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:07:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:07:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:07:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:07:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:07:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:07:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:07:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:07:54.771 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:07:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:07:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:07:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:07:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:07:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:07:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:07:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:07:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:07:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:07:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:07:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:07:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:07:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.773 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:07:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:07:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:07:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:07:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:07:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:07:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:07:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:07:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:07:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:07:54.774 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:07:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:07:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:07:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:07:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:07:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:07:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:07:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:07:54.775 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:54.776 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:54.776 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.776 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:07:54.776 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:07:54.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 10 librados: watch_flush enter 2024-09-16T05:07:54.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 10 librados: watch_flush exit 2024-09-16T05:07:54.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 20 client.4398.objecter shutdown clearing up homeless session... 2024-09-16T05:07:54.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 10 client.4398.objecter successfully canceled tick 2024-09-16T05:07:54.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 1 -- 172.21.15.114:0/3593565361 >> v2:172.21.15.149:6812/1185531135 conn(0x7f5204037810 msgr2=0x7f5204039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 1 --2- 172.21.15.114:0/3593565361 >> v2:172.21.15.149:6812/1185531135 conn(0x7f5204037810 0x7f5204039cd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5210006fd0 tx=0x7f52100090e0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 10 monclient: shutdown 2024-09-16T05:07:54.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:54.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 1 -- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f521809fad0 msgr2=0x7f5218133060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 1 --2- 172.21.15.114:0/3593565361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f521809fad0 0x7f5218133060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f520c0ae040 tx=0x7f520c01ac20 comp rx=0 tx=0).stop 2024-09-16T05:07:54.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f5217fff640 1 -- 172.21.15.114:0/3593565361 reap_dead start 2024-09-16T05:07:54.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f5217fff640 1 -- 172.21.15.114:0/3593565361 reap_dead start 2024-09-16T05:07:54.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 1 -- 172.21.15.114:0/3593565361 shutdown_connections 2024-09-16T05:07:54.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 1 -- 172.21.15.114:0/3593565361 >> 172.21.15.114:0/3593565361 conn(0x7f52180a6380 msgr2=0x7f5218138c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:54.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 1 -- 172.21.15.114:0/3593565361 shutdown_connections 2024-09-16T05:07:54.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 1 -- 172.21.15.114:0/3593565361 wait complete. 2024-09-16T05:07:54.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.738+0000 7f521e016640 1 librados: shutdown 2024-09-16T05:07:54.779 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:07:54.779 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:07:54.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:07:54.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 10 monclient: get_monmap_and_config 2024-09-16T05:07:54.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 10 monclient: build_initial_monmap 2024-09-16T05:07:54.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 10 monclient: monmap: 2024-09-16T05:07:54.885 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:54.885 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.885 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:54.888475+0000 2024-09-16T05:07:54.885 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:54.888475+0000 2024-09-16T05:07:54.885 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:54.885 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.885 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:54.885 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:54.886 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:54.886 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:54.886 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:54.886 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 1 Processor -- start 2024-09-16T05:07:54.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 1 -- start start 2024-09-16T05:07:54.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 10 monclient: init 2024-09-16T05:07:54.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:54.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-16T05:07:54.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff4e8137480 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:54.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e80a57b0 0x7ff4e813c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 10 monclient(hunting): picked mon.noname-b con 0x7ff4e80a57b0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:54.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff4e813c890 0x7ff4e813ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 10 monclient(hunting): picked mon.noname-a con 0x7ff4e813c890 addr v2:172.21.15.114:3300/0 2024-09-16T05:07:54.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4e809cfa0 con 0x7ff4e8137480 2024-09-16T05:07:54.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff4e8137f70 con 0x7ff4e813c890 2024-09-16T05:07:54.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff4e80a52f0 con 0x7ff4e80a57b0 2024-09-16T05:07:54.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:54.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4ed689640 10 monclient(hunting): authenticate will time out at 959.517662s 2024-09-16T05:07:54.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e77fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff4e813c890 0x7ff4e813ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e80a57b0 0x7ff4e813c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e6ffd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff4e8137480 0x7ff4e8137860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38730/0 (socket says 172.21.15.114:38730) 2024-09-16T05:07:54.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e6ffd640 1 -- 172.21.15.114:0/2723569575 learned_addr learned my addr 172.21.15.114:0/2723569575 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:54.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 1 --2- 172.21.15.114:0/2723569575 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e80a57b0 0x7ff4e813c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55862/0 (socket says 172.21.15.114:55862) 2024-09-16T05:07:54.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e77fe640 10 monclient(hunting): get_auth_request con 0x7ff4e813c890 auth_method 0 2024-09-16T05:07:54.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 10 monclient(hunting): get_auth_request con 0x7ff4e80a57b0 auth_method 0 2024-09-16T05:07:54.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 10 monclient(hunting): handle_auth_done global_id 4328 payload 274 2024-09-16T05:07:54.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 1 -- 172.21.15.114:0/2723569575 >> v2:172.21.15.114:3300/0 conn(0x7ff4e813c890 msgr2=0x7ff4e813ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:54.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 1 --2- 172.21.15.114:0/2723569575 >> v2:172.21.15.114:3300/0 conn(0x7ff4e813c890 0x7ff4e813ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 1 -- 172.21.15.114:0/2723569575 >> v1:172.21.15.114:6789/0 conn(0x7ff4e8137480 legacy=0x7ff4e8137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 10 monclient: _finish_hunting 0 2024-09-16T05:07:54.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 1 monclient: found mon.noname-b 2024-09-16T05:07:54.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:54.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:54.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e67fc640 1 -- 172.21.15.114:0/2723569575 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4e80a5570 con 0x7ff4e80a57b0 2024-09-16T05:07:54.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.886+0000 7ff4e77fe640 1 --2- 172.21.15.114:0/2723569575 >> v2:172.21.15.114:3300/0 conn(0x7ff4e813c890 0x7ff4e813ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:54.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 1 --2- 172.21.15.114:0/2723569575 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e80a57b0 0x7ff4e813c350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00097c0 tx=0x7ff4d001c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29a4ece26ee50efd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e5ffb640 1 -- 172.21.15.114:0/2723569575 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d00b0070 con 0x7ff4e80a57b0 2024-09-16T05:07:54.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:54.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e5ffb640 10 monclient: dump: 2024-09-16T05:07:54.899 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.899 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.899 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.899 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.899 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.900 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.900 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.900 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.900 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.900 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:07:54.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:07:54.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:54.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e5ffb640 1 -- 172.21.15.114:0/2723569575 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4d00ab600 con 0x7ff4e80a57b0 2024-09-16T05:07:54.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:54.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e5ffb640 1 -- 172.21.15.114:0/2723569575 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d00ab9d0 con 0x7ff4e80a57b0 2024-09-16T05:07:54.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:54.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e5ffb640 10 monclient: dump: 2024-09-16T05:07:54.901 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.901 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.901 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.901 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.901 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.901 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.901 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.901 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.901 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.901 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 5 monclient: authenticate success, global_id 4328 2024-09-16T05:07:54.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:54.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 4 set_mon_vals no callback set 2024-09-16T05:07:54.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient: shutdown 2024-09-16T05:07:54.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:54.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 -- 172.21.15.114:0/2723569575 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e80a57b0 msgr2=0x7ff4e813c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 --2- 172.21.15.114:0/2723569575 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e80a57b0 0x7ff4e813c350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00097c0 tx=0x7ff4d001c330 comp rx=0 tx=0).stop 2024-09-16T05:07:54.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 -- 172.21.15.114:0/2723569575 shutdown_connections 2024-09-16T05:07:54.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 --2- 172.21.15.114:0/2723569575 >> v2:172.21.15.114:3300/0 conn(0x7ff4e813c890 0x7ff4e813ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 --2- 172.21.15.114:0/2723569575 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e80a57b0 0x7ff4e813c350 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 -- 172.21.15.114:0/2723569575 >> 172.21.15.114:0/2723569575 conn(0x7ff4e80a9c40 msgr2=0x7ff4e80aa050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:54.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 -- 172.21.15.114:0/2723569575 shutdown_connections 2024-09-16T05:07:54.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 -- 172.21.15.114:0/2723569575 wait complete. 2024-09-16T05:07:54.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient: build_initial_monmap 2024-09-16T05:07:54.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient: monmap: 2024-09-16T05:07:54.902 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:54.903 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:54.903 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:54.903 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:54.903 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:54.903 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.903 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:54.903 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:54.903 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:54.903 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 librados: starting msgr at 2024-09-16T05:07:54.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 librados: starting objecter 2024-09-16T05:07:54.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 Processor -- start 2024-09-16T05:07:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 -- start start 2024-09-16T05:07:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 librados: setting wanted keys 2024-09-16T05:07:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 librados: calling monclient init 2024-09-16T05:07:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient: init 2024-09-16T05:07:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:07:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4e80a57b0 0x7ff4e81cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient(hunting): picked mon.noname-b con 0x7ff4e80a57b0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e8137480 0x7ff4e8136270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient(hunting): picked mon.noname-c con 0x7ff4e8137480 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e813c890 0x7ff4e80a00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient(hunting): picked mon.noname-a con 0x7ff4e813c890 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e80a5e80 con 0x7ff4e813c890 2024-09-16T05:07:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e8072b20 con 0x7ff4e80a57b0 2024-09-16T05:07:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff4e81419e0 con 0x7ff4e8137480 2024-09-16T05:07:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient(hunting): authenticate will time out at 959.522503s 2024-09-16T05:07:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e8137480 0x7ff4e8136270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e8137480 0x7ff4e8136270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55866/0 (socket says 172.21.15.114:55866) 2024-09-16T05:07:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e813c890 0x7ff4e80a00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 1 -- 172.21.15.114:0/2478203131 learned_addr learned my addr 172.21.15.114:0/2478203131 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 10 monclient(hunting): get_auth_request con 0x7ff4e8137480 auth_method 0 2024-09-16T05:07:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e77fe640 10 monclient(hunting): get_auth_request con 0x7ff4e813c890 auth_method 0 2024-09-16T05:07:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e6ffd640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4e80a57b0 0x7ff4e81cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 10 monclient(hunting): handle_auth_done global_id 4331 payload 824 2024-09-16T05:07:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 1 -- 172.21.15.114:0/2478203131 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4e80a57b0 msgr2=0x7ff4e81cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4e80a57b0 0x7ff4e81cecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 1 -- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e813c890 msgr2=0x7ff4e80a00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e813c890 0x7ff4e80a00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 10 monclient: _finish_hunting 0 2024-09-16T05:07:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 1 monclient: found mon.noname-c 2024-09-16T05:07:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 1 -- 172.21.15.114:0/2478203131 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4e8120920 con 0x7ff4e8137480 2024-09-16T05:07:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e6ffd640 10 monclient: get_auth_request con 0x7ff4e80a57b0 auth_method 0 2024-09-16T05:07:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e77fe640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e813c890 0x7ff4e80a00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:07:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e6ffd640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4e80a57b0 0x7ff4e81cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:07:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e67fc640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e8137480 0x7ff4e8136270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0019f60 tx=0x7ff4d00190d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4df7fe640 10 client.?.objecter ms_handle_connect 0x7ff4e8137480 2024-09-16T05:07:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4df7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4df7fe640 1 -- 172.21.15.114:0/2478203131 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d00b0040 con 0x7ff4e8137480 2024-09-16T05:07:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4df7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4df7fe640 10 monclient: dump: 2024-09-16T05:07:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.909 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4df7fe640 10 monclient: _finish_auth 0 2024-09-16T05:07:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4df7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:07:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4df7fe640 1 -- 172.21.15.114:0/2478203131 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4d00b43f0 con 0x7ff4e8137480 2024-09-16T05:07:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 5 monclient: authenticate success, global_id 4331 2024-09-16T05:07:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient: _renew_subs 2024-09-16T05:07:54.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:54.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 -- 172.21.15.114:0/2478203131 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4e8120c00 con 0x7ff4e8137480 2024-09-16T05:07:54.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4df7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:54.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4df7fe640 1 -- 172.21.15.114:0/2478203131 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0006e20 con 0x7ff4e8137480 2024-09-16T05:07:54.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:54.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4df7fe640 10 monclient: dump: 2024-09-16T05:07:54.910 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.910 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.910 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.910 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.910 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.910 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.910 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.911 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.911 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.911 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4e57fa640 4 set_mon_vals no callback set 2024-09-16T05:07:54.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:54.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient: _renew_subs 2024-09-16T05:07:54.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:54.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 -- 172.21.15.114:0/2478203131 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4e8120640 con 0x7ff4e8137480 2024-09-16T05:07:54.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 librados: init done 2024-09-16T05:07:54.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:54.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e80a05f0 0x7ff4e80a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.890+0000 7ff4ed689640 1 -- 172.21.15.114:0/2478203131 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff4e80a8ce0 con 0x7ff4e80a05f0 2024-09-16T05:07:54.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4e6ffd640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e80a05f0 0x7ff4e80a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4e6ffd640 10 monclient: get_auth_request con 0x7ff4e80a05f0 auth_method 0 2024-09-16T05:07:54.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 1 -- 172.21.15.114:0/2478203131 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff4d00b47c0 con 0x7ff4e8137480 2024-09-16T05:07:54.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4e6ffd640 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2024-09-16T05:07:54.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4e6ffd640 1 -- 172.21.15.114:0/2478203131 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4e80a0f10 con 0x7ff4e80a05f0 2024-09-16T05:07:54.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 1 --2- 172.21.15.114:0/2478203131 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff4c8037810 0x7ff4c8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 1 -- 172.21.15.114:0/2478203131 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7ff4d00e2140 con 0x7ff4e8137480 2024-09-16T05:07:54.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 10 client.4331.objecter ms_dispatch 0x7ff4e8000ba0 osd_map(19..19 src has 1..19) 2024-09-16T05:07:54.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 3 client.4331.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T05:07:54.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4e6ffd640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e80a05f0 0x7ff4e80a09d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000db60 tx=0x7ff4e000b4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 3 client.4331.objecter handle_osd_map decoding full epoch 19 2024-09-16T05:07:54.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 20 client.4331.objecter dump_active .. 0 homeless 2024-09-16T05:07:54.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 10 client.4331.objecter ms_handle_connect 0x7ff4e80a05f0 2024-09-16T05:07:54.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 10 client.4331.objecter resend_mon_ops 2024-09-16T05:07:54.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 1 -- 172.21.15.114:0/2478203131 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e000bbd0 con 0x7ff4e80a05f0 2024-09-16T05:07:54.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:54.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 10 monclient: dump: 2024-09-16T05:07:54.913 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.913 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.913 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.913 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.914 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4e77fe640 1 --2- 172.21.15.114:0/2478203131 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff4c8037810 0x7ff4c8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4e77fe640 10 monclient: get_auth_request con 0x7ff4c8037810 auth_method 0 2024-09-16T05:07:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 1 -- 172.21.15.114:0/2478203131 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff4e000bbd0 con 0x7ff4e80a05f0 2024-09-16T05:07:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 1 -- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e80a05f0 msgr2=0x7ff4e80a09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e80a05f0 0x7ff4e80a09d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000db60 tx=0x7ff4e000b4a0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 1 -- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e80a05f0 msgr2=0x7ff4e80a09d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:07:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e80a05f0 0x7ff4e80a09d0 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000db60 tx=0x7ff4e000b4a0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4df7fe640 10 client.4331.objecter ms_handle_connect 0x7ff4c8037810 2024-09-16T05:07:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.894+0000 7ff4e77fe640 1 --2- 172.21.15.114:0/2478203131 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff4c8037810 0x7ff4c8039cd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8073560 tx=0x7ff4d8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4ed689640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:07:54.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4ed689640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e813a130 0x7ff4e813a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:54.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4ed689640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:54.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4ed689640 1 -- 172.21.15.114:0/2478203131 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff4e8069390 con 0x7ff4e813a130 2024-09-16T05:07:54.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4e6ffd640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e813a130 0x7ff4e813a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:54.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4e6ffd640 10 monclient: get_auth_request con 0x7ff4e813a130 auth_method 0 2024-09-16T05:07:54.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:54.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:54.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:54.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:54.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:54.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:54.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4e6ffd640 10 monclient(hunting): handle_auth_done global_id 4337 payload 824 2024-09-16T05:07:54.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4e6ffd640 1 -- 172.21.15.114:0/2478203131 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff4e8137860 con 0x7ff4e813a130 2024-09-16T05:07:54.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4df7fe640 10 client.4331.objecter ms_handle_connect 0x7ff4e813a130 2024-09-16T05:07:54.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4e6ffd640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e813a130 0x7ff4e813a540 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000db60 tx=0x7ff4e0001260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:54.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4df7fe640 10 client.4331.objecter resend_mon_ops 2024-09-16T05:07:54.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4df7fe640 1 -- 172.21.15.114:0/2478203131 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e000be50 con 0x7ff4e813a130 2024-09-16T05:07:54.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:54.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:54.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4df7fe640 10 monclient: dump: 2024-09-16T05:07:54.918 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:54.918 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:54.918 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.918 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:54.918 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:54.919 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:54.919 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:54.919 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:54.919 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:54.919 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:54.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4df7fe640 1 -- 172.21.15.114:0/2478203131 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7ff4e000be50 con 0x7ff4e813a130 2024-09-16T05:07:54.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4df7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:07:54.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4df7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:54.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4df7fe640 1 -- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e813a130 msgr2=0x7ff4e813a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4df7fe640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e813a130 0x7ff4e813a540 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000db60 tx=0x7ff4e0001260 comp rx=0 tx=0).stop 2024-09-16T05:07:54.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4df7fe640 1 -- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e813a130 msgr2=0x7ff4e813a540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:07:54.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.898+0000 7ff4df7fe640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e813a130 0x7ff4e813a540 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000db60 tx=0x7ff4e0001260 comp rx=0 tx=0).stop 2024-09-16T05:07:54.920 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:07:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:07:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:07:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:07:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 8, 2024-09-16T05:07:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 32887, 2024-09-16T05:07:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:07:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:07:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:07:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:07:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:54.920 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 27, 2024-09-16T05:07:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:07:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:07:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:07:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:07:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:07:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:07:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:07:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:07:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:07:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:07:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:07:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:07:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:07:54.921 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:07:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:07:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:07:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:07:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:07:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:07:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:07:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:07:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:07:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:07:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:07:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:07:54.922 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:07:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:07:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:07:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:07:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:07:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:07:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:07:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:07:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:07:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:07:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:07:54.923 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:07:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:07:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:07:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:07:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:07:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:07:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:07:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:07:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:07:54.924 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:07:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:07:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:07:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:07:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:07:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:07:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:07:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:07:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:07:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:07:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:07:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:07:54.925 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:07:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:07:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:07:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:07:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:07:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:07:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:07:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:07:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:54.926 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:07:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:07:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:07:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:07:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:07:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:07:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:07:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:07:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:07:54.927 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:07:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:07:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:07:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:07:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:54.928 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:07:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:07:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:07:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:07:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:07:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:07:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:07:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:07:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:07:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.929 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:07:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:07:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:07:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:07:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:07:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:07:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:07:54.930 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:07:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:07:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:07:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:07:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:07:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:07:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:07:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:07:54.931 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:07:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:07:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:07:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:07:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:07:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:07:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:07:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:54.932 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:07:54.933 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:07:54.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4ed689640 10 librados: watch_flush enter 2024-09-16T05:07:54.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4ed689640 10 librados: watch_flush exit 2024-09-16T05:07:54.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4ed689640 20 client.4331.objecter shutdown clearing up homeless session... 2024-09-16T05:07:54.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4ed689640 10 client.4331.objecter successfully canceled tick 2024-09-16T05:07:54.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4ed689640 1 -- 172.21.15.114:0/2478203131 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff4c8037810 msgr2=0x7ff4c8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4ed689640 1 --2- 172.21.15.114:0/2478203131 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff4c8037810 0x7ff4c8039cd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8073560 tx=0x7ff4d8009190 comp rx=0 tx=0).stop 2024-09-16T05:07:54.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4ed689640 10 monclient: shutdown 2024-09-16T05:07:54.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4ed689640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:54.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4ed689640 1 -- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e8137480 msgr2=0x7ff4e8136270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:54.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4ed689640 1 --2- 172.21.15.114:0/2478203131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4e8137480 0x7ff4e8136270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0019f60 tx=0x7ff4d00190d0 comp rx=0 tx=0).stop 2024-09-16T05:07:54.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4e77fe640 1 -- 172.21.15.114:0/2478203131 reap_dead start 2024-09-16T05:07:54.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4e77fe640 1 -- 172.21.15.114:0/2478203131 reap_dead start 2024-09-16T05:07:54.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4ed689640 1 -- 172.21.15.114:0/2478203131 shutdown_connections 2024-09-16T05:07:54.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4ed689640 1 -- 172.21.15.114:0/2478203131 >> 172.21.15.114:0/2478203131 conn(0x7ff4e80a9c40 msgr2=0x7ff4e813d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:54.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4ed689640 1 -- 172.21.15.114:0/2478203131 shutdown_connections 2024-09-16T05:07:54.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4ed689640 1 -- 172.21.15.114:0/2478203131 wait complete. 2024-09-16T05:07:54.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:54.902+0000 7ff4ed689640 1 librados: shutdown 2024-09-16T05:07:54.936 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:07:54.936 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:07:54.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:07:55.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 10 monclient: get_monmap_and_config 2024-09-16T05:07:55.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 10 monclient: build_initial_monmap 2024-09-16T05:07:55.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 10 monclient: monmap: 2024-09-16T05:07:55.046 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:55.046 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:55.046 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:07:55.049503+0000 2024-09-16T05:07:55.046 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:07:55.049503+0000 2024-09-16T05:07:55.046 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:55.046 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:55.046 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:07:55.047 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:07:55.047 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:07:55.047 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:07:55.047 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:07:55.047 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:55.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 1 Processor -- start 2024-09-16T05:07:55.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 1 -- start start 2024-09-16T05:07:55.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 10 monclient: init 2024-09-16T05:07:55.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:55.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-16T05:07:55.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ac4098500 0x7f4ac40988e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:55.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 10 monclient(hunting): picked mon.noname-b con 0x7f4ac4098500 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:55.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ac409c9b0 addr v1:172.21.15.114:6789/0 2024-09-16T05:07:55.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ac409d4a0 addr v1:172.21.15.149:6789/0 2024-09-16T05:07:55.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ac409db80 con 0x7f4ac409c9b0 2024-09-16T05:07:55.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ac409e2d0 con 0x7f4ac409d4a0 2024-09-16T05:07:55.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:55.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4ac409e070 con 0x7f4ac4098500 2024-09-16T05:07:55.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:55.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4aca199640 10 monclient(hunting): authenticate will time out at 959.679131s 2024-09-16T05:07:55.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ac4098500 0x7f4ac40988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:55.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4abbfff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4ac409c9b0 0x7f4ac409cd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38736/0 (socket says 172.21.15.114:38736) 2024-09-16T05:07:55.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4abbfff640 1 -- 172.21.15.114:0/1125503774 learned_addr learned my addr 172.21.15.114:0/1125503774 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:07:55.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 10 monclient(hunting): get_auth_request con 0x7f4ac4098500 auth_method 0 2024-09-16T05:07:55.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:55.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:55.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:55.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:55.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:55.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:55.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac2ffd640 1 -- 172.21.15.114:0/1125503774 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 885575430 0 0) 0x7f4ac409db80 con 0x7f4ac409c9b0 2024-09-16T05:07:55.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 10 monclient(hunting): handle_auth_done global_id 4340 payload 274 2024-09-16T05:07:55.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 1 -- 172.21.15.114:0/1125503774 >> v1:172.21.15.149:6789/0 conn(0x7f4ac409d4a0 legacy=0x7f4ac413b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:07:55.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 1 -- 172.21.15.114:0/1125503774 >> v1:172.21.15.114:6789/0 conn(0x7f4ac409c9b0 legacy=0x7f4ac409cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:55.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:07:55.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 1 monclient: found mon.noname-b 2024-09-16T05:07:55.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:55.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:07:55.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac37fe640 1 -- 172.21.15.114:0/1125503774 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac409f270 con 0x7f4ac4098500 2024-09-16T05:07:55.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.046+0000 7f4ac2ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:07:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4ac37fe640 1 --2- 172.21.15.114:0/1125503774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ac4098500 0x7f4ac40988e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400ba80 tx=0x7f4ab401e7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fad4fffe0094ccd9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4ac2ffd640 1 -- 172.21.15.114:0/1125503774 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab40b0020 con 0x7f4ac4098500 2024-09-16T05:07:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4ac2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4ac2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:07:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4ac2ffd640 10 monclient: dump: 2024-09-16T05:07:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:55.059 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:55.060 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:55.060 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:55.060 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:55.060 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:55.060 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:55.060 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:55.060 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:55.060 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:55.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4ac2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:07:55.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4ac2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:07:55.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4ac2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:55.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4ac2ffd640 1 -- 172.21.15.114:0/1125503774 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ab40a8650 con 0x7f4ac4098500 2024-09-16T05:07:55.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 5 monclient: authenticate success, global_id 4340 2024-09-16T05:07:55.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:07:55.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4ac2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:55.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 10 monclient: get_monmap_and_config success 2024-09-16T05:07:55.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 4 set_mon_vals no callback set 2024-09-16T05:07:55.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4ac2ffd640 1 -- 172.21.15.114:0/1125503774 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab40a8a20 con 0x7f4ac4098500 2024-09-16T05:07:55.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4ac2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:55.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4ac2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:55.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4ac2ffd640 10 monclient: dump: 2024-09-16T05:07:55.061 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:55.061 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:55.061 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:55.062 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 10 monclient: shutdown 2024-09-16T05:07:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 1 -- 172.21.15.114:0/1125503774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ac4098500 msgr2=0x7f4ac40988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 1 --2- 172.21.15.114:0/1125503774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ac4098500 0x7f4ac40988e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400ba80 tx=0x7f4ab401e7f0 comp rx=0 tx=0).stop 2024-09-16T05:07:55.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 1 -- 172.21.15.114:0/1125503774 shutdown_connections 2024-09-16T05:07:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 1 --2- 172.21.15.114:0/1125503774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ac4098500 0x7f4ac40988e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 1 -- 172.21.15.114:0/1125503774 >> 172.21.15.114:0/1125503774 conn(0x7f4ac40a1fb0 msgr2=0x7f4ac40a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 1 -- 172.21.15.114:0/1125503774 shutdown_connections 2024-09-16T05:07:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 1 -- 172.21.15.114:0/1125503774 wait complete. 2024-09-16T05:07:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 10 monclient: build_initial_monmap 2024-09-16T05:07:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 10 monclient: monmap: 2024-09-16T05:07:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:07:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:07:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:07:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:07:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:07:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:07:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:07:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:07:55.064 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 1 librados: starting msgr at 2024-09-16T05:07:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 1 librados: starting objecter 2024-09-16T05:07:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 1 Processor -- start 2024-09-16T05:07:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 1 -- start start 2024-09-16T05:07:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 1 librados: setting wanted keys 2024-09-16T05:07:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 1 librados: calling monclient init 2024-09-16T05:07:55.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.050+0000 7f4aca199640 10 monclient: init 2024-09-16T05:07:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 monclient: _reopen_session rank -1 2024-09-16T05:07:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:07:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ac4098500 0x7f4ac41cf330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 monclient(hunting): picked mon.noname-a con 0x7f4ac4098500 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:07:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac409c9b0 0x7f4ac4097860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 monclient(hunting): picked mon.noname-b con 0x7f4ac409c9b0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:07:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ac409d4a0 0x7f4ac4070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 monclient(hunting): picked mon.noname-c con 0x7f4ac409d4a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:07:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac4099fa0 con 0x7f4ac4098500 2024-09-16T05:07:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac4099e20 con 0x7f4ac409c9b0 2024-09-16T05:07:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:55.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4ac413f000 con 0x7f4ac409d4a0 2024-09-16T05:07:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 monclient(hunting): _renew_subs 2024-09-16T05:07:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 monclient(hunting): authenticate will time out at 959.684179s 2024-09-16T05:07:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ac409d4a0 0x7f4ac4070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ac409d4a0 0x7f4ac4070970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55914/0 (socket says 172.21.15.114:55914) 2024-09-16T05:07:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 1 -- 172.21.15.114:0/3525265760 learned_addr learned my addr 172.21.15.114:0/3525265760 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:07:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 10 monclient(hunting): get_auth_request con 0x7f4ac409d4a0 auth_method 0 2024-09-16T05:07:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ac4098500 0x7f4ac41cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 10 monclient(hunting): get_auth_request con 0x7f4ac4098500 auth_method 0 2024-09-16T05:07:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4abbfff640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac409c9b0 0x7f4ac4097860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4abbfff640 10 monclient(hunting): get_auth_request con 0x7f4ac409c9b0 auth_method 0 2024-09-16T05:07:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4abbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4abbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4abbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 10 monclient(hunting): handle_auth_done global_id 4343 payload 824 2024-09-16T05:07:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 1 -- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac409c9b0 msgr2=0x7f4ac4097860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:55.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac409c9b0 0x7f4ac4097860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:55.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 1 -- 172.21.15.114:0/3525265760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ac4098500 msgr2=0x7f4ac41cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:55.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ac4098500 0x7f4ac41cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:55.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 10 monclient: _finish_hunting 0 2024-09-16T05:07:55.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 1 monclient: found mon.noname-c 2024-09-16T05:07:55.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:07:55.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:07:55.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 1 -- 172.21.15.114:0/3525265760 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac40a3520 con 0x7f4ac409d4a0 2024-09-16T05:07:55.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac3fff640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ac409d4a0 0x7f4ac4070970 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00c780 tx=0x7f4abc00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:55.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 client.?.objecter ms_handle_connect 0x7f4ac409d4a0 2024-09-16T05:07:55.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:07:55.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 1 -- 172.21.15.114:0/3525265760 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc004b70 con 0x7f4ac409d4a0 2024-09-16T05:07:55.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:55.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:07:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 monclient: dump: 2024-09-16T05:07:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:55.069 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:55.070 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:55.070 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:55.070 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:55.070 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:55.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 monclient: _finish_auth 0 2024-09-16T05:07:55.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:07:55.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:07:55.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 1 -- 172.21.15.114:0/3525265760 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4abc004d10 con 0x7f4ac409d4a0 2024-09-16T05:07:55.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:07:55.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 1 -- 172.21.15.114:0/3525265760 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc00aa90 con 0x7f4ac409d4a0 2024-09-16T05:07:55.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:55.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:07:55.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 monclient: dump: 2024-09-16T05:07:55.071 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:55.071 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:55.071 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:55.071 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:55.071 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:55.071 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:55.071 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:55.071 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:55.071 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:55.071 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:55.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 5 monclient: authenticate success, global_id 4343 2024-09-16T05:07:55.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 monclient: _renew_subs 2024-09-16T05:07:55.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac27fc640 4 set_mon_vals no callback set 2024-09-16T05:07:55.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:55.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 1 -- 172.21.15.114:0/3525265760 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac409c6a0 con 0x7f4ac409d4a0 2024-09-16T05:07:55.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:07:55.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 monclient: _renew_subs 2024-09-16T05:07:55.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:07:55.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 1 -- 172.21.15.114:0/3525265760 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac4092010 con 0x7f4ac409d4a0 2024-09-16T05:07:55.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 1 librados: init done 2024-09-16T05:07:55.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:55.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac4097da0 0x7f4ac402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:55.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:55.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4aca199640 1 -- 172.21.15.114:0/3525265760 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac4069390 con 0x7f4ac4097da0 2024-09-16T05:07:55.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 1 -- 172.21.15.114:0/3525265760 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4abc00ac30 con 0x7f4ac409d4a0 2024-09-16T05:07:55.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac4097da0 0x7f4ac402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:55.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 10 monclient: get_auth_request con 0x7f4ac4097da0 auth_method 0 2024-09-16T05:07:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 1 --2- 172.21.15.114:0/3525265760 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4aa4037810 0x7f4aa4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 1 -- 172.21.15.114:0/3525265760 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f4abc0488a0 con 0x7f4ac409d4a0 2024-09-16T05:07:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 client.4343.objecter ms_dispatch 0x7f4ac4000ba0 osd_map(19..19 src has 1..19) 2024-09-16T05:07:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 3 client.4343.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T05:07:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 3 client.4343.objecter handle_osd_map decoding full epoch 19 2024-09-16T05:07:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 20 client.4343.objecter dump_active .. 0 homeless 2024-09-16T05:07:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4abbfff640 1 --2- 172.21.15.114:0/3525265760 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4aa4037810 0x7f4aa4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4abbfff640 10 monclient: get_auth_request con 0x7f4aa4037810 auth_method 0 2024-09-16T05:07:55.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 10 monclient(hunting): handle_auth_done global_id 4375 payload 824 2024-09-16T05:07:55.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 1 -- 172.21.15.114:0/3525265760 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ac40988e0 con 0x7f4ac4097da0 2024-09-16T05:07:55.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 client.4343.objecter ms_handle_connect 0x7f4ac4097da0 2024-09-16T05:07:55.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 client.4343.objecter resend_mon_ops 2024-09-16T05:07:55.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac37fe640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac4097da0 0x7f4ac402c650 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4ac406cd00 tx=0x7f4ab401b590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:55.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 1 -- 172.21.15.114:0/3525265760 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab40bb070 con 0x7f4ac4097da0 2024-09-16T05:07:55.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:55.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:55.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 monclient: dump: 2024-09-16T05:07:55.074 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:55.074 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:55.074 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:55.074 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:55.074 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:55.074 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:55.075 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 client.4343.objecter ms_handle_connect 0x7f4aa4037810 2024-09-16T05:07:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4abbfff640 1 --2- 172.21.15.114:0/3525265760 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4aa4037810 0x7f4aa4039cd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4aac007fd0 tx=0x7f4aac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 1 -- 172.21.15.114:0/3525265760 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4ab40b5420 con 0x7f4ac4097da0 2024-09-16T05:07:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:07:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:07:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 1 -- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac4097da0 msgr2=0x7f4ac402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac4097da0 0x7f4ac402c650 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4ac406cd00 tx=0x7f4ab401b590 comp rx=0 tx=0).stop 2024-09-16T05:07:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 1 -- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac4097da0 msgr2=0x7f4ac402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:07:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.054+0000 7f4ac0ff9640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac4097da0 0x7f4ac402c650 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4ac406cd00 tx=0x7f4ab401b590 comp rx=0 tx=0).stop 2024-09-16T05:07:55.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4aca199640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:07:55.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4aca199640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac402cb90 0x7f4ac402cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:07:55.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4aca199640 10 monclient(hunting): start opening mon connection 2024-09-16T05:07:55.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4aca199640 1 -- 172.21.15.114:0/3525265760 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac413c730 con 0x7f4ac402cb90 2024-09-16T05:07:55.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac37fe640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac402cb90 0x7f4ac402cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:07:55.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac37fe640 10 monclient: get_auth_request con 0x7f4ac402cb90 auth_method 0 2024-09-16T05:07:55.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:07:55.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:07:55.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:07:55.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:07:55.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:07:55.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:07:55.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac37fe640 10 monclient(hunting): handle_auth_done global_id 4378 payload 824 2024-09-16T05:07:55.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac37fe640 1 -- 172.21.15.114:0/3525265760 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4ac40686f0 con 0x7f4ac402cb90 2024-09-16T05:07:55.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac37fe640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac402cb90 0x7f4ac402cfa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400ba80 tx=0x7f4ab40ad000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:07:55.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac0ff9640 10 client.4343.objecter ms_handle_connect 0x7f4ac402cb90 2024-09-16T05:07:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac0ff9640 10 client.4343.objecter resend_mon_ops 2024-09-16T05:07:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac0ff9640 1 -- 172.21.15.114:0/3525265760 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab40b0040 con 0x7f4ac402cb90 2024-09-16T05:07:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:07:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:07:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac0ff9640 10 monclient: dump: 2024-09-16T05:07:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:07:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:07:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:07:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:07:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:07:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:07:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:07:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:07:55.080 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:07:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac0ff9640 1 -- 172.21.15.114:0/3525265760 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4886 (secure 0 0 0) 0x7f4ab40c6870 con 0x7f4ac402cb90 2024-09-16T05:07:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:07:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:07:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac0ff9640 1 -- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac402cb90 msgr2=0x7f4ac402cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac0ff9640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac402cb90 0x7f4ac402cfa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400ba80 tx=0x7f4ab40ad000 comp rx=0 tx=0).stop 2024-09-16T05:07:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac0ff9640 1 -- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac402cb90 msgr2=0x7f4ac402cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:07:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4ac0ff9640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ac402cb90 0x7f4ac402cfa0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400ba80 tx=0x7f4ab40ad000 comp rx=0 tx=0).stop 2024-09-16T05:07:55.080 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:07:55.080 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:07:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:07:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:07:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 8, 2024-09-16T05:07:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 33007, 2024-09-16T05:07:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:07:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:07:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:07:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:07:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 27, 2024-09-16T05:07:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:07:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:07:55.081 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:07:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:07:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:07:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:07:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:07:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:07:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:07:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:07:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:07:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:07:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:07:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:07:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:07:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:07:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:07:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:07:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:07:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:07:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:07:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:07:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:07:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:07:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:07:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:07:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:07:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:55.083 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:07:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:07:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:07:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:07:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:07:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:07:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:07:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:07:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:07:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:07:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:07:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:07:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:07:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:07:55.084 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:07:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:07:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:07:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:07:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:07:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:07:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:07:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:07:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:07:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:07:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:07:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:07:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:07:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:07:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:07:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:07:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:07:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:07:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:07:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:07:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:07:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:07:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:55.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:07:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:07:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:07:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:07:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:07:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:07:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:07:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:55.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:07:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:07:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:07:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:07:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:07:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:07:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:07:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:07:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:55.088 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:07:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:07:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:07:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:07:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:07:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:07:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:07:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:07:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:07:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:07:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:07:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:07:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:07:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:07:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:07:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:07:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:07:55.090 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:07:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:07:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:07:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:07:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:07:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:07:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:07:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:07:55.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:07:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:07:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:07:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:07:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:07:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:07:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:07:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:07:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:07:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:07:55.092 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:07:55.093 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:07:55.093 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:07:55.093 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:07:55.093 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:07:55.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:07:55.093 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:07:55.093 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:07:55.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4aca199640 10 librados: watch_flush enter 2024-09-16T05:07:55.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4aca199640 10 librados: watch_flush exit 2024-09-16T05:07:55.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4aca199640 20 client.4343.objecter shutdown clearing up homeless session... 2024-09-16T05:07:55.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4aca199640 10 client.4343.objecter successfully canceled tick 2024-09-16T05:07:55.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4aca199640 1 -- 172.21.15.114:0/3525265760 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4aa4037810 msgr2=0x7f4aa4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:55.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.062+0000 7f4aca199640 1 --2- 172.21.15.114:0/3525265760 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4aa4037810 0x7f4aa4039cd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4aac007fd0 tx=0x7f4aac009040 comp rx=0 tx=0).stop 2024-09-16T05:07:55.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.066+0000 7f4aca199640 10 monclient: shutdown 2024-09-16T05:07:55.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.066+0000 7f4aca199640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:07:55.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.066+0000 7f4aca199640 1 -- 172.21.15.114:0/3525265760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ac409d4a0 msgr2=0x7f4ac4070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:07:55.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.066+0000 7f4aca199640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ac409d4a0 0x7f4ac4070970 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00c780 tx=0x7f4abc00cc40 comp rx=0 tx=0).stop 2024-09-16T05:07:55.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.066+0000 7f4ac3fff640 1 -- 172.21.15.114:0/3525265760 reap_dead start 2024-09-16T05:07:55.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.066+0000 7f4aca199640 1 -- 172.21.15.114:0/3525265760 shutdown_connections 2024-09-16T05:07:55.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.066+0000 7f4aca199640 1 --2- 172.21.15.114:0/3525265760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4ac409d4a0 0x7f4ac4070970 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:07:55.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.066+0000 7f4aca199640 1 -- 172.21.15.114:0/3525265760 >> 172.21.15.114:0/3525265760 conn(0x7f4ac40a1fb0 msgr2=0x7f4ac4139900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:07:55.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.066+0000 7f4aca199640 1 -- 172.21.15.114:0/3525265760 shutdown_connections 2024-09-16T05:07:55.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.066+0000 7f4aca199640 1 -- 172.21.15.114:0/3525265760 wait complete. 2024-09-16T05:07:55.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:07:55.066+0000 7f4aca199640 1 librados: shutdown 2024-09-16T05:07:55.096 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:07:55.097 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T05:07:55.097 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-16T05:07:55.097 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T05:07:55.097 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T05:07:55.097 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T05:07:55.097 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:07:55.107 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T05:07:55.107 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T05:07:55.107 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T05:07:55.108 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T05:07:55.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-09-16T05:08:01.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-09-16T05:08:10.108 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T05:08:10.109 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T05:08:10.109 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:08:10.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:08:10.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 10 monclient: get_monmap_and_config 2024-09-16T05:08:10.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 10 monclient: build_initial_monmap 2024-09-16T05:08:10.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 10 monclient: monmap: 2024-09-16T05:08:10.238 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:08:10.238 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:10.238 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:08:10.241530+0000 2024-09-16T05:08:10.238 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:08:10.241530+0000 2024-09-16T05:08:10.238 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:08:10.238 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:10.238 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:08:10.238 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:08:10.239 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:08:10.239 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:08:10.239 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:08:10.239 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:10.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 1 Processor -- start 2024-09-16T05:08:10.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 1 -- start start 2024-09-16T05:08:10.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 10 monclient: init 2024-09-16T05:08:10.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 10 monclient: _reopen_session rank -1 2024-09-16T05:08:10.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-16T05:08:10.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 0x7f44740a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:10.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 10 monclient(hunting): picked mon.noname-c con 0x7f447409b6c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:08:10.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44740a1400 0x7f44740a1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:10.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 10 monclient(hunting): picked mon.noname-b con 0x7f44740a1400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:08:10.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4474139640 addr v1:172.21.15.114:6789/0 2024-09-16T05:08:10.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f447409d4e0 con 0x7f4474139640 2024-09-16T05:08:10.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:10.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f447409e390 con 0x7f447409b6c0 2024-09-16T05:08:10.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:10.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f44740a1d50 con 0x7f44740a1400 2024-09-16T05:08:10.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 10 monclient(hunting): _renew_subs 2024-09-16T05:08:10.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f4478ab1640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44740a1400 msgr2=0x7f44740a1810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:08:10.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f4478ab1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44740a1400 0x7f44740a1810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:08:10.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f447b516640 10 monclient(hunting): authenticate will time out at 974.870892s 2024-09-16T05:08:10.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f4479ab3640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4474139640 0x7f447413ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36016/0 (socket says 172.21.15.114:36016) 2024-09-16T05:08:10.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f4479ab3640 1 -- 172.21.15.114:0/3808107088 learned_addr learned my addr 172.21.15.114:0/3808107088 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:08:10.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 0x7f44740a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:10.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f44792b2640 10 monclient(hunting): get_auth_request con 0x7f447409b6c0 auth_method 0 2024-09-16T05:08:10.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f44792b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:10.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f44792b2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:10.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f44792b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:10.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 msgr2=0x7f44740a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:08:10.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 msgr2=0x7f44740a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:08:10.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 0x7f44740a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:08:10.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.238+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 0x7f44740a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:08:10.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.442+0000 7f4478ab1640 1 -- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44740a1400 msgr2=0x7f44740a1810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:08:10.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.442+0000 7f4478ab1640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44740a1400 0x7f44740a1810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:08:10.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.442+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 0x7f44740a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:10.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.442+0000 7f44792b2640 10 monclient(hunting): get_auth_request con 0x7f447409b6c0 auth_method 0 2024-09-16T05:08:10.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.442+0000 7f44792b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:10.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.442+0000 7f44792b2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:10.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.442+0000 7f44792b2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:08:10.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.442+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 msgr2=0x7f44740a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:08:10.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.442+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 msgr2=0x7f44740a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:08:10.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.442+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 0x7f44740a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:08:10.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.442+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 0x7f44740a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:08:10.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.846+0000 7f4478ab1640 1 -- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44740a1400 msgr2=0x7f44740a1810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:08:10.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.846+0000 7f4478ab1640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44740a1400 0x7f44740a1810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:08:10.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.846+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 0x7f44740a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:10.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.846+0000 7f44792b2640 10 monclient(hunting): get_auth_request con 0x7f447409b6c0 auth_method 0 2024-09-16T05:08:10.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.846+0000 7f44792b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:10.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.846+0000 7f44792b2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:10.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.846+0000 7f44792b2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:08:10.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.846+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 msgr2=0x7f44740a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:08:10.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.846+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 msgr2=0x7f44740a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:08:10.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.846+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 0x7f44740a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:08:10.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:10.846+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 0x7f44740a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:08:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:11.650+0000 7f4478ab1640 1 -- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44740a1400 msgr2=0x7f44740a1810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:08:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:11.650+0000 7f4478ab1640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44740a1400 0x7f44740a1810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:08:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:11.650+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 0x7f44740a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:11.650+0000 7f44792b2640 10 monclient(hunting): get_auth_request con 0x7f447409b6c0 auth_method 0 2024-09-16T05:08:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:11.650+0000 7f44792b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:11.650+0000 7f44792b2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:11.650+0000 7f44792b2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:08:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:11.650+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 msgr2=0x7f44740a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:08:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:11.650+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 msgr2=0x7f44740a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:08:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:11.650+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 0x7f44740a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:08:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:11.650+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 0x7f44740a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:08:13.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 10 monclient(hunting): tick 2024-09-16T05:08:13.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:08:13.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 1 monclient(hunting): continuing hunt 2024-09-16T05:08:13.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:08:13.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 1 -- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44740a1400 msgr2=0x7f44740a1810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:08:13.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44740a1400 0x7f44740a1810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:13.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 msgr2=0x7f44740a0ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:08:13.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.149:3300/0 conn(0x7f447409b6c0 0x7f44740a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:13.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 1 -- 172.21.15.114:0/3808107088 >> v1:172.21.15.114:6789/0 conn(0x7f4474139640 legacy=0x7f447413ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:13.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 10 monclient: _add_conns ranks=[3,1] 2024-09-16T05:08:13.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 0x7f4458005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:13.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 10 monclient(hunting): picked mon.noname-a con 0x7f44580036f0 addr v2:172.21.15.114:3300/0 2024-09-16T05:08:13.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4458006240 addr v1:172.21.15.149:6789/0 2024-09-16T05:08:13.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 1 -- 172.21.15.114:0/3808107088 --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f447409f430 con 0x7f4458006240 2024-09-16T05:08:13.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:13.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 1 -- 172.21.15.114:0/3808107088 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f447409e390 con 0x7f44580036f0 2024-09-16T05:08:13.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f446affd640 10 monclient(hunting): _renew_subs 2024-09-16T05:08:13.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 0x7f4458005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:13.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f44792b2640 10 monclient(hunting): get_auth_request con 0x7f44580036f0 auth_method 0 2024-09-16T05:08:13.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f44792b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:13.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f44792b2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:13.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f44792b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:13.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 msgr2=0x7f4458005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:08:13.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 msgr2=0x7f4458005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:08:13.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 0x7f4458005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:08:13.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.242+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 0x7f4458005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:08:13.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.446+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 0x7f4458005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:13.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.446+0000 7f44792b2640 10 monclient(hunting): get_auth_request con 0x7f44580036f0 auth_method 0 2024-09-16T05:08:13.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.446+0000 7f44792b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:13.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.446+0000 7f44792b2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:13.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.446+0000 7f44792b2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:08:13.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.446+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 msgr2=0x7f4458005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:08:13.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.446+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 msgr2=0x7f4458005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:08:13.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.446+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 0x7f4458005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:08:13.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.446+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 0x7f4458005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:08:13.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.850+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 0x7f4458005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:13.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.850+0000 7f44792b2640 10 monclient(hunting): get_auth_request con 0x7f44580036f0 auth_method 0 2024-09-16T05:08:13.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.850+0000 7f44792b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:13.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.850+0000 7f44792b2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:13.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.850+0000 7f44792b2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:08:13.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.850+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 msgr2=0x7f4458005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:08:13.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.850+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 msgr2=0x7f4458005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:08:13.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.850+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 0x7f4458005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:08:13.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:13.850+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 0x7f4458005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:08:14.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:14.654+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 0x7f4458005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:14.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:14.654+0000 7f44792b2640 10 monclient(hunting): get_auth_request con 0x7f44580036f0 auth_method 0 2024-09-16T05:08:14.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:14.654+0000 7f44792b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:14.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:14.654+0000 7f44792b2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:14.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:14.654+0000 7f44792b2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:08:14.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:14.654+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 msgr2=0x7f4458005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:08:14.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:14.654+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 msgr2=0x7f4458005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:08:14.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:14.654+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 0x7f4458005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:08:14.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:14.654+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 0x7f4458005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:08:15.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 1 -- 172.21.15.114:0/3808107088 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3818324052 0 0) 0x7f447409f430 con 0x7f4458006240 2024-09-16T05:08:15.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:08:15.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:15.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 10 monclient(hunting): my global_id is 4381 2024-09-16T05:08:15.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 1 -- 172.21.15.114:0/3808107088 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4460003530 con 0x7f4458006240 2024-09-16T05:08:15.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 1 -- 172.21.15.114:0/3808107088 <== mon.1 v1:172.21.15.149:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1072225319 0 0) 0x7f4460003530 con 0x7f4458006240 2024-09-16T05:08:15.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 1 -- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 msgr2=0x7f4458005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:08:15.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 1 --2- 172.21.15.114:0/3808107088 >> v2:172.21.15.114:3300/0 conn(0x7f44580036f0 0x7f4458005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:08:15.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 1 monclient: found mon.noname-c-legacy 2024-09-16T05:08:15.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:08:15.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.149:6789/0 2024-09-16T05:08:15.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 1 -- 172.21.15.114:0/3808107088 --> v1:172.21.15.149:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445800afc0 con 0x7f4458006240 2024-09-16T05:08:15.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 10 monclient: _finish_auth 0 2024-09-16T05:08:15.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:08:15.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:08:15.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 1 -- 172.21.15.114:0/3808107088 <== mon.1 v1:172.21.15.149:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f44700043a0 con 0x7f4458006240 2024-09-16T05:08:15.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T05:08:15.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 10 monclient: dump: 2024-09-16T05:08:15.163 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.163 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.163 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.164 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.164 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.164 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.164 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.164 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.164 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.164 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] but i'm connected to v1:172.21.15.149:6789/0, reconnecting 2024-09-16T05:08:15.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 10 monclient: _reopen_session rank -1 2024-09-16T05:08:15.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 1 -- 172.21.15.114:0/3808107088 >> v1:172.21.15.149:6789/0 conn(0x7f4458006240 legacy=0x7f4458008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f4479ab3640 1 -- 172.21.15.114:0/3808107088 reap_dead start 2024-09-16T05:08:15.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:08:15.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44600039c0 0x7f4460003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.158+0000 7f446bfff640 10 monclient(hunting): picked mon.c con 0x7f44600039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:08:15.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f446bfff640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460004810 0x7f4460004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f446bfff640 10 monclient(hunting): picked mon.a con 0x7f4460004810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:08:15.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f446bfff640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4460005220 0x7f446000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f446bfff640 10 monclient(hunting): picked mon.b con 0x7f4460005220 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:08:15.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f446bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f446bfff640 1 -- 172.21.15.114:0/3808107088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f447409e390 con 0x7f4460004810 2024-09-16T05:08:15.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f446bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f44792b2640 1 -- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44600039c0 msgr2=0x7f4460003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:08:15.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f446bfff640 1 -- 172.21.15.114:0/3808107088 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f446000c150 con 0x7f4460005220 2024-09-16T05:08:15.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f446bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f446bfff640 1 -- 172.21.15.114:0/3808107088 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f446000c350 con 0x7f44600039c0 2024-09-16T05:08:15.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f44792b2640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44600039c0 0x7f4460003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:08:15.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f446bfff640 10 monclient(hunting): _renew_subs 2024-09-16T05:08:15.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f446bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:08:15.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4460005220 0x7f446000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 10 monclient(hunting): get_auth_request con 0x7f4460005220 auth_method 0 2024-09-16T05:08:15.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:08:15.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:15.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:15.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:08:15.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 10 monclient(hunting): handle_auth_done global_id 4381 payload 293 2024-09-16T05:08:15.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 1 -- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44600039c0 msgr2=0x7f4460003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:08:15.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44600039c0 0x7f4460003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 1 -- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460004810 msgr2=0x7f4460004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:08:15.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460004810 0x7f4460004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 10 monclient: _finish_hunting 0 2024-09-16T05:08:15.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 1 monclient: found mon.b 2024-09-16T05:08:15.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:08:15.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:08:15.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 1 -- 172.21.15.114:0/3808107088 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4460003530 con 0x7f4460005220 2024-09-16T05:08:15.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 10 monclient: _finish_auth 0 2024-09-16T05:08:15.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 10 monclient: _check_auth_tickets 2024-09-16T05:08:15.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f4479ab3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:08:15.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f447b516640 5 monclient: authenticate success, global_id 4381 2024-09-16T05:08:15.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.162+0000 7f447b516640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:08:15.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f4479ab3640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4460005220 0x7f446000b9d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f447409c630 tx=0x7f446c01c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cefe5fe9809f9acf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:15.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f446bfff640 1 -- 172.21.15.114:0/3808107088 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c001df0 con 0x7f4460005220 2024-09-16T05:08:15.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f446bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f446bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:08:15.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f446bfff640 10 monclient: dump: 2024-09-16T05:08:15.171 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.171 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.171 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.171 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.171 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.171 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.172 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.172 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.172 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.172 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f447b516640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:08:15.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f446bfff640 1 -- 172.21.15.114:0/3808107088 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f446c0ae070 con 0x7f4460005220 2024-09-16T05:08:15.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f446bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:08:15.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f447b516640 10 monclient: get_monmap_and_config success 2024-09-16T05:08:15.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f446bfff640 1 -- 172.21.15.114:0/3808107088 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c0a9640 con 0x7f4460005220 2024-09-16T05:08:15.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f447b516640 4 set_mon_vals no callback set 2024-09-16T05:08:15.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f446bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f446bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:08:15.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f446bfff640 10 monclient: dump: 2024-09-16T05:08:15.173 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.173 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.173 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.173 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.173 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.173 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.173 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.173 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.173 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.173 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f447b516640 10 monclient: shutdown 2024-09-16T05:08:15.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f447b516640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:08:15.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f447b516640 1 -- 172.21.15.114:0/3808107088 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4460005220 msgr2=0x7f446000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.170+0000 7f447b516640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4460005220 0x7f446000b9d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f447409c630 tx=0x7f446c01c820 comp rx=0 tx=0).stop 2024-09-16T05:08:15.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 1 -- 172.21.15.114:0/3808107088 shutdown_connections 2024-09-16T05:08:15.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460004810 0x7f4460004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4460005220 0x7f446000b9d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 1 --2- 172.21.15.114:0/3808107088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44600039c0 0x7f4460003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 1 -- 172.21.15.114:0/3808107088 >> 172.21.15.114:0/3808107088 conn(0x7f44740a1fb0 msgr2=0x7f44740a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:08:15.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 1 -- 172.21.15.114:0/3808107088 shutdown_connections 2024-09-16T05:08:15.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 1 -- 172.21.15.114:0/3808107088 wait complete. 2024-09-16T05:08:15.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 10 monclient: build_initial_monmap 2024-09-16T05:08:15.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 10 monclient: monmap: 2024-09-16T05:08:15.175 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:08:15.175 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:08:15.176 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:08:15.176 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:08:15.176 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:08:15.176 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.176 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:08:15.176 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:08:15.176 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:08:15.176 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 1 librados: starting msgr at 2024-09-16T05:08:15.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 1 librados: starting objecter 2024-09-16T05:08:15.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 1 Processor -- start 2024-09-16T05:08:15.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 1 -- start start 2024-09-16T05:08:15.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 1 librados: setting wanted keys 2024-09-16T05:08:15.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 1 librados: calling monclient init 2024-09-16T05:08:15.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.174+0000 7f447b516640 10 monclient: init 2024-09-16T05:08:15.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient: _reopen_session rank -1 2024-09-16T05:08:15.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:08:15.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44600039c0 0x7f4474072bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient(hunting): picked mon.noname-c con 0x7f44600039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:08:15.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460004810 0x7f4474134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient(hunting): picked mon.noname-a con 0x7f4460004810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:08:15.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4460005220 0x7f44741c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f44792b2640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44600039c0 msgr2=0x7f4474072bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:08:15.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient(hunting): picked mon.noname-b con 0x7f4460005220 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:08:15.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f447409cad0 con 0x7f4460004810 2024-09-16T05:08:15.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f447409baa0 con 0x7f4460005220 2024-09-16T05:08:15.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4474137e40 con 0x7f44600039c0 2024-09-16T05:08:15.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f44792b2640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44600039c0 0x7f4474072bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:08:15.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient(hunting): _renew_subs 2024-09-16T05:08:15.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient(hunting): authenticate will time out at 979.808571s 2024-09-16T05:08:15.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460004810 0x7f4474134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460004810 0x7f4474134c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33808/0 (socket says 172.21.15.114:33808) 2024-09-16T05:08:15.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 1 -- 172.21.15.114:0/3967153750 learned_addr learned my addr 172.21.15.114:0/3967153750 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:08:15.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 10 monclient(hunting): get_auth_request con 0x7f4460004810 auth_method 0 2024-09-16T05:08:15.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:15.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4479ab3640 1 --2- 172.21.15.114:0/3967153750 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4460005220 0x7f44741c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4479ab3640 10 monclient(hunting): get_auth_request con 0x7f4460005220 auth_method 0 2024-09-16T05:08:15.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4479ab3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4479ab3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4479ab3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:15.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:15.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:15.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:15.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4479ab3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4479ab3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4479ab3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 10 monclient(hunting): handle_auth_done global_id 4422 payload 824 2024-09-16T05:08:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 1 -- 172.21.15.114:0/3967153750 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44600039c0 msgr2=0x7f4474072bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:08:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 1 --2- 172.21.15.114:0/3967153750 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44600039c0 0x7f4474072bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 1 -- 172.21.15.114:0/3967153750 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4460005220 msgr2=0x7f44741c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 1 --2- 172.21.15.114:0/3967153750 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4460005220 0x7f44741c9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 10 monclient: _finish_hunting 0 2024-09-16T05:08:15.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 1 monclient: found mon.noname-a 2024-09-16T05:08:15.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:08:15.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:08:15.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 1 -- 172.21.15.114:0/3967153750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f447412c760 con 0x7f4460004810 2024-09-16T05:08:15.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4478ab1640 1 --2- 172.21.15.114:0/3967153750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460004810 0x7f4474134c80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4470003cc0 tx=0x7f447000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:15.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 10 client.?.objecter ms_handle_connect 0x7f4460004810 2024-09-16T05:08:15.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:08:15.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 1 -- 172.21.15.114:0/3967153750 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4470018070 con 0x7f4460004810 2024-09-16T05:08:15.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:08:15.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 10 monclient: dump: 2024-09-16T05:08:15.188 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.188 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.188 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.188 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.188 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.188 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.188 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.188 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.188 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.189 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 10 monclient: _finish_auth 0 2024-09-16T05:08:15.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:08:15.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:08:15.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 1 -- 172.21.15.114:0/3967153750 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44700072d0 con 0x7f4460004810 2024-09-16T05:08:15.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:08:15.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 1 -- 172.21.15.114:0/3967153750 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447000a880 con 0x7f4460004810 2024-09-16T05:08:15.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 5 monclient: authenticate success, global_id 4422 2024-09-16T05:08:15.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient: _renew_subs 2024-09-16T05:08:15.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:08:15.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 1 -- 172.21.15.114:0/3967153750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f447412ce50 con 0x7f4460004810 2024-09-16T05:08:15.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f446affd640 4 set_mon_vals no callback set 2024-09-16T05:08:15.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:08:15.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 10 monclient: dump: 2024-09-16T05:08:15.190 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.190 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.190 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.190 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.190 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.190 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.190 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.191 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.191 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.191 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:08:15.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient: _renew_subs 2024-09-16T05:08:15.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:08:15.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 1 -- 172.21.15.114:0/3967153750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44741351c0 con 0x7f4460004810 2024-09-16T05:08:15.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 1 librados: init done 2024-09-16T05:08:15.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:08:15.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:08:15.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:08:15.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f447b516640 1 -- 172.21.15.114:0/3967153750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44740a1bf0 con 0x7f4460004810 2024-09-16T05:08:15.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.178+0000 7f4469ffb640 1 -- 172.21.15.114:0/3967153750 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f447001b070 con 0x7f4460004810 2024-09-16T05:08:15.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.182+0000 7f4469ffb640 1 --2- 172.21.15.114:0/3967153750 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4458038010 0x7f445803a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.182+0000 7f4469ffb640 1 -- 172.21.15.114:0/3967153750 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f4470047b10 con 0x7f4460004810 2024-09-16T05:08:15.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.182+0000 7f4469ffb640 10 client.4422.objecter ms_dispatch 0x7f4474000ba0 osd_map(19..19 src has 1..19) 2024-09-16T05:08:15.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.182+0000 7f4469ffb640 3 client.4422.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T05:08:15.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.182+0000 7f4469ffb640 3 client.4422.objecter handle_osd_map decoding full epoch 19 2024-09-16T05:08:15.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.182+0000 7f4469ffb640 20 client.4422.objecter dump_active .. 0 homeless 2024-09-16T05:08:15.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.182+0000 7f44792b2640 1 --2- 172.21.15.114:0/3967153750 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4458038010 0x7f445803a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.182+0000 7f44792b2640 10 monclient: get_auth_request con 0x7f4458038010 auth_method 0 2024-09-16T05:08:15.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.182+0000 7f4469ffb640 10 client.4422.objecter ms_handle_connect 0x7f4458038010 2024-09-16T05:08:15.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.182+0000 7f44792b2640 1 --2- 172.21.15.114:0/3967153750 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4458038010 0x7f445803a4d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4464000ee0 tx=0x7f4464008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:15.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.190+0000 7f4469ffb640 1 -- 172.21.15.114:0/3967153750 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f447000ac00 con 0x7f4460004810 2024-09-16T05:08:15.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.190+0000 7f4469ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:08:15.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.190+0000 7f4469ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:08:15.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:08:15.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:08:15.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:08:15.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 1 -- 172.21.15.114:0/3967153750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f44741353f0 con 0x7f4460004810 2024-09-16T05:08:15.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f4469ffb640 1 -- 172.21.15.114:0/3967153750 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f4470004940 con 0x7f4460004810 2024-09-16T05:08:15.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f4469ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:08:15.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f4469ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:08:15.410 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":10,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:08:15.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 10 librados: watch_flush enter 2024-09-16T05:08:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 10 librados: watch_flush exit 2024-09-16T05:08:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 20 client.4422.objecter shutdown clearing up homeless session... 2024-09-16T05:08:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 10 client.4422.objecter successfully canceled tick 2024-09-16T05:08:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 1 -- 172.21.15.114:0/3967153750 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4458038010 msgr2=0x7f445803a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 1 --2- 172.21.15.114:0/3967153750 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4458038010 0x7f445803a4d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4464000ee0 tx=0x7f4464008040 comp rx=0 tx=0).stop 2024-09-16T05:08:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 10 monclient: shutdown 2024-09-16T05:08:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:08:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 1 -- 172.21.15.114:0/3967153750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460004810 msgr2=0x7f4474134c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 1 --2- 172.21.15.114:0/3967153750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460004810 0x7f4474134c80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4470003cc0 tx=0x7f447000cb90 comp rx=0 tx=0).stop 2024-09-16T05:08:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 1 -- 172.21.15.114:0/3967153750 shutdown_connections 2024-09-16T05:08:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 1 --2- 172.21.15.114:0/3967153750 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4458038010 0x7f445803a4d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 1 --2- 172.21.15.114:0/3967153750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460004810 0x7f4474134c80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 1 --2- 172.21.15.114:0/3967153750 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4460005220 0x7f44741c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 1 --2- 172.21.15.114:0/3967153750 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44600039c0 0x7f4474072bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 1 -- 172.21.15.114:0/3967153750 >> 172.21.15.114:0/3967153750 conn(0x7f44740a1fb0 msgr2=0x7f44740a32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:08:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 1 -- 172.21.15.114:0/3967153750 shutdown_connections 2024-09-16T05:08:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 1 -- 172.21.15.114:0/3967153750 wait complete. 2024-09-16T05:08:15.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.410+0000 7f447b516640 1 librados: shutdown 2024-09-16T05:08:15.427 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:08:15.427 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T05:08:15.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:08:15.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 10 monclient: get_monmap_and_config 2024-09-16T05:08:15.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 10 monclient: build_initial_monmap 2024-09-16T05:08:15.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 10 monclient: monmap: 2024-09-16T05:08:15.549 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:08:15.549 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.549 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:08:15.552715+0000 2024-09-16T05:08:15.549 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:08:15.552715+0000 2024-09-16T05:08:15.549 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:08:15.550 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.550 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:08:15.550 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:08:15.550 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:08:15.550 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:08:15.550 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:08:15.550 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 1 Processor -- start 2024-09-16T05:08:15.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 1 -- start start 2024-09-16T05:08:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 10 monclient: init 2024-09-16T05:08:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 10 monclient: _reopen_session rank -1 2024-09-16T05:08:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-16T05:08:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff1b0137730 0x7ff1b0137b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 10 monclient(hunting): picked mon.noname-c con 0x7ff1b0137730 addr v2:172.21.15.149:3300/0 2024-09-16T05:08:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1b00a5d00 addr v1:172.21.15.149:6789/0 2024-09-16T05:08:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff1b00a4980 0x7ff1b013f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 10 monclient(hunting): picked mon.noname-a con 0x7ff1b00a4980 addr v2:172.21.15.114:3300/0 2024-09-16T05:08:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1b00a55e0 con 0x7ff1b00a5d00 2024-09-16T05:08:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff1b009c450 con 0x7ff1b00a4980 2024-09-16T05:08:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7ff1b00a3710 con 0x7ff1b0137730 2024-09-16T05:08:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 10 monclient(hunting): _renew_subs 2024-09-16T05:08:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1b542a640 10 monclient(hunting): authenticate will time out at 980.181905s 2024-09-16T05:08:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff1b00a4980 0x7ff1b013f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff1b00a4980 0x7ff1b013f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33824/0 (socket says 172.21.15.114:33824) 2024-09-16T05:08:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 1 -- 172.21.15.114:0/379664800 learned_addr learned my addr 172.21.15.114:0/379664800 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:08:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 10 monclient(hunting): get_auth_request con 0x7ff1b00a4980 auth_method 0 2024-09-16T05:08:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1aeffd640 1 --2- 172.21.15.114:0/379664800 >> v2:172.21.15.149:3300/0 conn(0x7ff1b0137730 0x7ff1b0137b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1aeffd640 10 monclient(hunting): get_auth_request con 0x7ff1b0137730 auth_method 0 2024-09-16T05:08:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:15.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 10 monclient(hunting): handle_auth_done global_id 4425 payload 274 2024-09-16T05:08:15.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 1 -- 172.21.15.114:0/379664800 >> v2:172.21.15.149:3300/0 conn(0x7ff1b0137730 msgr2=0x7ff1b0137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:08:15.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 1 --2- 172.21.15.114:0/379664800 >> v2:172.21.15.149:3300/0 conn(0x7ff1b0137730 0x7ff1b0137b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 1 -- 172.21.15.114:0/379664800 >> v1:172.21.15.149:6789/0 conn(0x7ff1b00a5d00 legacy=0x7ff1b00a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:08:15.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 1 monclient: found mon.noname-a 2024-09-16T05:08:15.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:08:15.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:08:15.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.550+0000 7ff1af7fe640 1 -- 172.21.15.114:0/379664800 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b009be00 con 0x7ff1b00a4980 2024-09-16T05:08:15.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1af7fe640 1 --2- 172.21.15.114:0/379664800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b00a4980 0x7ff1b013f4e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00a73e0 tx=0x7ff1a801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d234077d66b208 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:15.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1ae7fc640 1 -- 172.21.15.114:0/379664800 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a80a5600 con 0x7ff1b00a4980 2024-09-16T05:08:15.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:08:15.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1ae7fc640 10 monclient: dump: 2024-09-16T05:08:15.566 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.566 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.566 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.566 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.567 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.567 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.567 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.567 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.567 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.567 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1ae7fc640 10 monclient: _finish_auth 0 2024-09-16T05:08:15.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1ae7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:08:15.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:08:15.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1ae7fc640 1 -- 172.21.15.114:0/379664800 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1a80a57a0 con 0x7ff1b00a4980 2024-09-16T05:08:15.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:08:15.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1ae7fc640 1 -- 172.21.15.114:0/379664800 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a80a5b70 con 0x7ff1b00a4980 2024-09-16T05:08:15.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 5 monclient: authenticate success, global_id 4425 2024-09-16T05:08:15.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient: get_monmap_and_config success 2024-09-16T05:08:15.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 4 set_mon_vals no callback set 2024-09-16T05:08:15.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:08:15.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1ae7fc640 10 monclient: dump: 2024-09-16T05:08:15.568 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.569 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient: shutdown 2024-09-16T05:08:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:08:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 -- 172.21.15.114:0/379664800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b00a4980 msgr2=0x7ff1b013f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 --2- 172.21.15.114:0/379664800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b00a4980 0x7ff1b013f4e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00a73e0 tx=0x7ff1a801c8b0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 -- 172.21.15.114:0/379664800 shutdown_connections 2024-09-16T05:08:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 --2- 172.21.15.114:0/379664800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b00a4980 0x7ff1b013f4e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 --2- 172.21.15.114:0/379664800 >> v2:172.21.15.149:3300/0 conn(0x7ff1b0137730 0x7ff1b0137b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 -- 172.21.15.114:0/379664800 >> 172.21.15.114:0/379664800 conn(0x7ff1b00a9c70 msgr2=0x7ff1b00aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:08:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 -- 172.21.15.114:0/379664800 shutdown_connections 2024-09-16T05:08:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 -- 172.21.15.114:0/379664800 wait complete. 2024-09-16T05:08:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient: build_initial_monmap 2024-09-16T05:08:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient: monmap: 2024-09-16T05:08:15.571 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:08:15.571 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:08:15.571 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:08:15.571 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:08:15.571 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:08:15.571 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.571 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:08:15.571 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:08:15.571 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:08:15.571 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 librados: starting msgr at 2024-09-16T05:08:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 librados: starting objecter 2024-09-16T05:08:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 Processor -- start 2024-09-16T05:08:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 -- start start 2024-09-16T05:08:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 librados: setting wanted keys 2024-09-16T05:08:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 librados: calling monclient init 2024-09-16T05:08:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient: init 2024-09-16T05:08:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient: _reopen_session rank -1 2024-09-16T05:08:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:08:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff1b00a4980 0x7ff1b013e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient(hunting): picked mon.noname-c con 0x7ff1b00a4980 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:08:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b00a5d00 0x7ff1b013e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient(hunting): picked mon.noname-a con 0x7ff1b00a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:08:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff1b0137730 0x7ff1b0138f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient(hunting): picked mon.noname-b con 0x7ff1b0137730 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:08:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b0137320 con 0x7ff1b00a5d00 2024-09-16T05:08:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b01371a0 con 0x7ff1b0137730 2024-09-16T05:08:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff1b0142240 con 0x7ff1b00a4980 2024-09-16T05:08:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient(hunting): _renew_subs 2024-09-16T05:08:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient(hunting): authenticate will time out at 980.186995s 2024-09-16T05:08:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1aeffd640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff1b00a4980 msgr2=0x7ff1b013e440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:08:15.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1aeffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff1b00a4980 0x7ff1b013e440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:08:15.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b00a5d00 0x7ff1b013e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b00a5d00 0x7ff1b013e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33830/0 (socket says 172.21.15.114:33830) 2024-09-16T05:08:15.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 1 -- 172.21.15.114:0/667805939 learned_addr learned my addr 172.21.15.114:0/667805939 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:08:15.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 10 monclient(hunting): get_auth_request con 0x7ff1b00a5d00 auth_method 0 2024-09-16T05:08:15.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:15.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:15.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:15.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:15.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1af7fe640 1 --2- 172.21.15.114:0/667805939 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff1b0137730 0x7ff1b0138f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1af7fe640 10 monclient(hunting): get_auth_request con 0x7ff1b0137730 auth_method 0 2024-09-16T05:08:15.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:15.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 10 monclient(hunting): handle_auth_done global_id 4428 payload 824 2024-09-16T05:08:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 1 -- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff1b00a4980 msgr2=0x7ff1b013e440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:08:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 1 --2- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff1b00a4980 0x7ff1b013e440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 1 -- 172.21.15.114:0/667805939 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff1b0137730 msgr2=0x7ff1b0138f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 1 --2- 172.21.15.114:0/667805939 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff1b0137730 0x7ff1b0138f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:08:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 1 monclient: found mon.noname-a 2024-09-16T05:08:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:08:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:08:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 1 -- 172.21.15.114:0/667805939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b013cf10 con 0x7ff1b00a5d00 2024-09-16T05:08:15.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a7fff640 1 --2- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b00a5d00 0x7ff1b013e980 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff19800c430 tx=0x7ff19800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:15.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a77fe640 10 client.?.objecter ms_handle_connect 0x7ff1b00a5d00 2024-09-16T05:08:15.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a77fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:08:15.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a77fe640 1 -- 172.21.15.114:0/667805939 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1980052d0 con 0x7ff1b00a5d00 2024-09-16T05:08:15.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:08:15.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a77fe640 10 monclient: dump: 2024-09-16T05:08:15.577 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.577 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.577 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.577 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.578 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.578 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.578 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.578 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.578 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.578 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a77fe640 10 monclient: _finish_auth 0 2024-09-16T05:08:15.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a77fe640 10 monclient: _check_auth_tickets 2024-09-16T05:08:15.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:08:15.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a77fe640 1 -- 172.21.15.114:0/667805939 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff198005470 con 0x7ff1b00a5d00 2024-09-16T05:08:15.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 5 monclient: authenticate success, global_id 4428 2024-09-16T05:08:15.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient: _renew_subs 2024-09-16T05:08:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:08:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 -- 172.21.15.114:0/667805939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b013f590 con 0x7ff1b00a5d00 2024-09-16T05:08:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:08:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a77fe640 1 -- 172.21.15.114:0/667805939 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff198007260 con 0x7ff1b00a5d00 2024-09-16T05:08:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:08:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1a77fe640 10 monclient: dump: 2024-09-16T05:08:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.580 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.580 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.580 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.580 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.580 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1adffb640 4 set_mon_vals no callback set 2024-09-16T05:08:15.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:08:15.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient: _renew_subs 2024-09-16T05:08:15.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:08:15.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 -- 172.21.15.114:0/667805939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1b009cce0 con 0x7ff1b00a5d00 2024-09-16T05:08:15.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.554+0000 7ff1b542a640 1 librados: init done 2024-09-16T05:08:15.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1b542a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:08:15.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1b542a640 1 --2- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b01cf4b0 0x7ff1b01394c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1b542a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1b542a640 1 -- 172.21.15.114:0/667805939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b0068f10 con 0x7ff1b01cf4b0 2024-09-16T05:08:15.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1aeffd640 1 --2- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b01cf4b0 0x7ff1b01394c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1aeffd640 10 monclient: get_auth_request con 0x7ff1b01cf4b0 auth_method 0 2024-09-16T05:08:15.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:15.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 1 -- 172.21.15.114:0/667805939 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff198007400 con 0x7ff1b00a5d00 2024-09-16T05:08:15.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:15.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:15.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:15.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 1 --2- 172.21.15.114:0/667805939 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff1900377c0 0x7ff190039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1aeffd640 10 monclient(hunting): handle_auth_done global_id 4431 payload 824 2024-09-16T05:08:15.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1aeffd640 1 -- 172.21.15.114:0/667805939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1b0137b10 con 0x7ff1b01cf4b0 2024-09-16T05:08:15.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 1 -- 172.21.15.114:0/667805939 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7ff198047b70 con 0x7ff1b00a5d00 2024-09-16T05:08:15.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 10 client.4428.objecter ms_dispatch 0x7ff1b0000ba0 osd_map(19..19 src has 1..19) 2024-09-16T05:08:15.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 3 client.4428.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T05:08:15.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 3 client.4428.objecter handle_osd_map decoding full epoch 19 2024-09-16T05:08:15.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 20 client.4428.objecter dump_active .. 0 homeless 2024-09-16T05:08:15.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1aeffd640 1 --2- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b01cf4b0 0x7ff1b01394c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009510 tx=0x7ff1a000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:15.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 10 client.4428.objecter ms_handle_connect 0x7ff1b01cf4b0 2024-09-16T05:08:15.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 10 client.4428.objecter resend_mon_ops 2024-09-16T05:08:15.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1af7fe640 1 --2- 172.21.15.114:0/667805939 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff1900377c0 0x7ff190039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1af7fe640 10 monclient: get_auth_request con 0x7ff1900377c0 auth_method 0 2024-09-16T05:08:15.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 1 -- 172.21.15.114:0/667805939 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0017070 con 0x7ff1b01cf4b0 2024-09-16T05:08:15.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:08:15.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 10 monclient: dump: 2024-09-16T05:08:15.584 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.584 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.584 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.584 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.584 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.584 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.584 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.584 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.584 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.584 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 1 -- 172.21.15.114:0/667805939 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff1a000c9b0 con 0x7ff1b01cf4b0 2024-09-16T05:08:15.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:08:15.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:08:15.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 1 -- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b01cf4b0 msgr2=0x7ff1b01394c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 1 --2- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b01cf4b0 0x7ff1b01394c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009510 tx=0x7ff1a000ec90 comp rx=0 tx=0).stop 2024-09-16T05:08:15.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 1 -- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b01cf4b0 msgr2=0x7ff1b01394c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:08:15.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 1 --2- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b01cf4b0 0x7ff1b01394c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1a77fe640 10 client.4428.objecter ms_handle_connect 0x7ff1900377c0 2024-09-16T05:08:15.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.558+0000 7ff1af7fe640 1 --2- 172.21.15.114:0/667805939 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff1900377c0 0x7ff190039c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff1a80ae000 tx=0x7ff1a80a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:15.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.562+0000 7ff1b542a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:08:15.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.562+0000 7ff1b542a640 1 --2- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b0139a00 0x7ff1b009f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.562+0000 7ff1b542a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.562+0000 7ff1b542a640 1 -- 172.21.15.114:0/667805939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b0139e10 con 0x7ff1b0139a00 2024-09-16T05:08:15.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.562+0000 7ff1aeffd640 1 --2- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b0139a00 0x7ff1b009f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.562+0000 7ff1aeffd640 10 monclient: get_auth_request con 0x7ff1b0139a00 auth_method 0 2024-09-16T05:08:15.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.562+0000 7ff1aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.562+0000 7ff1aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.562+0000 7ff1aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:15.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.562+0000 7ff1aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:15.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.562+0000 7ff1aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:15.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.562+0000 7ff1aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:15.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1aeffd640 10 monclient(hunting): handle_auth_done global_id 4434 payload 824 2024-09-16T05:08:15.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1aeffd640 1 -- 172.21.15.114:0/667805939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff1b013b9a0 con 0x7ff1b0139a00 2024-09-16T05:08:15.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1aeffd640 1 --2- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b0139a00 0x7ff1b009f1f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0019000 tx=0x7ff1a0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:15.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1a77fe640 10 client.4428.objecter ms_handle_connect 0x7ff1b0139a00 2024-09-16T05:08:15.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1a77fe640 10 client.4428.objecter resend_mon_ops 2024-09-16T05:08:15.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1a77fe640 1 -- 172.21.15.114:0/667805939 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0017070 con 0x7ff1b0139a00 2024-09-16T05:08:15.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:08:15.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1a77fe640 10 monclient: dump: 2024-09-16T05:08:15.587 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.587 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.587 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.587 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.587 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.587 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.587 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.587 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.587 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.587 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1a77fe640 1 -- 172.21.15.114:0/667805939 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7ff1a0006730 con 0x7ff1b0139a00 2024-09-16T05:08:15.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1a77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:08:15.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1a77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:08:15.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1a77fe640 1 -- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b0139a00 msgr2=0x7ff1b009f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1a77fe640 1 --2- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b0139a00 0x7ff1b009f1f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0019000 tx=0x7ff1a0004740 comp rx=0 tx=0).stop 2024-09-16T05:08:15.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1a77fe640 1 -- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b0139a00 msgr2=0x7ff1b009f1f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:08:15.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1a77fe640 1 --2- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b0139a00 0x7ff1b009f1f0 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0019000 tx=0x7ff1a0004740 comp rx=0 tx=0).stop 2024-09-16T05:08:15.588 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:08:15.588 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:08:15.588 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:08:15.589 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:08:15.589 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 10, 2024-09-16T05:08:15.589 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 53555, 2024-09-16T05:08:15.589 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:08:15.589 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:08:15.589 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-16T05:08:15.589 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:08:15.589 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:08:15.589 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:08:15.589 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:08:15.589 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:08:15.589 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:08:15.589 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:08:15.589 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:08:15.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:08:15.590 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:08:15.590 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:08:15.590 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:08:15.590 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:08:15.590 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:08:15.590 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:08:15.590 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:08:15.590 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:08:15.590 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:08:15.590 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:08:15.590 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:08:15.590 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:08:15.590 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:08:15.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:08:15.591 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:08:15.591 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:08:15.591 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:08:15.591 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:08:15.591 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:08:15.591 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:08:15.591 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:08:15.591 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:08:15.591 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.591 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:08:15.591 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:08:15.591 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:08:15.591 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:08:15.592 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:08:15.592 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:08:15.592 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:08:15.592 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:08:15.592 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:08:15.592 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:08:15.592 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:08:15.592 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:08:15.592 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:08:15.592 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:08:15.592 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:08:15.592 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:08:15.592 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:08:15.592 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:08:15.593 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:08:15.593 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:08:15.593 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:08:15.593 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:08:15.593 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:08:15.593 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:08:15.593 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:08:15.593 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:08:15.593 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:08:15.593 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:08:15.593 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:08:15.593 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:08:15.593 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.593 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:08:15.594 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.594 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:08:15.594 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:08:15.594 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:08:15.594 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:08:15.594 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.594 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:08:15.594 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:08:15.594 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:08:15.594 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.594 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.594 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:08:15.594 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:08:15.594 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:08:15.595 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:08:15.595 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:08:15.595 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.595 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:08:15.595 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:08:15.595 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:08:15.595 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:08:15.595 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:08:15.595 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.595 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.595 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:08:15.595 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:08:15.595 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:08:15.595 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:08:15.595 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.596 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:08:15.596 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:08:15.596 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:08:15.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.596 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.596 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:08:15.596 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:08:15.596 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:08:15.596 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:08:15.596 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:08:15.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.596 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:08:15.596 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:08:15.596 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:08:15.597 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:08:15.597 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:08:15.597 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.597 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.597 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:08:15.597 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:08:15.597 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:08:15.597 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:08:15.597 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.597 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:08:15.597 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:08:15.597 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:08:15.597 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.597 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.598 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:08:15.598 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:08:15.598 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:08:15.598 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:08:15.598 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:08:15.598 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.598 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:08:15.598 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:08:15.598 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:08:15.598 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:08:15.598 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:08:15.598 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:08:15.598 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:08:15.598 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.598 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:08:15.599 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:08:15.599 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.599 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:08:15.599 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:08:15.599 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:08:15.599 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:08:15.599 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:08:15.599 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:08:15.599 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.599 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:08:15.599 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:08:15.599 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:08:15.599 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:08:15.599 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:08:15.600 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:08:15.600 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.600 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:08:15.600 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:08:15.600 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:08:15.600 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:08:15.600 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:08:15.600 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.600 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:08:15.600 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:08:15.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1b542a640 10 librados: watch_flush enter 2024-09-16T05:08:15.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1b542a640 10 librados: watch_flush exit 2024-09-16T05:08:15.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1b542a640 20 client.4428.objecter shutdown clearing up homeless session... 2024-09-16T05:08:15.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1b542a640 10 client.4428.objecter successfully canceled tick 2024-09-16T05:08:15.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1b542a640 1 -- 172.21.15.114:0/667805939 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff1900377c0 msgr2=0x7ff190039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1b542a640 1 --2- 172.21.15.114:0/667805939 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff1900377c0 0x7ff190039c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff1a80ae000 tx=0x7ff1a80a7070 comp rx=0 tx=0).stop 2024-09-16T05:08:15.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1b542a640 10 monclient: shutdown 2024-09-16T05:08:15.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1b542a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:08:15.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1b542a640 1 -- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b00a5d00 msgr2=0x7ff1b013e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1b542a640 1 --2- 172.21.15.114:0/667805939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1b00a5d00 0x7ff1b013e980 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff19800c430 tx=0x7ff19800c8f0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1af7fe640 1 -- 172.21.15.114:0/667805939 reap_dead start 2024-09-16T05:08:15.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1af7fe640 1 -- 172.21.15.114:0/667805939 reap_dead start 2024-09-16T05:08:15.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1b542a640 1 -- 172.21.15.114:0/667805939 shutdown_connections 2024-09-16T05:08:15.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1b542a640 1 -- 172.21.15.114:0/667805939 >> 172.21.15.114:0/667805939 conn(0x7ff1b00a9c70 msgr2=0x7ff1b00a8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:08:15.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1b542a640 1 -- 172.21.15.114:0/667805939 shutdown_connections 2024-09-16T05:08:15.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1b542a640 1 -- 172.21.15.114:0/667805939 wait complete. 2024-09-16T05:08:15.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.566+0000 7ff1b542a640 1 librados: shutdown 2024-09-16T05:08:15.605 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:08:15.605 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T05:08:15.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:08:15.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.714+0000 7fd365c16640 10 monclient: get_monmap_and_config 2024-09-16T05:08:15.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.714+0000 7fd365c16640 10 monclient: build_initial_monmap 2024-09-16T05:08:15.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.714+0000 7fd365c16640 10 monclient: monmap: 2024-09-16T05:08:15.717 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:08:15.717 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.717 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:08:15.720231+0000 2024-09-16T05:08:15.717 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:08:15.720231+0000 2024-09-16T05:08:15.717 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:08:15.717 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.717 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:08:15.717 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:08:15.717 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:08:15.717 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:08:15.717 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:08:15.718 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 1 Processor -- start 2024-09-16T05:08:15.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 1 -- start start 2024-09-16T05:08:15.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 10 monclient: init 2024-09-16T05:08:15.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 10 monclient: _reopen_session rank -1 2024-09-16T05:08:15.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T05:08:15.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3600a0ec0 addr v1:172.21.15.114:6789/0 2024-09-16T05:08:15.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd36009e1d0 0x7fd36009e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 10 monclient(hunting): picked mon.noname-c con 0x7fd36009e1d0 addr v2:172.21.15.149:3300/0 2024-09-16T05:08:15.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd360139640 addr v1:172.21.15.149:6789/0 2024-09-16T05:08:15.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd36009a760 con 0x7fd3600a0ec0 2024-09-16T05:08:15.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3600a1870 con 0x7fd360139640 2024-09-16T05:08:15.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fd3600994b0 con 0x7fd36009e1d0 2024-09-16T05:08:15.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 10 monclient(hunting): _renew_subs 2024-09-16T05:08:15.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 10 monclient(hunting): authenticate will time out at 980.349420s 2024-09-16T05:08:15.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd3600a0ec0 0x7fd36009dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36018/0 (socket says 172.21.15.114:36018) 2024-09-16T05:08:15.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 1 -- 172.21.15.114:0/623367276 learned_addr learned my addr 172.21.15.114:0/623367276 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:08:15.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35effd640 1 --2- 172.21.15.114:0/623367276 >> v2:172.21.15.149:3300/0 conn(0x7fd36009e1d0 0x7fd36009e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35effd640 10 monclient(hunting): get_auth_request con 0x7fd36009e1d0 auth_method 0 2024-09-16T05:08:15.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:15.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 -- 172.21.15.114:0/623367276 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2216273043 0 0) 0x7fd36009a760 con 0x7fd3600a0ec0 2024-09-16T05:08:15.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:08:15.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:15.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient(hunting): my global_id is 4437 2024-09-16T05:08:15.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 -- 172.21.15.114:0/623367276 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd348003530 con 0x7fd3600a0ec0 2024-09-16T05:08:15.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:15.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:15.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:15.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 -- 172.21.15.114:0/623367276 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 276935666 0 0) 0x7fd348003530 con 0x7fd3600a0ec0 2024-09-16T05:08:15.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 -- 172.21.15.114:0/623367276 >> v2:172.21.15.149:3300/0 conn(0x7fd36009e1d0 msgr2=0x7fd36009e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:08:15.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 --2- 172.21.15.114:0/623367276 >> v2:172.21.15.149:3300/0 conn(0x7fd36009e1d0 0x7fd36009e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 -- 172.21.15.114:0/623367276 >> v1:172.21.15.149:6789/0 conn(0x7fd360139640 legacy=0x7fd36013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:08:15.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:08:15.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:08:15.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:08:15.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 -- 172.21.15.114:0/623367276 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd360099790 con 0x7fd3600a0ec0 2024-09-16T05:08:15.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:08:15.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:08:15.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:08:15.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 5 monclient: authenticate success, global_id 4437 2024-09-16T05:08:15.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 -- 172.21.15.114:0/623367276 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7fd3580016a0 con 0x7fd3600a0ec0 2024-09-16T05:08:15.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:08:15.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:08:15.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient: dump: 2024-09-16T05:08:15.734 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.734 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.734 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.734 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.734 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.734 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.734 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.735 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.735 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.735 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:08:15.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient: _reopen_session rank -1 2024-09-16T05:08:15.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 -- 172.21.15.114:0/623367276 >> v1:172.21.15.114:6789/0 conn(0x7fd3600a0ec0 legacy=0x7fd36009dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:08:15.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 --2- 172.21.15.114:0/623367276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3480039c0 0x7fd348003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient(hunting): picked mon.a con 0x7fd3480039c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:08:15.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 --2- 172.21.15.114:0/623367276 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd348004810 0x7fd348004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient(hunting): picked mon.b con 0x7fd348004810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:08:15.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 --2- 172.21.15.114:0/623367276 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3480051d0 0x7fd34800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient(hunting): picked mon.c con 0x7fd3480051d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:08:15.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 -- 172.21.15.114:0/623367276 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd3600994b0 con 0x7fd3480039c0 2024-09-16T05:08:15.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 -- 172.21.15.114:0/623367276 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fd34800c170 con 0x7fd348004810 2024-09-16T05:08:15.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 1 -- 172.21.15.114:0/623367276 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd34800c370 con 0x7fd3480051d0 2024-09-16T05:08:15.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient(hunting): _renew_subs 2024-09-16T05:08:15.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:08:15.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd365c16640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:08:15.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35ffff640 1 -- 172.21.15.114:0/623367276 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3480051d0 msgr2=0x7fd34800b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:08:15.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35ffff640 1 --2- 172.21.15.114:0/623367276 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3480051d0 0x7fd34800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:08:15.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 1 --2- 172.21.15.114:0/623367276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3480039c0 0x7fd348003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 10 monclient(hunting): get_auth_request con 0x7fd3480039c0 auth_method 0 2024-09-16T05:08:15.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:08:15.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:15.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:15.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:08:15.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35effd640 1 --2- 172.21.15.114:0/623367276 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd348004810 0x7fd348004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35effd640 10 monclient(hunting): get_auth_request con 0x7fd348004810 auth_method 0 2024-09-16T05:08:15.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:08:15.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 10 monclient(hunting): handle_auth_done global_id 4437 payload 293 2024-09-16T05:08:15.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 1 -- 172.21.15.114:0/623367276 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3480051d0 msgr2=0x7fd34800b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:08:15.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 1 --2- 172.21.15.114:0/623367276 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3480051d0 0x7fd34800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 1 -- 172.21.15.114:0/623367276 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd348004810 msgr2=0x7fd348004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:08:15.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 1 --2- 172.21.15.114:0/623367276 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd348004810 0x7fd348004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:08:15.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 1 monclient: found mon.a 2024-09-16T05:08:15.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:08:15.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:08:15.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 1 -- 172.21.15.114:0/623367276 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd348003530 con 0x7fd3480039c0 2024-09-16T05:08:15.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:08:15.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:08:15.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:08:15.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.718+0000 7fd35ffff640 1 -- 172.21.15.114:0/623367276 reap_dead start 2024-09-16T05:08:15.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35f7fe640 1 --2- 172.21.15.114:0/623367276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3480039c0 0x7fd348003de0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd35800c210 tx=0x7fd35801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a287d07a7b153ed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:15.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35e7fc640 1 -- 172.21.15.114:0/623367276 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd358007d40 con 0x7fd3480039c0 2024-09-16T05:08:15.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:08:15.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35e7fc640 10 monclient: dump: 2024-09-16T05:08:15.740 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.740 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.740 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.740 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.740 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.740 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.740 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.740 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.740 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.740 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35e7fc640 1 -- 172.21.15.114:0/623367276 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3580a9070 con 0x7fd3480039c0 2024-09-16T05:08:15.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:08:15.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35e7fc640 1 -- 172.21.15.114:0/623367276 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd358005d00 con 0x7fd3480039c0 2024-09-16T05:08:15.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient: get_monmap_and_config success 2024-09-16T05:08:15.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 4 set_mon_vals no callback set 2024-09-16T05:08:15.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:08:15.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35e7fc640 10 monclient: dump: 2024-09-16T05:08:15.741 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.741 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.741 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.741 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.741 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.742 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.742 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.742 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.742 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.742 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient: shutdown 2024-09-16T05:08:15.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:08:15.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 -- 172.21.15.114:0/623367276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3480039c0 msgr2=0x7fd348003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 --2- 172.21.15.114:0/623367276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3480039c0 0x7fd348003de0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd35800c210 tx=0x7fd35801cc40 comp rx=0 tx=0).stop 2024-09-16T05:08:15.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 -- 172.21.15.114:0/623367276 shutdown_connections 2024-09-16T05:08:15.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 --2- 172.21.15.114:0/623367276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3480039c0 0x7fd348003de0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 -- 172.21.15.114:0/623367276 >> 172.21.15.114:0/623367276 conn(0x7fd3600a1fb0 msgr2=0x7fd3600a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:08:15.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 -- 172.21.15.114:0/623367276 shutdown_connections 2024-09-16T05:08:15.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 -- 172.21.15.114:0/623367276 wait complete. 2024-09-16T05:08:15.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient: build_initial_monmap 2024-09-16T05:08:15.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient: monmap: 2024-09-16T05:08:15.743 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:08:15.743 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:08:15.743 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:08:15.743 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:08:15.743 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:08:15.743 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.743 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:08:15.744 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:08:15.744 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:08:15.744 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 librados: starting msgr at 2024-09-16T05:08:15.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 librados: starting objecter 2024-09-16T05:08:15.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 Processor -- start 2024-09-16T05:08:15.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 -- start start 2024-09-16T05:08:15.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 librados: setting wanted keys 2024-09-16T05:08:15.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 librados: calling monclient init 2024-09-16T05:08:15.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient: init 2024-09-16T05:08:15.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient: _reopen_session rank -1 2024-09-16T05:08:15.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:08:15.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3480039c0 0x7fd3601cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient(hunting): picked mon.noname-c con 0x7fd3480039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:08:15.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd360073240 0x7fd3601cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient(hunting): picked mon.noname-a con 0x7fd360073240 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:08:15.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd360071050 0x7fd360071500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient(hunting): picked mon.noname-b con 0x7fd360071050 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:08:15.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd36009d340 con 0x7fd360073240 2024-09-16T05:08:15.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fd36009d1c0 con 0x7fd360071050 2024-09-16T05:08:15.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd36009d4c0 con 0x7fd3480039c0 2024-09-16T05:08:15.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient(hunting): _renew_subs 2024-09-16T05:08:15.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient(hunting): authenticate will time out at 980.355025s 2024-09-16T05:08:15.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35f7fe640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3480039c0 msgr2=0x7fd3601cfaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:08:15.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35f7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3480039c0 0x7fd3601cfaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:08:15.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd360073240 0x7fd3601cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd360073240 0x7fd3601cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33864/0 (socket says 172.21.15.114:33864) 2024-09-16T05:08:15.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 1 -- 172.21.15.114:0/1443570475 learned_addr learned my addr 172.21.15.114:0/1443570475 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:08:15.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 10 monclient(hunting): get_auth_request con 0x7fd360073240 auth_method 0 2024-09-16T05:08:15.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:15.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:15.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:15.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35ffff640 1 --2- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd360071050 0x7fd360071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:15.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35ffff640 10 monclient(hunting): get_auth_request con 0x7fd360071050 auth_method 0 2024-09-16T05:08:15.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:15.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:15.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:15.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:15.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 10 monclient(hunting): handle_auth_done global_id 4440 payload 824 2024-09-16T05:08:15.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 1 -- 172.21.15.114:0/1443570475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3480039c0 msgr2=0x7fd3601cfaa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:08:15.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 1 --2- 172.21.15.114:0/1443570475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3480039c0 0x7fd3601cfaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 1 -- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd360071050 msgr2=0x7fd360071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 1 --2- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd360071050 0x7fd360071500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 10 monclient: _finish_hunting 0 2024-09-16T05:08:15.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 1 monclient: found mon.noname-a 2024-09-16T05:08:15.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:08:15.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:08:15.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 1 -- 172.21.15.114:0/1443570475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd360092250 con 0x7fd360073240 2024-09-16T05:08:15.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35effd640 1 --2- 172.21.15.114:0/1443570475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd360073240 0x7fd3601cffe0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd3540114e0 tx=0x7fd3540119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:15.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd347fff640 10 client.?.objecter ms_handle_connect 0x7fd360073240 2024-09-16T05:08:15.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd347fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:08:15.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd347fff640 1 -- 172.21.15.114:0/1443570475 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd354019070 con 0x7fd360073240 2024-09-16T05:08:15.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd347fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd347fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:08:15.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd347fff640 10 monclient: dump: 2024-09-16T05:08:15.749 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.749 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.749 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.749 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.749 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.750 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.750 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.750 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.750 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.750 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd347fff640 10 monclient: _finish_auth 0 2024-09-16T05:08:15.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd347fff640 10 monclient: _check_auth_tickets 2024-09-16T05:08:15.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd347fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:08:15.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd347fff640 1 -- 172.21.15.114:0/1443570475 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd354005db0 con 0x7fd360073240 2024-09-16T05:08:15.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd347fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:08:15.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd347fff640 1 -- 172.21.15.114:0/1443570475 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd354006270 con 0x7fd360073240 2024-09-16T05:08:15.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 5 monclient: authenticate success, global_id 4440 2024-09-16T05:08:15.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient: _renew_subs 2024-09-16T05:08:15.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:08:15.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 -- 172.21.15.114:0/1443570475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd360092530 con 0x7fd360073240 2024-09-16T05:08:15.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd347fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd347fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:08:15.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd347fff640 10 monclient: dump: 2024-09-16T05:08:15.751 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.751 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.751 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.751 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.751 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.751 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.751 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.751 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.751 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.751 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd35dffb640 4 set_mon_vals no callback set 2024-09-16T05:08:15.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:08:15.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient: _renew_subs 2024-09-16T05:08:15.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:08:15.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.722+0000 7fd365c16640 1 -- 172.21.15.114:0/1443570475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd36009c6a0 con 0x7fd360073240 2024-09-16T05:08:15.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd365c16640 1 librados: init done 2024-09-16T05:08:15.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd365c16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:08:15.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd365c16640 1 --2- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd360071a40 0x7fd360071e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd365c16640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd365c16640 1 -- 172.21.15.114:0/1443570475 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fd3600954a0 con 0x7fd360071a40 2024-09-16T05:08:15.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 1 -- 172.21.15.114:0/1443570475 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd35401c070 con 0x7fd360073240 2024-09-16T05:08:15.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd35f7fe640 1 --2- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd360071a40 0x7fd360071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 1 --2- 172.21.15.114:0/1443570475 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd34c0377f0 0x7fd34c039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd35f7fe640 10 monclient: get_auth_request con 0x7fd360071a40 auth_method 0 2024-09-16T05:08:15.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd35f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd35f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:15.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 1 -- 172.21.15.114:0/1443570475 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7fd354048a40 con 0x7fd360073240 2024-09-16T05:08:15.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 10 client.4440.objecter ms_dispatch 0x7fd360000ba0 osd_map(19..19 src has 1..19) 2024-09-16T05:08:15.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 3 client.4440.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T05:08:15.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 3 client.4440.objecter handle_osd_map decoding full epoch 19 2024-09-16T05:08:15.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd35f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:15.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd35f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:15.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd35f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:15.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 20 client.4440.objecter dump_active .. 0 homeless 2024-09-16T05:08:15.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd35ffff640 1 --2- 172.21.15.114:0/1443570475 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd34c0377f0 0x7fd34c039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd35ffff640 10 monclient: get_auth_request con 0x7fd34c0377f0 auth_method 0 2024-09-16T05:08:15.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd35f7fe640 10 monclient(hunting): handle_auth_done global_id 4405 payload 824 2024-09-16T05:08:15.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd35f7fe640 1 -- 172.21.15.114:0/1443570475 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd36009d850 con 0x7fd360071a40 2024-09-16T05:08:15.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 10 client.4440.objecter ms_handle_connect 0x7fd360071a40 2024-09-16T05:08:15.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 10 client.4440.objecter resend_mon_ops 2024-09-16T05:08:15.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd35f7fe640 1 --2- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd360071a40 0x7fd360071e20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd36009a220 tx=0x7fd348005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:15.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 10 client.4440.objecter ms_handle_connect 0x7fd34c0377f0 2024-09-16T05:08:15.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd35ffff640 1 --2- 172.21.15.114:0/1443570475 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd34c0377f0 0x7fd34c039cb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd36009a190 tx=0x7fd350009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:15.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 1 -- 172.21.15.114:0/1443570475 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd358005d00 con 0x7fd360071a40 2024-09-16T05:08:15.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:08:15.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 10 monclient: dump: 2024-09-16T05:08:15.754 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.754 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.754 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.755 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.755 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.755 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.755 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.755 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.755 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.755 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 1 -- 172.21.15.114:0/1443570475 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd3580a9030 con 0x7fd360071a40 2024-09-16T05:08:15.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:08:15.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:08:15.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 1 -- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd360071a40 msgr2=0x7fd360071e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 1 --2- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd360071a40 0x7fd360071e20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd36009a220 tx=0x7fd348005840 comp rx=0 tx=0).stop 2024-09-16T05:08:15.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 1 -- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd360071a40 msgr2=0x7fd360071e20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:08:15.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.726+0000 7fd347fff640 1 --2- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd360071a40 0x7fd360071e20 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd36009a220 tx=0x7fd348005840 comp rx=0 tx=0).stop 2024-09-16T05:08:15.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:08:15.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 1 --2- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd36006d990 0x7fd36006dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:15.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:15.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 1 -- 172.21.15.114:0/1443570475 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fd3600686f0 con 0x7fd36006d990 2024-09-16T05:08:15.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd35f7fe640 1 --2- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd36006d990 0x7fd36006dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:15.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd35f7fe640 10 monclient: get_auth_request con 0x7fd36006d990 auth_method 0 2024-09-16T05:08:15.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd35f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:15.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd35f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:15.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:15.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd35f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:15.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd35f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:15.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd35f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:15.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd35f7fe640 10 monclient(hunting): handle_auth_done global_id 4408 payload 824 2024-09-16T05:08:15.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd35f7fe640 1 -- 172.21.15.114:0/1443570475 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd360072360 con 0x7fd36006d990 2024-09-16T05:08:15.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd35f7fe640 1 --2- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd36006d990 0x7fd36006dda0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd35801ca70 tx=0x7fd358007900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:15.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd347fff640 10 client.4440.objecter ms_handle_connect 0x7fd36006d990 2024-09-16T05:08:15.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd347fff640 10 client.4440.objecter resend_mon_ops 2024-09-16T05:08:15.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd347fff640 1 -- 172.21.15.114:0/1443570475 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3580072c0 con 0x7fd36006d990 2024-09-16T05:08:15.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd347fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:15.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd347fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:08:15.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd347fff640 10 monclient: dump: 2024-09-16T05:08:15.759 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:08:15.759 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:15.759 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.759 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:15.759 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:08:15.759 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:08:15.759 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:15.760 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:15.760 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:15.760 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:08:15.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd347fff640 1 -- 172.21.15.114:0/1443570475 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7fd3580c1070 con 0x7fd36006d990 2024-09-16T05:08:15.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd347fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:08:15.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd347fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:08:15.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd347fff640 1 -- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd36006d990 msgr2=0x7fd36006dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd347fff640 1 --2- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd36006d990 0x7fd36006dda0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd35801ca70 tx=0x7fd358007900 comp rx=0 tx=0).stop 2024-09-16T05:08:15.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd347fff640 1 -- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd36006d990 msgr2=0x7fd36006dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:08:15.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd347fff640 1 --2- 172.21.15.114:0/1443570475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd36006d990 0x7fd36006dda0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.761 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:08:15.761 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:08:15.761 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:08:15.761 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:08:15.761 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 10, 2024-09-16T05:08:15.761 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 53679, 2024-09-16T05:08:15.761 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:08:15.761 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:08:15.761 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-16T05:08:15.761 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:08:15.762 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:08:15.762 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:08:15.762 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:08:15.762 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:08:15.762 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:08:15.762 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:08:15.762 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:08:15.762 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:08:15.763 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:08:15.763 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:08:15.763 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:08:15.763 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:08:15.763 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:08:15.763 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:08:15.763 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:08:15.763 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:08:15.763 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:08:15.763 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:08:15.764 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:08:15.764 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:08:15.764 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:08:15.764 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:08:15.764 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:08:15.764 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:08:15.764 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:08:15.764 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:08:15.764 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:08:15.764 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:08:15.764 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:08:15.764 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:08:15.765 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.765 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:08:15.765 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:08:15.765 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:08:15.765 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:08:15.765 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:08:15.765 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:08:15.765 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:08:15.765 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:08:15.765 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:08:15.765 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:08:15.766 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:08:15.766 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:08:15.766 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:08:15.766 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:08:15.766 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:08:15.766 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:08:15.766 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:08:15.766 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:08:15.766 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:08:15.766 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:08:15.766 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:08:15.767 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:08:15.767 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:08:15.767 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:08:15.767 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:08:15.767 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:08:15.767 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:08:15.767 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:08:15.767 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:08:15.767 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:08:15.767 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.767 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:08:15.768 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.768 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:08:15.768 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:08:15.768 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:08:15.768 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:08:15.768 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.768 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:08:15.768 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:08:15.768 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:08:15.768 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.769 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.769 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:08:15.769 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:08:15.769 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:08:15.769 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:08:15.769 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:08:15.769 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.769 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:08:15.769 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:08:15.769 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:08:15.769 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:08:15.769 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:08:15.770 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.770 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.770 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:08:15.770 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:08:15.770 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:08:15.770 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:08:15.770 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.770 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:08:15.770 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:08:15.770 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:08:15.770 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.771 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.771 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:08:15.771 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:08:15.771 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:08:15.771 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:08:15.771 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:08:15.771 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.771 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:08:15.771 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:08:15.771 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:08:15.771 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:08:15.772 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:08:15.772 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.772 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.772 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:08:15.772 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:08:15.772 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:08:15.772 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:08:15.772 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.772 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:08:15.773 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:08:15.773 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:08:15.773 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.773 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.773 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:08:15.773 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:08:15.773 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:08:15.773 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:08:15.773 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:08:15.773 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.773 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:08:15.774 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:08:15.774 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:08:15.774 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:08:15.774 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:08:15.774 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:08:15.774 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:08:15.774 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.774 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:08:15.774 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:08:15.774 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.774 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:08:15.774 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:08:15.775 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:08:15.775 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:08:15.775 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:08:15.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:08:15.775 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.775 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:08:15.775 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:08:15.775 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:08:15.775 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:08:15.775 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:08:15.775 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:08:15.775 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.776 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:08:15.776 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:08:15.776 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:08:15.776 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:08:15.776 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:08:15.776 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:08:15.776 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:08:15.776 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:08:15.776 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:08:15.776 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:08:15.777 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:08:15.777 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:08:15.777 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:08:15.777 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:08:15.777 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:08:15.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 10 librados: watch_flush enter 2024-09-16T05:08:15.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 10 librados: watch_flush exit 2024-09-16T05:08:15.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 20 client.4440.objecter shutdown clearing up homeless session... 2024-09-16T05:08:15.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 10 client.4440.objecter successfully canceled tick 2024-09-16T05:08:15.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 1 -- 172.21.15.114:0/1443570475 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd34c0377f0 msgr2=0x7fd34c039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 1 --2- 172.21.15.114:0/1443570475 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd34c0377f0 0x7fd34c039cb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd36009a190 tx=0x7fd350009040 comp rx=0 tx=0).stop 2024-09-16T05:08:15.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 10 monclient: shutdown 2024-09-16T05:08:15.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:08:15.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 1 -- 172.21.15.114:0/1443570475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd360073240 msgr2=0x7fd3601cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:15.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 1 --2- 172.21.15.114:0/1443570475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd360073240 0x7fd3601cffe0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd3540114e0 tx=0x7fd3540119a0 comp rx=0 tx=0).stop 2024-09-16T05:08:15.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd35ffff640 1 -- 172.21.15.114:0/1443570475 reap_dead start 2024-09-16T05:08:15.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd35ffff640 1 -- 172.21.15.114:0/1443570475 reap_dead start 2024-09-16T05:08:15.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 1 -- 172.21.15.114:0/1443570475 shutdown_connections 2024-09-16T05:08:15.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 1 -- 172.21.15.114:0/1443570475 >> 172.21.15.114:0/1443570475 conn(0x7fd3600a1fb0 msgr2=0x7fd360097a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:08:15.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 1 -- 172.21.15.114:0/1443570475 shutdown_connections 2024-09-16T05:08:15.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 1 -- 172.21.15.114:0/1443570475 wait complete. 2024-09-16T05:08:15.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:08:15.734+0000 7fd365c16640 1 librados: shutdown 2024-09-16T05:08:15.780 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:08:15.780 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:08:48.164 INFO:tasks.workunit.client.0.smithi149.stderr:Note: switching to 'ad35db06e8924912f01e32b35576297c0f1352b4'. 2024-09-16T05:08:48.164 INFO:tasks.workunit.client.0.smithi149.stderr: 2024-09-16T05:08:48.164 INFO:tasks.workunit.client.0.smithi149.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T05:08:48.164 INFO:tasks.workunit.client.0.smithi149.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T05:08:48.164 INFO:tasks.workunit.client.0.smithi149.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T05:08:48.165 INFO:tasks.workunit.client.0.smithi149.stderr: 2024-09-16T05:08:48.165 INFO:tasks.workunit.client.0.smithi149.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T05:08:48.165 INFO:tasks.workunit.client.0.smithi149.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T05:08:48.165 INFO:tasks.workunit.client.0.smithi149.stderr: 2024-09-16T05:08:48.165 INFO:tasks.workunit.client.0.smithi149.stderr: git switch -c 2024-09-16T05:08:48.165 INFO:tasks.workunit.client.0.smithi149.stderr: 2024-09-16T05:08:48.165 INFO:tasks.workunit.client.0.smithi149.stderr:Or undo this operation with: 2024-09-16T05:08:48.165 INFO:tasks.workunit.client.0.smithi149.stderr: 2024-09-16T05:08:48.165 INFO:tasks.workunit.client.0.smithi149.stderr: git switch - 2024-09-16T05:08:48.165 INFO:tasks.workunit.client.0.smithi149.stderr: 2024-09-16T05:08:48.165 INFO:tasks.workunit.client.0.smithi149.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T05:08:48.165 INFO:tasks.workunit.client.0.smithi149.stderr: 2024-09-16T05:08:48.165 INFO:tasks.workunit.client.0.smithi149.stderr:HEAD is now at ad35db06e89 Merge pull request #59794 from zdover23/wip-doc-2024-09-15-backport-59613-to-squid 2024-09-16T05:08:48.170 DEBUG:teuthology.orchestra.run.smithi149:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T05:08:48.175 INFO:tasks.workunit.client.0.smithi149.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T05:08:48.179 INFO:tasks.workunit.client.0.smithi149.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T05:08:48.179 INFO:tasks.workunit.client.0.smithi149.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T05:08:48.976 INFO:tasks.workunit.client.0.smithi149.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T05:08:49.062 INFO:tasks.workunit.client.0.smithi149.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T05:08:49.124 INFO:tasks.workunit.client.0.smithi149.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T05:08:49.124 INFO:tasks.workunit.client.0.smithi149.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T05:08:49.124 INFO:tasks.workunit.client.0.smithi149.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T05:08:49.168 INFO:tasks.workunit.client.0.smithi149.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T05:08:49.171 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-09-16T05:08:49.171 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T05:08:49.218 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-16T05:08:49.219 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-16T05:08:49.220 DEBUG:teuthology.orchestra.run.smithi149:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-16T05:08:49.270 INFO:tasks.workunit.client.0.smithi149.stderr:+ parallel=1 2024-09-16T05:08:49.271 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' '' = --serial ']' 2024-09-16T05:08:49.271 INFO:tasks.workunit.client.0.smithi149.stderr:+ crimson=0 2024-09-16T05:08:49.271 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' '' = --crimson ']' 2024-09-16T05:08:49.271 INFO:tasks.workunit.client.0.smithi149.stderr:+ color= 2024-09-16T05:08:49.271 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -t 1 ']' 2024-09-16T05:08:49.271 INFO:tasks.workunit.client.0.smithi149.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-16T05:08:49.271 INFO:tasks.workunit.client.0.smithi149.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-16T05:08:49.271 INFO:tasks.workunit.client.0.smithi149.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-16T05:08:49.272 INFO:tasks.workunit.client.0.smithi149.stderr:+ declare -A pids 2024-09-16T05:08:49.272 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.272 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.272 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_aio 2024-09-16T05:08:49.272 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_aio' 2024-09-16T05:08:49.273 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_aio 2024-09-16T05:08:49.273 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.275 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_aio 2024-09-16T05:08:49.276 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29440 2024-09-16T05:08:49.276 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_aio on pid 29440' 2024-09-16T05:08:49.276 INFO:tasks.workunit.client.0.smithi149.stdout:test api_aio on pid 29440 2024-09-16T05:08:49.277 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29440 2024-09-16T05:08:49.277 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.277 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.277 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-16T05:08:49.277 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_aio_pp 2024-09-16T05:08:49.277 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_aio_pp' 2024-09-16T05:08:49.277 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_aio_pp 2024-09-16T05:08:49.278 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.279 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.279 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.279 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.279 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.279 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-09-16T05:08:49.279 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-16T05:08:49.279 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_aio: /' 2024-09-16T05:08:49.279 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_aio_pp 2024-09-16T05:08:49.279 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29448 2024-09-16T05:08:49.280 INFO:tasks.workunit.client.0.smithi149.stdout:test api_aio_pp on pid 29448 2024-09-16T05:08:49.281 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_aio_pp on pid 29448' 2024-09-16T05:08:49.281 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29448 2024-09-16T05:08:49.281 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.281 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.281 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-16T05:08:49.281 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_io 2024-09-16T05:08:49.281 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_io' 2024-09-16T05:08:49.282 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_io 2024-09-16T05:08:49.282 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.282 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.282 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.282 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.282 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.283 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-09-16T05:08:49.283 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-16T05:08:49.283 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-16T05:08:49.283 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_io 2024-09-16T05:08:49.283 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29456 2024-09-16T05:08:49.284 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_io on pid 29456' 2024-09-16T05:08:49.284 INFO:tasks.workunit.client.0.smithi149.stdout:test api_io on pid 29456 2024-09-16T05:08:49.286 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29456 2024-09-16T05:08:49.286 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.286 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.286 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-16T05:08:49.286 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_io_pp 2024-09-16T05:08:49.286 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_io_pp' 2024-09-16T05:08:49.286 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_io_pp 2024-09-16T05:08:49.286 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.286 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.286 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.286 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.286 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.287 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-09-16T05:08:49.287 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_io.log 2024-09-16T05:08:49.287 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_io: /' 2024-09-16T05:08:49.287 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_io_pp 2024-09-16T05:08:49.287 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29464 2024-09-16T05:08:49.287 INFO:tasks.workunit.client.0.smithi149.stdout:test api_io_pp on pid 29464 2024-09-16T05:08:49.288 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_io_pp on pid 29464' 2024-09-16T05:08:49.288 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29464 2024-09-16T05:08:49.288 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.289 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.289 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-16T05:08:49.289 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_asio 2024-09-16T05:08:49.289 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_asio' 2024-09-16T05:08:49.289 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.289 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.289 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.289 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.289 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_asio 2024-09-16T05:08:49.289 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.289 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-09-16T05:08:49.289 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-16T05:08:49.289 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-16T05:08:49.290 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_asio 2024-09-16T05:08:49.290 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29472 2024-09-16T05:08:49.290 INFO:tasks.workunit.client.0.smithi149.stdout:test api_asio on pid 29472 2024-09-16T05:08:49.292 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_asio on pid 29472' 2024-09-16T05:08:49.292 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29472 2024-09-16T05:08:49.292 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.292 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.292 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-16T05:08:49.293 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_list 2024-09-16T05:08:49.293 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_list' 2024-09-16T05:08:49.293 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.293 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.293 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.293 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.293 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_list 2024-09-16T05:08:49.293 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.293 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-09-16T05:08:49.293 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-16T05:08:49.293 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_asio: /' 2024-09-16T05:08:49.294 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_list 2024-09-16T05:08:49.294 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29480 2024-09-16T05:08:49.294 INFO:tasks.workunit.client.0.smithi149.stdout:test api_list on pid 29480 2024-09-16T05:08:49.295 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_list on pid 29480' 2024-09-16T05:08:49.295 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29480 2024-09-16T05:08:49.296 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.296 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.296 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-16T05:08:49.296 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_lock 2024-09-16T05:08:49.296 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_lock' 2024-09-16T05:08:49.296 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_lock 2024-09-16T05:08:49.296 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.296 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.296 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.296 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.296 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.296 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-09-16T05:08:49.297 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_list: /' 2024-09-16T05:08:49.297 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_list.log 2024-09-16T05:08:49.297 INFO:tasks.workunit.client.0.smithi149.stdout:test api_lock on pid 29488 2024-09-16T05:08:49.298 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_lock 2024-09-16T05:08:49.298 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29488 2024-09-16T05:08:49.298 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_lock on pid 29488' 2024-09-16T05:08:49.298 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29488 2024-09-16T05:08:49.298 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.298 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.298 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-16T05:08:49.299 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.299 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.299 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.299 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.300 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-09-16T05:08:49.300 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_lock_pp 2024-09-16T05:08:49.300 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_lock_pp' 2024-09-16T05:08:49.300 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_lock: /' 2024-09-16T05:08:49.302 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.304 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_lock_pp 2024-09-16T05:08:49.305 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_lock_pp 2024-09-16T05:08:49.305 INFO:tasks.workunit.client.0.smithi149.stdout:test api_lock_pp on pid 29496 2024-09-16T05:08:49.305 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29496 2024-09-16T05:08:49.305 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_lock_pp on pid 29496' 2024-09-16T05:08:49.305 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29496 2024-09-16T05:08:49.305 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.305 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.306 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-16T05:08:49.306 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.306 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.307 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.307 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.307 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-09-16T05:08:49.314 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_misc 2024-09-16T05:08:49.314 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_misc' 2024-09-16T05:08:49.317 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-16T05:08:49.320 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-16T05:08:49.320 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-16T05:08:49.322 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.323 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_misc 2024-09-16T05:08:49.324 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_misc 2024-09-16T05:08:49.326 INFO:tasks.workunit.client.0.smithi149.stdout:test api_misc on pid 29531 2024-09-16T05:08:49.327 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29531 2024-09-16T05:08:49.327 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_misc on pid 29531' 2024-09-16T05:08:49.327 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29531 2024-09-16T05:08:49.327 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.327 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.327 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-16T05:08:49.327 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_misc_pp 2024-09-16T05:08:49.327 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_misc_pp' 2024-09-16T05:08:49.327 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.327 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.327 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.327 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.328 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-16T05:08:49.329 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_misc_pp 2024-09-16T05:08:49.329 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_misc: /' 2024-09-16T05:08:49.329 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-09-16T05:08:49.329 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.332 INFO:tasks.workunit.client.0.smithi149.stdout:test api_misc_pp on pid 29544 2024-09-16T05:08:49.332 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_misc_pp 2024-09-16T05:08:49.332 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29544 2024-09-16T05:08:49.332 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_misc_pp on pid 29544' 2024-09-16T05:08:49.332 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29544 2024-09-16T05:08:49.333 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.333 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.334 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-16T05:08:49.334 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_tier_pp 2024-09-16T05:08:49.334 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_tier_pp' 2024-09-16T05:08:49.335 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_tier_pp 2024-09-16T05:08:49.336 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-16T05:08:49.336 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [----------] Global test environment set-up. 2024-09-16T05:08:49.337 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-16T05:08:49.337 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 10 monclient: get_monmap_and_config 2024-09-16T05:08:49.337 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 10 monclient: build_initial_monmap 2024-09-16T05:08:49.337 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 10 monclient: monmap: 2024-09-16T05:08:49.337 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: epoch 0 2024-09-16T05:08:49.337 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:08:49.337 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: last_changed 0.000000 2024-09-16T05:08:49.337 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: created 0.000000 2024-09-16T05:08:49.337 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-16T05:08:49.337 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: election_strategy: 1 2024-09-16T05:08:49.337 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:08:49.337 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:08:49.337 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:08:49.338 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.338 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 1 Processor -- start 2024-09-16T05:08:49.338 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 1 -- start start 2024-09-16T05:08:49.338 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 10 monclient: init 2024-09-16T05:08:49.338 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 10 monclient: _reopen_session rank -1 2024-09-16T05:08:49.338 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:08:49.338 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1a8d8f690 0x55c1a8d65eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:49.338 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 10 monclient(hunting): picked mon.noname-a con 0x55c1a8d8f690 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:08:49.338 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c1a8d96310 0x55c1a8e27e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:49.338 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 10 monclient(hunting): picked mon.noname-b con 0x55c1a8d96310 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:08:49.338 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1a8e283d0 0x55c1a8ebcb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:49.338 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 10 monclient(hunting): picked mon.noname-c con 0x55c1a8e283d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:08:49.338 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:49.338 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c1a8d8ed20 con 0x55c1a8d8f690 2024-09-16T05:08:49.339 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:49.339 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x55c1a8d8ea80 con 0x55c1a8d96310 2024-09-16T05:08:49.339 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:49.339 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x55c1a8d8d100 con 0x55c1a8e283d0 2024-09-16T05:08:49.339 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 10 monclient(hunting): _renew_subs 2024-09-16T05:08:49.339 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0513247880 10 monclient(hunting): authenticate will time out at 1005.402267s 2024-09-16T05:08:49.339 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1a8d8f690 0x55c1a8d65eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:49.339 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f051055a640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c1a8d96310 0x55c1a8e27e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:49.339 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1a8d8f690 0x55c1a8d65eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.149:60440/0 (socket says 172.21.15.149:60440) 2024-09-16T05:08:49.340 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f051055a640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c1a8d96310 0x55c1a8e27e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:38036/0 (socket says 172.21.15.149:38036) 2024-09-16T05:08:49.340 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 1 -- 172.21.15.149:0/2902970955 learned_addr learned my addr 172.21.15.149:0/2902970955 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-09-16T05:08:49.340 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 10 monclient(hunting): get_auth_request con 0x55c1a8d8f690 auth_method 0 2024-09-16T05:08:49.340 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:49.340 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:49.340 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:49.340 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f051155c640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1a8e283d0 msgr2=0x55c1a8ebcb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:08:49.340 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f051155c640 1 --2- 172.21.15.149:0/2902970955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1a8e283d0 0x55c1a8ebcb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:08:49.340 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f051055a640 10 monclient(hunting): get_auth_request con 0x55c1a8d96310 auth_method 0 2024-09-16T05:08:49.340 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f051055a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:49.340 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f051055a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:49.340 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f051055a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:49.340 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:49.341 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:49.341 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:49.341 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f051055a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:49.341 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f051055a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:49.341 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f051055a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:49.341 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 10 monclient(hunting): handle_auth_done global_id 4467 payload 274 2024-09-16T05:08:49.342 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.342 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.342 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.343 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.343 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-16T05:08:49.343 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 1 -- 172.21.15.149:0/2902970955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1a8e283d0 msgr2=0x55c1a8ebcb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:08:49.343 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 1 --2- 172.21.15.149:0/2902970955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1a8e283d0 0x55c1a8ebcb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:49.343 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 1 -- 172.21.15.149:0/2902970955 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c1a8d96310 msgr2=0x55c1a8e27e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:08:49.343 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 1 --2- 172.21.15.149:0/2902970955 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c1a8d96310 0x55c1a8e27e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:49.343 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 10 monclient: _finish_hunting 0 2024-09-16T05:08:49.343 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 1 monclient: found mon.noname-a 2024-09-16T05:08:49.343 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:08:49.343 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:08:49.344 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 1 -- 172.21.15.149:0/2902970955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1a8d66e60 con 0x55c1a8d8f690 2024-09-16T05:08:49.344 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f051055a640 1 --2- 172.21.15.149:0/2902970955 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c1a8d96310 0x55c1a8e27e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:08:49.344 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.332+0000 7f0510d5b640 1 --2- 172.21.15.149:0/2902970955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1a8d8f690 0x55c1a8d65eb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f050401fbf0 tx=0x7f0504003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae70ca5ca42a945c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:49.344 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f050f558640 1 -- 172.21.15.149:0/2902970955 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05040b3070 con 0x55c1a8d8f690 2024-09-16T05:08:49.345 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f050f558640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:49.345 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f050f558640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:08:49.345 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f050f558640 10 monclient: dump: 2024-09-16T05:08:49.345 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: epoch 1 2024-09-16T05:08:49.345 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:49.345 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:49.345 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:49.345 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: min_mon_release 19 (squid) 2024-09-16T05:08:49.345 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: election_strategy: 1 2024-09-16T05:08:49.345 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:49.346 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:49.346 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:49.346 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.346 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f050f558640 10 monclient: _finish_auth 0 2024-09-16T05:08:49.346 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f050f558640 10 monclient: _check_auth_tickets 2024-09-16T05:08:49.346 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f050f558640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:08:49.346 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f0513247880 5 monclient: authenticate success, global_id 4467 2024-09-16T05:08:49.346 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-16T05:08:49.347 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-09-16T05:08:49.347 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.347 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f0513247880 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:08:49.347 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f050f558640 1 -- 172.21.15.149:0/2902970955 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0504012280 con 0x55c1a8d8f690 2024-09-16T05:08:49.347 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f050f558640 10 monclient: handle_config config(0 keys) 2024-09-16T05:08:49.347 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f0513247880 10 monclient: get_monmap_and_config success 2024-09-16T05:08:49.348 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f0513247880 4 set_mon_vals no callback set 2024-09-16T05:08:49.348 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f0513247880 10 monclient: shutdown 2024-09-16T05:08:49.348 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f0513247880 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:08:49.348 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f0513247880 1 -- 172.21.15.149:0/2902970955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1a8d8f690 msgr2=0x55c1a8d65eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:49.348 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.336+0000 7f0513247880 1 --2- 172.21.15.149:0/2902970955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1a8d8f690 0x55c1a8d65eb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f050401fbf0 tx=0x7f0504003030 comp rx=0 tx=0).stop 2024-09-16T05:08:49.348 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 1 -- 172.21.15.149:0/2902970955 shutdown_connections 2024-09-16T05:08:49.348 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 1 --2- 172.21.15.149:0/2902970955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1a8e283d0 0x55c1a8ebcb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:49.348 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 1 --2- 172.21.15.149:0/2902970955 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c1a8d96310 0x55c1a8e27e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:49.348 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 1 --2- 172.21.15.149:0/2902970955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1a8d8f690 0x55c1a8d65eb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:49.348 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 1 -- 172.21.15.149:0/2902970955 >> 172.21.15.149:0/2902970955 conn(0x55c1a8d94100 msgr2=0x55c1a8d64530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:08:49.348 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 1 -- 172.21.15.149:0/2902970955 shutdown_connections 2024-09-16T05:08:49.348 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 1 -- 172.21.15.149:0/2902970955 wait complete. 2024-09-16T05:08:49.348 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 10 monclient: build_initial_monmap 2024-09-16T05:08:49.349 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 10 monclient: monmap: 2024-09-16T05:08:49.349 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: epoch 0 2024-09-16T05:08:49.349 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:08:49.349 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: last_changed 0.000000 2024-09-16T05:08:49.349 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: created 0.000000 2024-09-16T05:08:49.349 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-16T05:08:49.349 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: election_strategy: 1 2024-09-16T05:08:49.349 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:08:49.349 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:08:49.349 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:08:49.349 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.349 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 1 librados: starting msgr at 2024-09-16T05:08:49.349 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 1 librados: starting objecter 2024-09-16T05:08:49.349 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 1 Processor -- start 2024-09-16T05:08:49.349 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 1 -- start start 2024-09-16T05:08:49.350 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 1 librados: setting wanted keys 2024-09-16T05:08:49.350 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 1 librados: calling monclient init 2024-09-16T05:08:49.350 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.340+0000 7f0513247880 10 monclient: init 2024-09-16T05:08:49.350 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 monclient: _reopen_session rank -1 2024-09-16T05:08:49.350 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:08:49.350 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c1a8d8f690 0x55c1a8e27880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:49.350 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 monclient(hunting): picked mon.noname-b con 0x55c1a8d8f690 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:08:49.350 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1a8d96310 0x55c1a8e198c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:49.350 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 monclient(hunting): picked mon.noname-a con 0x55c1a8d96310 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:08:49.350 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1a8e283d0 0x55c1a8e19e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:49.350 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 monclient(hunting): picked mon.noname-c con 0x55c1a8e283d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:08:49.350 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:49.350 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c1a8d8f210 con 0x55c1a8d96310 2024-09-16T05:08:49.351 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:49.351 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x55c1a8d8fbf0 con 0x55c1a8d8f690 2024-09-16T05:08:49.351 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 monclient(hunting): start opening mon connection 2024-09-16T05:08:49.351 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x55c1a8d62790 con 0x55c1a8e283d0 2024-09-16T05:08:49.351 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 monclient(hunting): _renew_subs 2024-09-16T05:08:49.351 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 monclient(hunting): authenticate will time out at 1005.412390s 2024-09-16T05:08:49.351 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f051155c640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1a8e283d0 msgr2=0x55c1a8e19e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:08:49.351 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f051155c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1a8e283d0 0x55c1a8e19e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:08:49.351 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c1a8d8f690 0x55c1a8e27880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:49.351 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c1a8d8f690 0x55c1a8e27880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:38052/0 (socket says 172.21.15.149:38052) 2024-09-16T05:08:49.351 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f05test api_tier_pp on pid 29611 2024-09-16T05:08:49.352 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_tier_pp 2024-09-16T05:08:49.352 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29611 2024-09-16T05:08:49.352 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_tier_pp on pid 29611' 2024-09-16T05:08:49.352 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29611 2024-09-16T05:08:49.352 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.352 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.352 INFO:tasks.workunit.client.0.smithi149.stdout:10d5b640 1 -- 172.21.15.149:0/3191936048 learned_addr learned my addr 172.21.15.149:0/3191936048 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-09-16T05:08:49.353 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 10 monclient(hunting): get_auth_request con 0x55c1a8d8f690 auth_method 0 2024-09-16T05:08:49.353 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:49.353 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:49.353 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:49.353 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f051055a640 1 --2- 172.21.15.149:0/3191936048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1a8d96310 0x55c1a8e198c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:49.353 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f051055a640 10 monclient(hunting): get_auth_request con 0x55c1a8d96310 auth_method 0 2024-09-16T05:08:49.353 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f051055a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:08:49.353 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f051055a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:08:49.353 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f051055a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:08:49.353 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:08:49.353 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:08:49.353 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:08:49.353 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 10 monclient(hunting): handle_auth_done global_id 4429 payload 824 2024-09-16T05:08:49.353 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 1 -- 172.21.15.149:0/3191936048 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1a8e283d0 msgr2=0x55c1a8e19e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:08:49.354 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 1 --2- 172.21.15.149:0/3191936048 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1a8e283d0 0x55c1a8e19e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:49.354 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 1 -- 172.21.15.149:0/3191936048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1a8d96310 msgr2=0x55c1a8e198c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:49.354 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 1 --2- 172.21.15.149:0/3191936048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1a8d96310 0x55c1a8e198c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:49.354 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 10 monclient: _finish_hunting 0 2024-09-16T05:08:49.354 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 1 monclient: found mon.noname-b 2024-09-16T05:08:49.354 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:08:49.354 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.149:3300/0 2024-09-16T05:08:49.354 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 1 -- 172.21.15.149:0/3191936048 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1a8e1cbd0 con 0x55c1a8d8f690 2024-09-16T05:08:49.354 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0510d5b640 1 --2- 172.21.15.149:0/3191936048 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c1a8d8f690 0x55c1a8e27880 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f050401fbf0 tx=0x7f050401c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:49.354 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050e556640 10 client.?.objecter ms_handle_connect 0x55c1a8d8f690 2024-09-16T05:08:49.354 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050e556640 10 client.?.objecter resend_mon_ops 2024-09-16T05:08:49.354 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050e556640 1 -- 172.21.15.149:0/3191936048 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05040c5070 con 0x55c1a8d8f690 2024-09-16T05:08:49.354 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050e556640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:49.355 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050e556640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:08:49.355 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050e556640 10 monclient: dump: 2024-09-16T05:08:49.355 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: epoch 1 2024-09-16T05:08:49.355 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:49.355 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:49.355 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:49.355 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: min_mon_release 19 (squid) 2024-09-16T05:08:49.355 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: election_strategy: 1 2024-09-16T05:08:49.355 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:49.355 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:49.355 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:49.355 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.355 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050e556640 10 monclient: _finish_auth 0 2024-09-16T05:08:49.356 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050e556640 10 monclient: _check_auth_tickets 2024-09-16T05:08:49.356 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050e556640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:08:49.356 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 5 monclient: authenticate success, global_id 4429 2024-09-16T05:08:49.356 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 monclient: _renew_subs 2024-09-16T05:08:49.356 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:08:49.356 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c1a8e1ce70 con 0x55c1a8d8f690 2024-09-16T05:08:49.356 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:08:49.356 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 monclient: _renew_subs 2024-09-16T05:08:49.356 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:08:49.356 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1a8d654b0 con 0x55c1a8d8f690 2024-09-16T05:08:49.356 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 1 librados: init done 2024-09-16T05:08:49.356 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f0513247880 10 librados: wait_for_osdmap waiting 2024-09-16T05:08:49.356 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050e556640 1 -- 172.21.15.149:0/3191936048 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0504008e90 con 0x55c1a8d8f690 2024-09-16T05:08:49.356 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050e556640 10 monclient: handle_config config(0 keys) 2024-09-16T05:08:49.357 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050e556640 1 -- 172.21.15.149:0/3191936048 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05040b3040 con 0x55c1a8d8f690 2024-09-16T05:08:49.357 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050e556640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:08:49.357 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050e556640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:08:49.357 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050e556640 10 monclient: dump: 2024-09-16T05:08:49.358 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-09-16T05:08:49.358 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_pool 2024-09-16T05:08:49.358 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_pool' 2024-09-16T05:08:49.358 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.358 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.358 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_pool 2024-09-16T05:08:49.358 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.358 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.358 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-09-16T05:08:49.358 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.359 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-09-16T05:08:49.359 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: epoch 1 2024-09-16T05:08:49.359 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:08:49.359 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:49.359 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:08:49.359 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: min_mon_release 19 (squid) 2024-09-16T05:08:49.359 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: election_strategy: 1 2024-09-16T05:08:49.359 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:08:49.359 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:08:49.359 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:08:49.359 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.360 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.344+0000 7f050f558640 4 set_mon_vals no callback set 2024-09-16T05:08:49.360 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f050e556640 1 -- 172.21.15.149:0/3191936048 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f05040c8070 con 0x55c1a8d8f690 2024-09-16T05:08:49.360 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f050e556640 1 --2- 172.21.15.149:0/3191936048 >> v2:172.21.15.149:6812/1185531135 conn(0x7f04fc035970 0x7f04fc037e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:49.360 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f051055a640 1 --2- 172.21.15.149:0/3191936048 >> v2:172.21.15.149:6812/1185531135 conn(0x7f04fc035970 0x7f04fc037e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:49.360 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f050e556640 1 -- 172.21.15.149:0/3191936048 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f05040e2730 con 0x55c1a8d8f690 2024-09-16T05:08:49.360 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f050e556640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_map(19..19 src has 1..19) 2024-09-16T05:08:49.360 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f051055a640 10 monclient: get_auth_request con 0x7f04fc035970 auth_method 0 2024-09-16T05:08:49.360 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f050e556640 3 client.4429.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T05:08:49.360 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f050e556640 3 client.4429.objecter handle_osd_map decoding full epoch 19 2024-09-16T05:08:49.360 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f050e556640 20 client.4429.objecter dump_active .. 0 homeless 2024-09-16T05:08:49.360 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f0513247880 10 librados: wait_for_osdmap done waiting 2024-09-16T05:08:49.360 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f0513247880 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:08:49.360 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0504012280 con 0x55c1a8d8f690 2024-09-16T05:08:49.361 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f050e556640 1 -- 172.21.15.149:0/3191936048 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f05040e9060 con 0x55c1a8d8f690 2024-09-16T05:08:49.361 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f050e556640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T05:08:49.361 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f050ed57640 10 client.4429.objecter _get_latest_version latest 19, have it 2024-09-16T05:08:49.361 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f0513247880 10 client.4429.objecter create_pool name=ceph_test_rados_api_asio 2024-09-16T05:08:49.361 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f0513247880 10 client.4429.objecter pool_op_submit 1 2024-09-16T05:08:49.361 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f0513247880 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:08:49.361 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x55c1a8d1a0f0 con 0x55c1a8d8f690 2024-09-16T05:08:49.362 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_tier_pp: /' 2024-09-16T05:08:49.362 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_pool 2024-09-16T05:08:49.362 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29628 2024-09-16T05:08:49.362 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f051055test api_pool on pid 29628 2024-09-16T05:08:49.364 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_pool on pid 29628' 2024-09-16T05:08:49.364 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29628 2024-09-16T05:08:49.364 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.364 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.364 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-16T05:08:49.364 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.364 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.364 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.364 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.364 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-09-16T05:08:49.364 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-16T05:08:49.365 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_pool: /' 2024-09-16T05:08:49.365 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_snapshots 2024-09-16T05:08:49.365 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_snapshots' 2024-09-16T05:08:49.365 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.373 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_snapshots 2024-09-16T05:08:49.374 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_snapshots 2024-09-16T05:08:49.375 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29667 2024-09-16T05:08:49.375 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_snapshots on pid 29667' 2024-09-16T05:08:49.375 INFO:tasks.workunit.client.0.smithi149.stdout:test api_snapshots on pid 29667 2024-09-16T05:08:49.376 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29667 2024-09-16T05:08:49.376 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.376 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.377 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-16T05:08:49.377 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_snapshots_pp 2024-09-16T05:08:49.377 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_snapshots_pp' 2024-09-16T05:08:49.377 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_snapshots_pp 2024-09-16T05:08:49.377 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.377 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.377 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.377 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.377 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-09-16T05:08:49.377 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-16T05:08:49.377 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-16T05:08:49.378 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.379 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_snapshots_pp 2024-09-16T05:08:49.380 INFO:tasks.workunit.client.0.smithi149.stdout:test api_snapshots_pp on pid 29683 2024-09-16T05:08:49.380 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29683 2024-09-16T05:08:49.380 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_snapshots_pp on pid 29683' 2024-09-16T05:08:49.381 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29683 2024-09-16T05:08:49.381 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.381 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.381 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-16T05:08:49.381 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_stat 2024-09-16T05:08:49.381 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_stat' 2024-09-16T05:08:49.382 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.382 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.382 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.382 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.382 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-09-16T05:08:49.382 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-16T05:08:49.385 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-16T05:08:49.386 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_stat 2024-09-16T05:08:49.389 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.392 INFO:tasks.workunit.client.0.smithi149.stdout:test api_stat on pid 29706 2024-09-16T05:08:49.392 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_stat 2024-09-16T05:08:49.392 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29706 2024-09-16T05:08:49.392 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_stat on pid 29706' 2024-09-16T05:08:49.392 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29706 2024-09-16T05:08:49.392 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.393 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.393 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-16T05:08:49.393 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_stat_pp 2024-09-16T05:08:49.394 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_stat_pp' 2024-09-16T05:08:49.394 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.394 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.394 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.394 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.394 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.394 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-09-16T05:08:49.396 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_stat: /' 2024-09-16T05:08:49.399 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-16T05:08:49.400 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_stat_pp 2024-09-16T05:08:49.401 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_stat_pp 2024-09-16T05:08:49.401 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29728 2024-09-16T05:08:49.402 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_stat_pp on pid 29728' 2024-09-16T05:08:49.402 INFO:tasks.workunit.client.0.smithi149.stdout:test api_stat_pp on pid 29728 2024-09-16T05:08:49.403 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29728 2024-09-16T05:08:49.403 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.403 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.403 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-16T05:08:49.403 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_watch_notify 2024-09-16T05:08:49.403 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_watch_notify' 2024-09-16T05:08:49.403 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_watch_notify 2024-09-16T05:08:49.403 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.404 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.404 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.404 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.404 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.404 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-09-16T05:08:49.404 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-16T05:08:49.404 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-16T05:08:49.405 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_watch_notify 2024-09-16T05:08:49.405 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29739 2024-09-16T05:08:49.405 INFO:tasks.workunit.client.0.smithi149.stdout:test api_watch_notify on pid 29739 2024-09-16T05:08:49.407 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_watch_notify on pid 29739' 2024-09-16T05:08:49.407 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29739 2024-09-16T05:08:49.407 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.407 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.407 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-16T05:08:49.407 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_watch_notify_pp 2024-09-16T05:08:49.407 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_watch_notify_pp' 2024-09-16T05:08:49.407 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_watch_notify_pp 2024-09-16T05:08:49.407 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.407 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.408 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.408 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.408 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-09-16T05:08:49.408 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-16T05:08:49.408 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-16T05:08:49.413 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.416 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_watch_notify_pp 2024-09-16T05:08:49.416 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29766 2024-09-16T05:08:49.416 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_watch_notify_pp on pid 29766' 2024-09-16T05:08:49.417 INFO:tasks.workunit.client.0.smithi149.stdout:test api_watch_notify_pp on pid 29766 2024-09-16T05:08:49.418 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29766 2024-09-16T05:08:49.418 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.418 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.418 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-16T05:08:49.418 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_cmd 2024-09-16T05:08:49.418 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_cmd' 2024-09-16T05:08:49.418 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.418 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_cmd 2024-09-16T05:08:49.419 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.419 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.419 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.419 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.419 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-09-16T05:08:49.419 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-16T05:08:49.420 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_cmd 2024-09-16T05:08:49.420 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-16T05:08:49.420 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29774 2024-09-16T05:08:49.420 INFO:tasks.workunit.client.0.smithi149.stdout:test api_cmd on pid 29774 2024-09-16T05:08:49.422 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_cmd on pid 29774' 2024-09-16T05:08:49.422 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29774 2024-09-16T05:08:49.422 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.422 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.422 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-16T05:08:49.422 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.422 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.422 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.422 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.422 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_cmd_pp 2024-09-16T05:08:49.423 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-09-16T05:08:49.423 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_cmd_pp' 2024-09-16T05:08:49.423 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-16T05:08:49.423 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_cmd: /' 2024-09-16T05:08:49.423 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_cmd_pp 2024-09-16T05:08:49.423 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.426 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_cmd_pp 2024-09-16T05:08:49.426 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29785 2024-09-16T05:08:49.426 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_cmd_pp on pid 29785' 2024-09-16T05:08:49.427 INFO:tasks.workunit.client.0.smithi149.stdout:test api_cmd_pp on pid 29785 2024-09-16T05:08:49.427 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29785 2024-09-16T05:08:49.428 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.428 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.428 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-16T05:08:49.428 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.428 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.428 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.428 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.428 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-09-16T05:08:49.430 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-16T05:08:49.431 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-16T05:08:49.434 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_service 2024-09-16T05:08:49.434 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_service' 2024-09-16T05:08:49.434 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.437 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_service 2024-09-16T05:08:49.438 INFO:tasks.workunit.client.0.smithi149.stdout:test api_service on pid 29813 2024-09-16T05:08:49.439 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_service 2024-09-16T05:08:49.439 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29813 2024-09-16T05:08:49.439 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_service on pid 29813' 2024-09-16T05:08:49.439 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29813 2024-09-16T05:08:49.439 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.439 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.439 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-16T05:08:49.440 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.440 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.440 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.441 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.441 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-09-16T05:08:49.441 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_service.log 2024-09-16T05:08:49.441 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_service: /' 2024-09-16T05:08:49.441 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_service_pp 2024-09-16T05:08:49.442 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_service_pp' 2024-09-16T05:08:49.450 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_service_pp 2024-09-16T05:08:49.459 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.463 INFO:tasks.workunit.client.0.smithi149.stdout:test api_service_pp on pid 29901 2024-09-16T05:08:49.464 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_service_pp 2024-09-16T05:08:49.464 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29901 2024-09-16T05:08:49.464 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_service_pp on pid 29901' 2024-09-16T05:08:49.464 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29901 2024-09-16T05:08:49.464 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.464 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.465 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_c_write_operations 2024-09-16T05:08:49.465 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-16T05:08:49.465 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_c_write_operations' 2024-09-16T05:08:49.465 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.466 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.466 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.466 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.466 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-16T05:08:49.466 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-16T05:08:49.467 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-09-16T05:08:49.467 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_c_write_operations 2024-09-16T05:08:49.467 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.468 INFO:tasks.workunit.client.0.smithi149.stdout:test api_c_write_operations on pid 29916 2024-09-16T05:08:49.469 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_c_write_operations 2024-09-16T05:08:49.469 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29916 2024-09-16T05:08:49.470 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_c_write_operations on pid 29916' 2024-09-16T05:08:49.470 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29916 2024-09-16T05:08:49.470 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.470 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.470 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-16T05:08:49.470 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_c_read_operations 2024-09-16T05:08:49.470 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_c_read_operations' 2024-09-16T05:08:49.470 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_c_read_operations 2024-09-16T05:08:49.470 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.471 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.471 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.471 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.472 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.472 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-09-16T05:08:49.472 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-16T05:08:49.472 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-16T05:08:49.473 INFO:tasks.workunit.client.0.smithi149.stdout:test api_c_read_operations on pid 29932 2024-09-16T05:08:49.474 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_c_read_operations 2024-09-16T05:08:49.474 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29932 2024-09-16T05:08:49.474 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_c_read_operations on pid 29932' 2024-09-16T05:08:49.474 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29932 2024-09-16T05:08:49.474 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.474 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.474 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s api_cls_remote_reads 2024-09-16T05:08:49.474 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' api_cls_remote_reads' 2024-09-16T05:08:49.474 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-16T05:08:49.475 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo api_cls_remote_reads 2024-09-16T05:08:49.475 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.476 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=api_cls_remote_reads 2024-09-16T05:08:49.476 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29939 2024-09-16T05:08:49.476 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test api_cls_remote_reads on pid 29939' 2024-09-16T05:08:49.477 INFO:tasks.workunit.client.0.smithi149.stdout:test api_cls_remote_reads on pid 29939 2024-09-16T05:08:49.479 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29939 2024-09-16T05:08:49.479 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.479 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.479 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-09-16T05:08:49.479 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s list_parallel 2024-09-16T05:08:49.479 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' list_parallel' 2024-09-16T05:08:49.479 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo list_parallel 2024-09-16T05:08:49.479 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.479 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.479 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.480 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.480 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.480 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-09-16T05:08:49.480 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-09-16T05:08:49.480 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-09-16T05:08:49.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=list_parallel 2024-09-16T05:08:49.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29953 2024-09-16T05:08:49.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test list_parallel on pid 29953' 2024-09-16T05:08:49.482 INFO:tasks.workunit.client.0.smithi149.stdout:test list_parallel on pid 29953 2024-09-16T05:08:49.486 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29953 2024-09-16T05:08:49.486 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.486 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.487 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-16T05:08:49.487 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s open_pools_parallel 2024-09-16T05:08:49.487 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.487 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.487 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.488 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.488 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' open_pools_parallel' 2024-09-16T05:08:49.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-09-16T05:08:49.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-09-16T05:08:49.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-16T05:08:49.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-16T05:08:49.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-16T05:08:49.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ list_parallel: /' 2024-09-16T05:08:49.490 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.491 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo open_pools_parallel 2024-09-16T05:08:49.493 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=open_pools_parallel 2024-09-16T05:08:49.493 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29973 2024-09-16T05:08:49.493 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test open_pools_parallel on pid 29973' 2024-09-16T05:08:49.494 INFO:tasks.workunit.client.0.smithi149.stdout:test open_pools_parallel on pid 29973 2024-09-16T05:08:49.495 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29973 2024-09-16T05:08:49.495 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T05:08:49.495 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.496 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s delete_pools_parallel 2024-09-16T05:08:49.496 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' delete_pools_parallel' 2024-09-16T05:08:49.496 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.496 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo delete_pools_parallel 2024-09-16T05:08:49.496 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-16T05:08:49.497 INFO:tasks.workunit.client.0.smithi149.stdout:test delete_pools_parallel on pid 29981 2024-09-16T05:08:49.498 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=delete_pools_parallel 2024-09-16T05:08:49.498 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29981 2024-09-16T05:08:49.498 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test delete_pools_parallel on pid 29981' 2024-09-16T05:08:49.498 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=29981 2024-09-16T05:08:49.498 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T05:08:49.498 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.499 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.499 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.499 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.499 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.499 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-09-16T05:08:49.499 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s cls 2024-09-16T05:08:49.499 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' cls' 2024-09-16T05:08:49.504 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-16T05:08:49.505 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.505 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.505 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.505 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.506 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-16T05:08:49.507 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-09-16T05:08:49.509 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-16T05:08:49.512 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-16T05:08:49.513 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-16T05:08:49.522 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.525 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo cls 2024-09-16T05:08:49.527 INFO:tasks.workunit.client.0.smithi149.stdout:test cls on pid 30037 2024-09-16T05:08:49.527 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=cls 2024-09-16T05:08:49.527 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30037 2024-09-16T05:08:49.527 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test cls on pid 30037' 2024-09-16T05:08:49.527 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=30037 2024-09-16T05:08:49.528 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T05:08:49.528 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.528 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s cmd 2024-09-16T05:08:49.528 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' cmd' 2024-09-16T05:08:49.528 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo cmd 2024-09-16T05:08:49.529 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.530 INFO:tasks.workunit.client.0.smithi149.stdout:test cmd on pid 30044 2024-09-16T05:08:49.530 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=cmd 2024-09-16T05:08:49.530 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30044 2024-09-16T05:08:49.530 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test cmd on pid 30044' 2024-09-16T05:08:49.530 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=30044 2024-09-16T05:08:49.531 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T05:08:49.531 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.531 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-09-16T05:08:49.532 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.532 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.532 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.532 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.532 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s handler_error 2024-09-16T05:08:49.532 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_neorados_cmd 2024-09-16T05:08:49.532 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' handler_error' 2024-09-16T05:08:49.532 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_neorados_cmd.log 2024-09-16T05:08:49.534 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-09-16T05:08:49.539 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.539 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.539 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.539 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.539 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_neorados_cls 2024-09-16T05:08:49.539 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_neorados_cls.log 2024-09-16T05:08:49.540 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ cls: /' 2024-09-16T05:08:49.540 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo handler_error 2024-09-16T05:08:49.540 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.542 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ cmd: /' 2024-09-16T05:08:49.544 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=handler_error 2024-09-16T05:08:49.544 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30096 2024-09-16T05:08:49.544 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test handler_error on pid 30096' 2024-09-16T05:08:49.544 INFO:tasks.workunit.client.0.smithi149.stdout:test handler_error on pid 30096 2024-09-16T05:08:49.545 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=30096 2024-09-16T05:08:49.545 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T05:08:49.545 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.546 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s io 2024-09-16T05:08:49.546 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' io' 2024-09-16T05:08:49.546 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-09-16T05:08:49.546 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo io 2024-09-16T05:08:49.546 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.547 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.547 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.547 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.547 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.547 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_neorados_handler_error 2024-09-16T05:08:49.547 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ handler_error: /' 2024-09-16T05:08:49.551 INFO:tasks.workunit.client.0.smithi149.stdout:test io on pid 30115 2024-09-16T05:08:49.551 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=io 2024-09-16T05:08:49.551 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30115 2024-09-16T05:08:49.551 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test io on pid 30115' 2024-09-16T05:08:49.551 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=30115 2024-09-16T05:08:49.551 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T05:08:49.551 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.552 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-09-16T05:08:49.552 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.552 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.553 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.553 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.553 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_neorados_io 2024-09-16T05:08:49.553 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ io: /' 2024-09-16T05:08:49.555 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_neorados_handler_error.log 2024-09-16T05:08:49.561 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_neorados_io.log 2024-09-16T05:08:49.566 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s ec_io 2024-09-16T05:08:49.567 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' ec_io' 2024-09-16T05:08:49.568 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.576 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo ec_io 2024-09-16T05:08:49.577 INFO:tasks.workunit.client.0.smithi149.stdout:test ec_io on pid 30186 2024-09-16T05:08:49.578 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=ec_io 2024-09-16T05:08:49.578 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30186 2024-09-16T05:08:49.578 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test ec_io on pid 30186' 2024-09-16T05:08:49.578 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=30186 2024-09-16T05:08:49.578 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-09-16T05:08:49.578 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T05:08:49.578 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.578 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s list 2024-09-16T05:08:49.579 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' list' 2024-09-16T05:08:49.580 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.580 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.580 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.580 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.580 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_neorados_ec_io 2024-09-16T05:08:49.580 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_neorados_ec_io.log 2024-09-16T05:08:49.582 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo list 2024-09-16T05:08:49.582 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.584 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ ec_io: /' 2024-09-16T05:08:49.585 INFO:tasks.workunit.client.0.smithi149.stdout:test list on pid 30216 2024-09-16T05:08:49.585 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=list 2024-09-16T05:08:49.585 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30216 2024-09-16T05:08:49.585 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test list on pid 30216' 2024-09-16T05:08:49.586 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=30216 2024-09-16T05:08:49.586 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T05:08:49.586 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.586 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-09-16T05:08:49.586 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s ec_list 2024-09-16T05:08:49.587 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' ec_list' 2024-09-16T05:08:49.587 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo ec_list 2024-09-16T05:08:49.587 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.587 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.587 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.587 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.587 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.587 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_neorados_list 2024-09-16T05:08:49.587 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_neorados_list.log 2024-09-16T05:08:49.588 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ list: /' 2024-09-16T05:08:49.589 INFO:tasks.workunit.client.0.smithi149.stdout:test ec_list on pid 30228 2024-09-16T05:08:49.589 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=ec_list 2024-09-16T05:08:49.589 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30228 2024-09-16T05:08:49.589 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test ec_list on pid 30228' 2024-09-16T05:08:49.589 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=30228 2024-09-16T05:08:49.590 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T05:08:49.590 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.590 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-09-16T05:08:49.590 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s misc 2024-09-16T05:08:49.590 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' misc' 2024-09-16T05:08:49.591 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo misc 2024-09-16T05:08:49.591 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.591 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.591 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.591 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.591 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.591 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_neorados_ec_list 2024-09-16T05:08:49.591 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ ec_list: /' 2024-09-16T05:08:49.592 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_neorados_ec_list.log 2024-09-16T05:08:49.592 INFO:tasks.workunit.client.0.smithi149.stdout:test misc on pid 30240 2024-09-16T05:08:49.593 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=misc 2024-09-16T05:08:49.593 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30240 2024-09-16T05:08:49.593 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test misc on pid 30240' 2024-09-16T05:08:49.593 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=30240 2024-09-16T05:08:49.593 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T05:08:49.593 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.593 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-09-16T05:08:49.593 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s pool 2024-09-16T05:08:49.594 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' pool' 2024-09-16T05:08:49.594 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo pool 2024-09-16T05:08:49.595 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.595 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.595 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.595 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.595 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.595 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_neorados_misc 2024-09-16T05:08:49.595 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_neorados_misc.log 2024-09-16T05:08:49.595 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ misc: /' 2024-09-16T05:08:49.596 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=pool 2024-09-16T05:08:49.596 INFO:tasks.workunit.client.0.smithi149.stdout:test pool on pid 30248 2024-09-16T05:08:49.597 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30248 2024-09-16T05:08:49.597 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test pool on pid 30248' 2024-09-16T05:08:49.597 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=30248 2024-09-16T05:08:49.597 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-09-16T05:08:49.597 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T05:08:49.597 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.597 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s read_operations 2024-09-16T05:08:49.597 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' read_operations' 2024-09-16T05:08:49.598 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo read_operations 2024-09-16T05:08:49.598 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.598 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.598 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.598 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.598 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.599 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_neorados_pool 2024-09-16T05:08:49.599 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=read_operations 2024-09-16T05:08:49.599 INFO:tasks.workunit.client.0.smithi149.stdout:test read_operations on pid 30256 2024-09-16T05:08:49.600 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30256 2024-09-16T05:08:49.600 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test read_operations on pid 30256' 2024-09-16T05:08:49.600 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=30256 2024-09-16T05:08:49.600 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T05:08:49.600 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.600 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-09-16T05:08:49.600 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ pool: /' 2024-09-16T05:08:49.601 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.601 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.601 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.601 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.601 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_neorados_read_operations 2024-09-16T05:08:49.601 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_neorados_read_operations.log 2024-09-16T05:08:49.602 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s snapshots 2024-09-16T05:08:49.602 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' snapshots' 2024-09-16T05:08:49.603 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo snapshots 2024-09-16T05:08:49.603 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.606 INFO:tasks.workunit.client.0.smithi149.stdout:test snapshots on pid 30265 2024-09-16T05:08:49.607 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=snapshots 2024-09-16T05:08:49.607 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30265 2024-09-16T05:08:49.607 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test snapshots on pid 30265' 2024-09-16T05:08:49.607 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=30265 2024-09-16T05:08:49.607 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T05:08:49.607 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.607 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_neorados_pool.log 2024-09-16T05:08:49.607 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s watch_notify 2024-09-16T05:08:49.608 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' watch_notify' 2024-09-16T05:08:49.608 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-09-16T05:08:49.610 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.610 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.610 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.610 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.610 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_neorados_snapshots 2024-09-16T05:08:49.611 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_neorados_snapshots.log 2024-09-16T05:08:49.613 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ snapshots: /' 2024-09-16T05:08:49.615 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ read_operations: /' 2024-09-16T05:08:49.622 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo watch_notify 2024-09-16T05:08:49.628 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.635 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=watch_notify 2024-09-16T05:08:49.635 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30359 2024-09-16T05:08:49.635 INFO:tasks.workunit.client.0.smithi149.stdout:test watch_notify on pid 30359 2024-09-16T05:08:49.636 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test watch_notify on pid 30359' 2024-09-16T05:08:49.636 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=30359 2024-09-16T05:08:49.636 INFO:tasks.workunit.client.0.smithi149.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T05:08:49.636 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-09-16T05:08:49.636 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.637 INFO:tasks.workunit.client.0.smithi149.stderr:++ printf %25s write_operations 2024-09-16T05:08:49.637 INFO:tasks.workunit.client.0.smithi149.stderr:+ r=' write_operations' 2024-09-16T05:08:49.637 INFO:tasks.workunit.client.0.smithi149.stderr:++ echo write_operations 2024-09-16T05:08:49.637 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.637 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.637 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.637 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.637 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_neorados_watch_notify 2024-09-16T05:08:49.638 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ watch_notify: /' 2024-09-16T05:08:49.638 INFO:tasks.workunit.client.0.smithi149.stderr:++ awk '{print $1}' 2024-09-16T05:08:49.638 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-09-16T05:08:49.641 INFO:tasks.workunit.client.0.smithi149.stderr:+ ff=write_operations 2024-09-16T05:08:49.641 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30378 2024-09-16T05:08:49.641 INFO:tasks.workunit.client.0.smithi149.stdout:test write_operations on pid 30378 2024-09-16T05:08:49.642 INFO:tasks.workunit.client.0.smithi149.stderr:+ echo 'test write_operations on pid 30378' 2024-09-16T05:08:49.642 INFO:tasks.workunit.client.0.smithi149.stderr:+ pids[$f]=30378 2024-09-16T05:08:49.643 INFO:tasks.workunit.client.0.smithi149.stderr:+ ret=0 2024-09-16T05:08:49.643 INFO:tasks.workunit.client.0.smithi149.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-09-16T05:08:49.643 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T05:08:49.643 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:08:49.643 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29611 2024-09-16T05:08:49.643 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29611 2024-09-16T05:08:49.643 INFO:tasks.workunit.client.0.smithi149.stderr:+ '[' -z '' ']' 2024-09-16T05:08:49.643 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.643 INFO:tasks.workunit.client.0.smithi149.stderr:+ case $- in 2024-09-16T05:08:49.643 INFO:tasks.workunit.client.0.smithi149.stderr:+ return 2024-09-16T05:08:49.645 INFO:tasks.workunit.client.0.smithi149.stderr:+ tee ceph_test_neorados_write_operations.log 2024-09-16T05:08:49.645 INFO:tasks.workunit.client.0.smithi149.stderr:+ ceph_test_neorados_write_operations 2024-09-16T05:08:49.645 INFO:tasks.workunit.client.0.smithi149.stderr:+ sed 's/^/ write_operations: /' 2024-09-16T05:08:50.369 INFO:tasks.workunit.client.0.smithi149.stdout:a640 1 --2- 172.21.15.149:0/3191936048 >> v2:172.21.15.149:6812/1185531135 conn(0x7f04fc035970 0x7f04fc037e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0500002230 tx=0x7f050000c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:50.369 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:49.348+0000 7f050e556640 10 client.4429.objecter ms_handle_connect 0x7f04fc035970 2024-09-16T05:08:50.369 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.359+0000 7f050e556640 1 -- 172.21.15.149:0/3191936048 <== mon.1 v2:172.21.15.149:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v20) ==== 43+0+0 (secure 0 0 0) 0x7f05040d3090 con 0x55c1a8d8f690 2024-09-16T05:08:50.369 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.359+0000 7f050e556640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 pool_op_reply(tid 1 (0) Success v20) 2024-09-16T05:08:50.369 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.359+0000 7f050e556640 10 client.4429.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v20) 2024-09-16T05:08:50.370 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.359+0000 7f050e556640 10 client.4429.objecter have request 1 at 0x55c1a8e1d540 Op: create 2024-09-16T05:08:50.370 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.359+0000 7f050e556640 20 client.4429.objecter waiting for client to reach epoch 20 before calling back 2024-09-16T05:08:50.370 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.359+0000 7f050e556640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:08:50.370 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.359+0000 7f050e556640 10 monclient: _renew_subs 2024-09-16T05:08:50.370 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.359+0000 7f050e556640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:08:50.370 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.359+0000 7f050e556640 1 -- 172.21.15.149:0/3191936048 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=20}) -- 0x7f04fc03ad60 con 0x55c1a8d8f690 2024-09-16T05:08:50.370 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.359+0000 7f050e556640 10 client.4429.objecter done 2024-09-16T05:08:50.370 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f050e556640 1 -- 172.21.15.149:0/3191936048 <== mon.1 v2:172.21.15.149:3300/0 8 ==== osd_map(20..20 src has 1..20) ==== 11367+0+0 (secure 0 0 0) 0x7f05040aec10 con 0x55c1a8d8f690 2024-09-16T05:08:50.370 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f050e556640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_map(20..20 src has 1..20) 2024-09-16T05:08:50.370 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f050e556640 3 client.4429.objecter handle_osd_map got epochs [20,20] > 19 2024-09-16T05:08:50.370 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f050e556640 3 client.4429.objecter handle_osd_map decoding incremental epoch 20 2024-09-16T05:08:50.370 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f050e556640 20 client.4429.objecter dump_active .. 0 homeless 2024-09-16T05:08:50.370 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f0513247880 10 librados: set snap read head -> 1 2024-09-16T05:08:50.370 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f0513247880 10 librados: writefull oid=exist nspace= 2024-09-16T05:08:50.371 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8e1a340 2024-09-16T05:08:50.371 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 20 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T05:08:50.371 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f0513247880 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T05:08:50.371 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:50.371 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f0513247880 1 --2- 172.21.15.149:0/3191936048 >> v2:172.21.15.149:6805/1362772130 conn(0x55c1a8ecc0d0 0x55c1a8ece4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:08:50.371 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:50.371 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f0513247880 10 client.4429.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 2 osd.3 2024-09-16T05:08:51.670 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.671 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 2 2024-09-16T05:08:51.671 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f0513247880 15 client.4429.objecter _send_op 2 to 6.1 on osd.3 2024-09-16T05:08:51.671 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:2 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) -- 0x55c1a8ecea00 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.671 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.671 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.363+0000 7f0513247880 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.671 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.367+0000 7f051155c640 1 --2- 172.21.15.149:0/3191936048 >> v2:172.21.15.149:6805/1362772130 conn(0x55c1a8ecc0d0 0x55c1a8ece4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:08:51.671 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.367+0000 7f051155c640 10 monclient: get_auth_request con 0x55c1a8ecc0d0 auth_method 0 2024-09-16T05:08:51.671 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.379+0000 7f050e556640 10 client.4429.objecter ms_handle_connect 0x55c1a8ecc0d0 2024-09-16T05:08:51.671 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:50.379+0000 7f051155c640 1 --2- 172.21.15.149:0/3191936048 >> v2:172.21.15.149:6805/1362772130 conn(0x55c1a8ecc0d0 0x55c1a8ece4c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:08:51.671 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.627+0000 7f050e556640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 1 ==== osd_map(21..21 src has 1..21) ==== 10637+0+0 (crc 0 0 0) 0x7f0508006c00 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.672 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.627+0000 7f050e556640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_map(21..21 src has 1..21) 2024-09-16T05:08:51.672 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.627+0000 7f050e556640 3 client.4429.objecter handle_osd_map got epochs [21,21] > 20 2024-09-16T05:08:51.672 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.627+0000 7f050e556640 3 client.4429.objecter handle_osd_map decoding incremental epoch 21 2024-09-16T05:08:51.672 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.627+0000 7f050e556640 10 client.4429.objecter checking op 2 2024-09-16T05:08:51.672 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.627+0000 7f050e556640 20 client.4429.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T05:08:51.672 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.627+0000 7f050e556640 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T05:08:51.672 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.627+0000 7f050e556640 20 client.4429.objecter dump_active .. 0 homeless 2024-09-16T05:08:51.672 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.627+0000 7f050e556640 20 client.4429.objecter 2 6.d3a59201 osd.3 exist [writefull 0~5 in=5b] 2024-09-16T05:08:51.672 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 2 ==== osd_op_reply(2 exist [writefull 0~5] v21'1 uv1 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.672 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(2 exist [writefull 0~5] v21'1 uv1 ondisk = 0) 2024-09-16T05:08:51.672 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.672 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 2 ondisk uv 1 in 6.1 attempt 0 2024-09-16T05:08:51.672 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 0 2024-09-16T05:08:51.673 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 2 2024-09-16T05:08:51.673 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter _finish_op 2 2024-09-16T05:08:51.673 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.673 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 2 2024-09-16T05:08:51.673 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-16T05:08:51.673 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 5 client.4429.objecter 0 in flight 2024-09-16T05:08:51.673 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 librados: Objecter returned from writefull r=0 2024-09-16T05:08:51.673 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8e1a340 2024-09-16T05:08:51.673 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T05:08:51.673 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T05:08:51.673 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.673 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.673 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _op_submit oid exist '@6' '@6' [read 0~256] tid 3 osd.3 2024-09-16T05:08:51.674 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.674 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 3 2024-09-16T05:08:51.674 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 15 client.4429.objecter _send_op 3 to 6.1 on osd.3 2024-09-16T05:08:51.674 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:3 6.1 6:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ecabb0 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.674 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.674 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.674 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8ecaf50 2024-09-16T05:08:51.674 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T05:08:51.674 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-16T05:08:51.674 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.674 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.674 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _op_submit oid noexist '@6' '@6' [read 0~256] tid 4 osd.3 2024-09-16T05:08:51.674 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.675 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 4 2024-09-16T05:08:51.675 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 15 client.4429.objecter _send_op 4 to 6.1 on osd.3 2024-09-16T05:08:51.675 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:4 6.1 6:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ecb400 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.675 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 5 2024-09-16T05:08:51.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 5 client.4429.objecter 2 in flight 2024-09-16T05:08:51.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-16T05:08:51.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 3 ondisk uv 1 in 6.1 attempt 0 2024-09-16T05:08:51.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 5 2024-09-16T05:08:51.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 3 2024-09-16T05:08:51.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter _finish_op 3 2024-09-16T05:08:51.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 5 2024-09-16T05:08:51.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 3 2024-09-16T05:08:51.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T05:08:51.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 4 ondisk uv 0 in 6.1 attempt 0 2024-09-16T05:08:51.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T05:08:51.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 0 2024-09-16T05:08:51.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 4 2024-09-16T05:08:51.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter _finish_op 4 2024-09-16T05:08:51.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 4 2024-09-16T05:08:51.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 5 client.4429.objecter 0 in flight 2024-09-16T05:08:51.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1 ms) 2024-09-16T05:08:51.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-16T05:08:51.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8ecaf50 2024-09-16T05:08:51.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T05:08:51.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T05:08:51.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _op_submit oid exist '@6' '@6' [read 0~256] tid 5 osd.3 2024-09-16T05:08:51.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 5 2024-09-16T05:08:51.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 15 client.4429.objecter _send_op 5 to 6.1 on osd.3 2024-09-16T05:08:51.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:5 6.1 6:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ecb7d0 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8e1a340 2024-09-16T05:08:51.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T05:08:51.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-16T05:08:51.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _op_submit oid noexist '@6' '@6' [read 0~256] tid 6 osd.3 2024-09-16T05:08:51.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 6 2024-09-16T05:08:51.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 15 client.4429.objecter _send_op 6 to 6.1 on osd.3 2024-09-16T05:08:51.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:6 6.1 6:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8eceda0 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 5 2024-09-16T05:08:51.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 5 client.4429.objecter 2 in flight 2024-09-16T05:08:51.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-16T05:08:51.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 5 ondisk uv 1 in 6.1 attempt 0 2024-09-16T05:08:51.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 5 2024-09-16T05:08:51.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 5 2024-09-16T05:08:51.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter _finish_op 5 2024-09-16T05:08:51.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 5 2024-09-16T05:08:51.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 5 2024-09-16T05:08:51.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T05:08:51.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 6 ondisk uv 0 in 6.1 attempt 0 2024-09-16T05:08:51.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T05:08:51.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 0 2024-09-16T05:08:51.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 6 2024-09-16T05:08:51.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter _finish_op 6 2024-09-16T05:08:51.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 6 2024-09-16T05:08:51.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 5 client.4429.objecter 0 in flight 2024-09-16T05:08:51.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-09-16T05:08:51.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-16T05:08:51.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8e1a340 2024-09-16T05:08:51.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T05:08:51.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T05:08:51.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _op_submit oid exist '@6' '@6' [read 0~256] tid 7 osd.3 2024-09-16T05:08:51.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 7 2024-09-16T05:08:51.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 15 client.4429.objecter _send_op 7 to 6.1 on osd.3 2024-09-16T05:08:51.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:7 6.1 6:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ecaf50 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8f0f7d0 2024-09-16T05:08:51.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T05:08:51.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-16T05:08:51.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _op_submit oid noexist '@6' '@6' [read 0~256] tid 8 osd.3 2024-09-16T05:08:51.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 8 2024-09-16T05:08:51.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 15 client.4429.objecter _send_op 8 to 6.1 on osd.3 2024-09-16T05:08:51.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:8 6.1 6:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8f0fc80 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 5 2024-09-16T05:08:51.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 5 client.4429.objecter 2 in flight 2024-09-16T05:08:51.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-09-16T05:08:51.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-16T05:08:51.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-16T05:08:51.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 7 ondisk uv 1 in 6.1 attempt 0 2024-09-16T05:08:51.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 5 2024-09-16T05:08:51.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 7 2024-09-16T05:08:51.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter _finish_op 7 2024-09-16T05:08:51.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 5 2024-09-16T05:08:51.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 7 2024-09-16T05:08:51.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T05:08:51.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 8 ondisk uv 0 in 6.1 attempt 0 2024-09-16T05:08:51.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T05:08:51.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 0 2024-09-16T05:08:51.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 8 2024-09-16T05:08:51.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter _finish_op 8 2024-09-16T05:08:51.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 8 2024-09-16T05:08:51.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f051155c640 5 client.4429.objecter 0 in flight 2024-09-16T05:08:51.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-16T05:08:51.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 librados: queue_aio_write 0x55c1a8ce07a0 completion 0x55c1a8cac450 write_seq 1 2024-09-16T05:08:51.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8e1a340 2024-09-16T05:08:51.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T05:08:51.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T05:08:51.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 10 client.4429.objecter _op_submit oid exist '@6' '@6' [write 0~5 in=5b] tid 9 osd.3 2024-09-16T05:08:51.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 9 2024-09-16T05:08:51.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 15 client.4429.objecter _send_op 9 to 6.1 on osd.3 2024-09-16T05:08:51.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:9 6.1 6:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ed0150 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.667+0000 7f0513247880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-16T05:08:51.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 9 ==== osd_op_reply(9 exist [write 0~5] v21'2 uv2 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(9 exist [write 0~5] v21'2 uv2 ondisk = 0) 2024-09-16T05:08:51.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 9 ondisk uv 2 in 6.1 attempt 0 2024-09-16T05:08:51.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 0 2024-09-16T05:08:51.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 9 2024-09-16T05:08:51.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f051155c640 15 client.4429.objecter _finish_op 9 2024-09-16T05:08:51.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 9 2024-09-16T05:08:51.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f051155c640 5 client.4429.objecter 0 in flight 2024-09-16T05:08:51.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f051155c640 20 librados: complete_aio_write 0x55c1a8cac450 2024-09-16T05:08:51.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (2 ms) 2024-09-16T05:08:51.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-16T05:08:51.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f0513247880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-16T05:08:51.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f0513247880 20 librados: queue_aio_write 0x55c1a8ce07a0 completion 0x55c1a8cb9520 write_seq 2 2024-09-16T05:08:51.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8e1a340 2024-09-16T05:08:51.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T05:08:51.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f0513247880 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T05:08:51.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f0513247880 10 client.4429.objecter _op_submit oid exist '@6' '@6' [write 0~5 in=5b] tid 10 osd.3 2024-09-16T05:08:51.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 10 2024-09-16T05:08:51.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f0513247880 15 client.4429.objecter _send_op 10 to 6.1 on osd.3 2024-09-16T05:08:51.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:10 6.1 6:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ed0150 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f0513247880 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.671+0000 7f0513247880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-16T05:08:51.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 10 ==== osd_op_reply(10 exist [write 0~5] v21'3 uv3 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(10 exist [write 0~5] v21'3 uv3 ondisk = 0) 2024-09-16T05:08:51.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 10 ondisk uv 3 in 6.1 attempt 0 2024-09-16T05:08:51.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 0 2024-09-16T05:08:51.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 10 2024-09-16T05:08:51.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f051155c640 15 client.4429.objecter _finish_op 10 2024-09-16T05:08:51.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 10 2024-09-16T05:08:51.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f051155c640 5 client.4429.objecter 0 in flight 2024-09-16T05:08:51.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f051155c640 20 librados: complete_aio_write 0x55c1a8cb9520 2024-09-16T05:08:51.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (9 ms) 2024-09-16T05:08:51.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-16T05:08:51.695 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f0513247880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-16T05:08:51.695 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f0513247880 20 librados: queue_aio_write 0x55c1a8ce07a0 completion 0x55c1a8e1a340 write_seq 3 2024-09-16T05:08:51.695 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8ef26d0 2024-09-16T05:08:51.695 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T05:08:51.695 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f0513247880 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T05:08:51.695 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.696 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.696 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f0513247880 10 client.4429.objecter _op_submit oid exist '@6' '@6' [write 0~5 in=5b] tid 11 osd.3 2024-09-16T05:08:51.696 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.696 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 11 2024-09-16T05:08:51.696 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f0513247880 15 client.4429.objecter _send_op 11 to 6.1 on osd.3 2024-09-16T05:08:51.696 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:11 6.1 6:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ef2b80 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.696 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.696 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f0513247880 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.696 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.679+0000 7f0513247880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-16T05:08:51.696 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 11 ==== osd_op_reply(11 exist [write 0~5] v21'4 uv4 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.696 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(11 exist [write 0~5] v21'4 uv4 ondisk = 0) 2024-09-16T05:08:51.696 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.696 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 11 ondisk uv 4 in 6.1 attempt 0 2024-09-16T05:08:51.697 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 0 2024-09-16T05:08:51.697 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 11 2024-09-16T05:08:51.697 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter _finish_op 11 2024-09-16T05:08:51.697 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.697 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 11 2024-09-16T05:08:51.697 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 5 client.4429.objecter 0 in flight 2024-09-16T05:08:51.697 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 20 librados: complete_aio_write 0x55c1a8e1a340 2024-09-16T05:08:51.697 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (11 ms) 2024-09-16T05:08:51.697 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-16T05:08:51.697 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8ecf140 2024-09-16T05:08:51.697 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T05:08:51.698 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T05:08:51.698 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.698 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.698 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _op_submit oid exist '@6' '@6' [read 0~0] tid 12 osd.3 2024-09-16T05:08:51.698 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.698 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 12 2024-09-16T05:08:51.699 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 15 client.4429.objecter _send_op 12 to 6.1 on osd.3 2024-09-16T05:08:51.699 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:12 6.1 6:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ecf5f0 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.699 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.699 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.699 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8ecf990 2024-09-16T05:08:51.699 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T05:08:51.699 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-16T05:08:51.699 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.699 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.699 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _op_submit oid noexist '@6' '@6' [read 0~0] tid 13 osd.3 2024-09-16T05:08:51.699 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.699 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 13 2024-09-16T05:08:51.699 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 15 client.4429.objecter _send_op 13 to 6.1 on osd.3 2024-09-16T05:08:51.700 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:13 6.1 6:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ef2fd0 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.700 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 5 2024-09-16T05:08:51.700 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 5 client.4429.objecter 2 in flight 2024-09-16T05:08:51.700 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.700 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-16T05:08:51.700 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.700 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 12 ondisk uv 4 in 6.1 attempt 0 2024-09-16T05:08:51.700 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 5 2024-09-16T05:08:51.700 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 12 2024-09-16T05:08:51.700 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter _finish_op 12 2024-09-16T05:08:51.700 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 5 2024-09-16T05:08:51.700 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 12 2024-09-16T05:08:51.700 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.701 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.701 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T05:08:51.701 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.701 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 13 ondisk uv 0 in 6.1 attempt 0 2024-09-16T05:08:51.701 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T05:08:51.701 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (0 ms) 2024-09-16T05:08:51.701 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 0 2024-09-16T05:08:51.701 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-16T05:08:51.701 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 13 2024-09-16T05:08:51.701 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter _finish_op 13 2024-09-16T05:08:51.701 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.701 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 13 2024-09-16T05:08:51.701 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 5 client.4429.objecter 0 in flight 2024-09-16T05:08:51.702 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8ecf140 2024-09-16T05:08:51.702 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T05:08:51.702 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T05:08:51.702 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.702 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.702 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _op_submit oid exist '@6' '@6' [read 0~0] tid 14 osd.3 2024-09-16T05:08:51.702 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.702 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 14 2024-09-16T05:08:51.702 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 15 client.4429.objecter _send_op 14 to 6.1 on osd.3 2024-09-16T05:08:51.702 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:14 6.1 6:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ecf5f0 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.702 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.702 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.702 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8ecfba0 2024-09-16T05:08:51.703 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T05:08:51.703 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-16T05:08:51.703 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.703 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.703 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _op_submit oid noexist '@6' '@6' [read 0~0] tid 15 osd.3 2024-09-16T05:08:51.703 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.703 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 15 2024-09-16T05:08:51.703 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 15 client.4429.objecter _send_op 15 to 6.1 on osd.3 2024-09-16T05:08:51.703 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:15 6.1 6:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ef2fd0 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.703 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 5 2024-09-16T05:08:51.703 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 5 client.4429.objecter 2 in flight 2024-09-16T05:08:51.703 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.703 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-16T05:08:51.703 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.704 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 14 ondisk uv 4 in 6.1 attempt 0 2024-09-16T05:08:51.704 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 5 2024-09-16T05:08:51.704 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 14 2024-09-16T05:08:51.704 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter _finish_op 14 2024-09-16T05:08:51.704 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 5 2024-09-16T05:08:51.704 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 14 2024-09-16T05:08:51.704 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.704 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.704 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T05:08:51.704 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.704 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 15 ondisk uv 0 in 6.1 attempt 0 2024-09-16T05:08:51.704 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T05:08:51.705 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 0 2024-09-16T05:08:51.705 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 15 2024-09-16T05:08:51.705 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter _finish_op 15 2024-09-16T05:08:51.705 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.705 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 15 2024-09-16T05:08:51.705 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 5 client.4429.objecter 0 in flight 2024-09-16T05:08:51.705 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-09-16T05:08:51.705 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-16T05:08:51.705 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8ecfba0 2024-09-16T05:08:51.705 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T05:08:51.705 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T05:08:51.705 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.705 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.705 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _op_submit oid exist '@6' '@6' [read 0~0] tid 16 osd.3 2024-09-16T05:08:51.706 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.706 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 16 2024-09-16T05:08:51.706 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 15 client.4429.objecter _send_op 16 to 6.1 on osd.3 2024-09-16T05:08:51.706 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:16 6.1 6:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ef2060 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.706 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.706 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.706 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8ef2610 2024-09-16T05:08:51.706 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T05:08:51.706 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-16T05:08:51.706 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.706 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.706 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _op_submit oid noexist '@6' '@6' [read 0~0] tid 17 osd.3 2024-09-16T05:08:51.706 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.706 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 17 2024-09-16T05:08:51.707 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 15 client.4429.objecter _send_op 17 to 6.1 on osd.3 2024-09-16T05:08:51.707 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:17 6.1 6:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ef2ac0 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.707 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 5 2024-09-16T05:08:51.707 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 5 client.4429.objecter 2 in flight 2024-09-16T05:08:51.707 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.707 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-16T05:08:51.707 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.707 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 16 ondisk uv 4 in 6.1 attempt 0 2024-09-16T05:08:51.707 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 5 2024-09-16T05:08:51.707 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 16 2024-09-16T05:08:51.707 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter _finish_op 16 2024-09-16T05:08:51.707 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 5 2024-09-16T05:08:51.707 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 16 2024-09-16T05:08:51.707 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.708 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.708 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T05:08:51.708 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.708 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 17 ondisk uv 0 in 6.1 attempt 0 2024-09-16T05:08:51.708 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T05:08:51.708 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 0 2024-09-16T05:08:51.708 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 17 2024-09-16T05:08:51.708 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter _finish_op 17 2024-09-16T05:08:51.708 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.708 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 17 2024-09-16T05:08:51.708 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f051155c640 5 client.4429.objecter 0 in flight 2024-09-16T05:08:51.708 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (0 ms) 2024-09-16T05:08:51.708 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-16T05:08:51.709 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 librados: queue_aio_write 0x55c1a8ce07a0 completion 0x55c1a8ecfa90 write_seq 4 2024-09-16T05:08:51.709 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8ecfba0 2024-09-16T05:08:51.709 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T05:08:51.709 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T05:08:51.709 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.709 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.710 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 10 client.4429.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 18 osd.3 2024-09-16T05:08:51.710 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.710 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 18 2024-09-16T05:08:51.710 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 15 client.4429.objecter _send_op 18 to 6.1 on osd.3 2024-09-16T05:08:51.710 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:18 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ecf140 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.710 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.710 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.691+0000 7f0513247880 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.710 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 18 ==== osd_op_reply(18 exist [writefull 0~5] v21'5 uv5 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.710 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(18 exist [writefull 0~5] v21'5 uv5 ondisk = 0) 2024-09-16T05:08:51.710 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.710 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 18 ondisk uv 5 in 6.1 attempt 0 2024-09-16T05:08:51.710 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (3 ms) 2024-09-16T05:08:51.711 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 0 2024-09-16T05:08:51.711 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-16T05:08:51.711 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 18 2024-09-16T05:08:51.711 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f051155c640 15 client.4429.objecter _finish_op 18 2024-09-16T05:08:51.711 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.711 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 18 2024-09-16T05:08:51.711 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f051155c640 5 client.4429.objecter 0 in flight 2024-09-16T05:08:51.711 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f051155c640 20 librados: complete_aio_write 0x55c1a8ecfa90 2024-09-16T05:08:51.711 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f0513247880 20 librados: queue_aio_write 0x55c1a8ce07a0 completion 0x55c1a8e1a490 write_seq 5 2024-09-16T05:08:51.711 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8ecfc10 2024-09-16T05:08:51.711 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T05:08:51.711 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f0513247880 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T05:08:51.712 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.712 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.712 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f0513247880 10 client.4429.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 19 osd.3 2024-09-16T05:08:51.712 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.712 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 19 2024-09-16T05:08:51.712 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f0513247880 15 client.4429.objecter _send_op 19 to 6.1 on osd.3 2024-09-16T05:08:51.712 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:19 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ecf140 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.712 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.712 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.695+0000 7f0513247880 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.712 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 19 ==== osd_op_reply(19 exist [writefull 0~5] v21'6 uv6 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.712 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(19 exist [writefull 0~5] v21'6 uv6 ondisk = 0) 2024-09-16T05:08:51.713 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.713 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 19 ondisk uv 6 in 6.1 attempt 0 2024-09-16T05:08:51.713 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 0 2024-09-16T05:08:51.713 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 19 2024-09-16T05:08:51.713 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f051155c640 15 client.4429.objecter _finish_op 19 2024-09-16T05:08:51.713 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.713 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 19 2024-09-16T05:08:51.713 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f051155c640 5 client.4429.objecter 0 in flight 2024-09-16T05:08:51.713 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f051155c640 20 librados: complete_aio_write 0x55c1a8e1a490 2024-09-16T05:08:51.713 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (5 ms) 2024-09-16T05:08:51.713 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-16T05:08:51.714 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f0513247880 20 librados: queue_aio_write 0x55c1a8ce07a0 completion 0x55c1a8e1a490 write_seq 6 2024-09-16T05:08:51.714 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f0513247880 10 client.4429.objecter _op_submit op 0x55c1a8ecfc10 2024-09-16T05:08:51.714 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f0513247880 20 client.4429.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T05:08:51.714 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f0513247880 20 client.4429.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-16T05:08:51.714 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f0513247880 10 client.4429.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [3,1] primary 3 2024-09-16T05:08:51.714 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f0513247880 20 client.4429.objecter _get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.714 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f0513247880 10 client.4429.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 20 osd.3 2024-09-16T05:08:51.714 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f0513247880 20 client.4429.objecter get_session s=0x55c1a8e1d770 osd=3 3 2024-09-16T05:08:51.714 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f0513247880 15 client.4429.objecter _session_op_assign 3 20 2024-09-16T05:08:51.714 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f0513247880 15 client.4429.objecter _send_op 20 to 6.1 on osd.3 2024-09-16T05:08:51.714 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 --> v2:172.21.15.149:6805/1362772130 -- osd_op(unknown.0.0:20 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21) -- 0x55c1a8ecf140 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.715 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.715 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.699+0000 7f0513247880 5 client.4429.objecter 1 in flight 2024-09-16T05:08:51.715 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 <== osd.3 v2:172.21.15.149:6805/1362772130 20 ==== osd_op_reply(20 exist [writefull 0~5] v21'7 uv7 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f0508002800 con 0x55c1a8ecc0d0 2024-09-16T05:08:51.715 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f051155c640 10 client.4429.objecter ms_dispatch 0x55c1a8ceb0c0 osd_op_reply(20 exist [writefull 0~5] v21'7 uv7 ondisk = 0) 2024-09-16T05:08:51.715 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f051155c640 10 client.4429.objecter in handle_osd_op_reply 2024-09-16T05:08:51.715 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f051155c640 7 client.4429.objecter handle_osd_op_reply 20 ondisk uv 7 in 6.1 attempt 0 2024-09-16T05:08:51.715 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f051155c640 10 client.4429.objecter op 0 rval 0 len 0 2024-09-16T05:08:51.715 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f051155c640 15 client.4429.objecter handle_osd_op_reply completed tid 20 2024-09-16T05:08:51.715 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f051155c640 15 client.4429.objecter _finish_op 20 2024-09-16T05:08:51.715 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f051155c640 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 4 2024-09-16T05:08:51.715 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f051155c640 15 client.4429.objecter _session_op_remove 3 20 2024-09-16T05:08:51.715 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f051155c640 5 client.4429.objecter 0 in flight 2024-09-16T05:08:51.716 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f051155c640 20 librados: complete_aio_write 0x55c1a8e1a490 2024-09-16T05:08:51.716 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (3 ms) 2024-09-16T05:08:51.716 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 10 librados: watch_flush enter 2024-09-16T05:08:51.716 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 10 librados: watch_flush exit 2024-09-16T05:08:51.716 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 10 client.4429.objecter close_session for osd.3 2024-09-16T05:08:51.716 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 >> v2:172.21.15.149:6805/1362772130 conn(0x55c1a8ecc0d0 msgr2=0x55c1a8ece4c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:51.716 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 1 --2- 172.21.15.149:0/3191936048 >> v2:172.21.15.149:6805/1362772130 conn(0x55c1a8ecc0d0 0x55c1a8ece4c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:08:51.716 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 20 client.4429.objecter put_session s=0x55c1a8e1d770 osd=3 1 2024-09-16T05:08:51.716 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 20 client.4429.objecter shutdown clearing up homeless session... 2024-09-16T05:08:51.716 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 10 client.4429.objecter successfully canceled tick 2024-09-16T05:08:51.716 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 >> v2:172.21.15.149:6812/1185531135 conn(0x7f04fc035970 msgr2=0x7f04fc037e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:51.717 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 1 --2- 172.21.15.149:0/3191936048 >> v2:172.21.15.149:6812/1185531135 conn(0x7f04fc035970 0x7f04fc037e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0500002230 tx=0x7f050000c040 comp rx=0 tx=0).stop 2024-09-16T05:08:51.717 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 10 monclient: shutdown 2024-09-16T05:08:51.717 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:08:51.717 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c1a8d8f690 msgr2=0x55c1a8e27880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:08:51.717 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 1 --2- 172.21.15.149:0/3191936048 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c1a8d8f690 0x55c1a8e27880 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f050401fbf0 tx=0x7f050401c0f0 comp rx=0 tx=0).stop 2024-09-16T05:08:51.717 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f051155c640 1 -- 172.21.15.149:0/3191936048 reap_dead start 2024-09-16T05:08:51.717 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 shutdown_connections 2024-09-16T05:08:51.718 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 >> 172.21.15.149:0/3191936048 conn(0x55c1a8d94100 msgr2=0x55c1a8d91df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:08:51.718 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 shutdown_connections 2024-09-16T05:08:51.718 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 1 -- 172.21.15.149:0/3191936048 wait complete. 2024-09-16T05:08:51.718 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.703+0000 7f0513247880 1 librados: shutdown 2024-09-16T05:08:51.718 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [----------] 12 tests from AsioRados (37 ms total) 2024-09-16T05:08:51.718 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: 2024-09-16T05:08:51.718 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [----------] Global test environment tear-down 2024-09-16T05:08:51.718 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (2374 ms total) 2024-09-16T05:08:51.718 INFO:tasks.workunit.client.0.smithi149.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-16T05:08:52.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-09-16T05:08:52.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [----------] Global test environment set-up. 2024-09-16T05:08:52.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-16T05:08:52.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-16T05:08:52.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ OK ] LibRadosList.ListObjects (340 ms) 2024-09-16T05:08:52.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-16T05:08:52.147 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (33 ms) 2024-09-16T05:08:52.147 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-16T05:08:52.147 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: myset foo1,foo2,foo3 2024-09-16T05:08:52.147 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo1 2024-09-16T05:08:52.147 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo2 2024-09-16T05:08:52.147 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo3 2024-09-16T05:08:52.147 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: myset foo1,foo4,foo5 2024-09-16T05:08:52.147 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo4 2024-09-16T05:08:52.147 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo5 2024-09-16T05:08:52.147 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo1 2024-09-16T05:08:52.147 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: myset foo6,foo7 2024-09-16T05:08:52.147 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo7 2024-09-16T05:08:52.147 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo6 2024-09-16T05:08:52.147 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-16T05:08:52.148 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: ns1:foo4 2024-09-16T05:08:52.148 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: ns1:foo5 2024-09-16T05:08:52.148 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: ns2:foo7 2024-09-16T05:08:52.148 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: ns2:foo6 2024-09-16T05:08:52.148 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: ns1:foo1 2024-09-16T05:08:52.148 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: :foo1 2024-09-16T05:08:52.148 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: :foo2 2024-09-16T05:08:52.148 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: :foo3 2024-09-16T05:08:52.148 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (107 ms) 2024-09-16T05:08:52.148 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-16T05:08:52.148 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 1 0 2024-09-16T05:08:52.148 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 10 0 2024-09-16T05:08:52.149 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 13 0 2024-09-16T05:08:52.149 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 7 0 2024-09-16T05:08:52.149 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 14 0 2024-09-16T05:08:52.149 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 0 0 2024-09-16T05:08:52.149 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 15 0 2024-09-16T05:08:52.149 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 11 0 2024-09-16T05:08:52.149 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 5 0 2024-09-16T05:08:52.149 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 8 0 2024-09-16T05:08:52.149 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 6 0 2024-09-16T05:08:52.149 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 3 0 2024-09-16T05:08:52.149 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 4 0 2024-09-16T05:08:52.150 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 12 0 2024-09-16T05:08:52.150 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 9 0 2024-09-16T05:08:52.150 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 2 0 2024-09-16T05:08:52.150 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-16T05:08:52.150 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (158 ms) 2024-09-16T05:08:52.150 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-16T05:08:52.150 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: x cursor=MIN 2024-09-16T05:08:52.150 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=1 cursor=9:02547ec2:::1:head 2024-09-16T05:08:52.150 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=10 cursor=9:52ea6a34:::10:head 2024-09-16T05:08:52.150 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=13 cursor=9:566253c9:::13:head 2024-09-16T05:08:52.150 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=7 cursor=9:5c6b0b28:::7:head 2024-09-16T05:08:52.150 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=14 cursor=9:62a1935d:::14:head 2024-09-16T05:08:52.150 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=0 cursor=9:6cac518f:::0:head 2024-09-16T05:08:52.151 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=15 cursor=9:863748b0:::15:head 2024-09-16T05:08:52.151 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=11 cursor=9:89d3ae78:::11:head 2024-09-16T05:08:52.151 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=5 cursor=9:b29083e3:::5:head 2024-09-16T05:08:52.151 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=8 cursor=9:bd63b0f1:::8:head 2024-09-16T05:08:52.151 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=6 cursor=9:c4fdafeb:::6:head 2024-09-16T05:08:52.151 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=3 cursor=9:cfc208b3:::3:head 2024-09-16T05:08:52.151 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=4 cursor=9:d83876eb:::4:head 2024-09-16T05:08:52.151 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=12 cursor=9:de5d7c5f:::12:head 2024-09-16T05:08:52.151 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=9 cursor=9:e960b815:::9:head 2024-09-16T05:08:52.151 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > oid=2 cursor=9:f905c69b:::2:head 2024-09-16T05:08:52.151 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-16T05:08:52.151 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=1 cursor=9:02547ec2:::1:head 2024-09-16T05:08:52.151 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:02547ec2:::1:head 2024-09-16T05:08:52.151 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:02547ec2:::1:head -> 1 2024-09-16T05:08:52.152 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=10 cursor=9:52ea6a34:::10:head 2024-09-16T05:08:52.152 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:52ea6a34:::10:head 2024-09-16T05:08:52.152 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:52ea6a34:::10:head -> 10 2024-09-16T05:08:52.152 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=13 cursor=9:566253c9:::13:head 2024-09-16T05:08:52.152 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:566253c9:::13:head 2024-09-16T05:08:52.152 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:566253c9:::13:head -> 13 2024-09-16T05:08:52.152 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=7 cursor=9:5c6b0b28:::7:head 2024-09-16T05:08:52.152 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:5c6b0b28:::7:head 2024-09-16T05:08:52.152 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:5c6b0b28:::7:head -> 7 2024-09-16T05:08:52.152 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=14 cursor=9:62a1935d:::14:head 2024-09-16T05:08:52.211 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:62a1935d:::14:head 2024-09-16T05:08:52.212 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:62a1935d:::14:head -> 14 2024-09-16T05:08:52.212 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=0 cursor=9:6cac518f:::0:head 2024-09-16T05:08:52.212 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:6cac518f:::0:head 2024-09-16T05:08:52.212 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:6cac518f:::0:head -> 0 2024-09-16T05:08:52.212 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=15 cursor=9:863748b0:::15:head 2024-09-16T05:08:52.212 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:863748b0:::15:head 2024-09-16T05:08:52.212 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:863748b0:::15:head -> 15 2024-09-16T05:08:52.212 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=11 cursor=9:89d3ae78:::11:head 2024-09-16T05:08:52.212 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:89d3ae78:::11:head 2024-09-16T05:08:52.212 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:89d3ae78:::11:head -> 11 2024-09-16T05:08:52.212 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=5 cursor=9:b29083e3:::5:head 2024-09-16T05:08:52.213 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:b29083e3:::5:head 2024-09-16T05:08:52.213 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:b29083e3:::5:head -> 5 2024-09-16T05:08:52.213 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=8 cursor=9:bd63b0f1:::8:head 2024-09-16T05:08:52.213 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:bd63b0f1:::8:head 2024-09-16T05:08:52.213 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:bd63b0f1:::8:head -> 8 2024-09-16T05:08:52.213 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=6 cursor=9:c4fdafeb:::6:head 2024-09-16T05:08:52.213 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:c4fdafeb:::6:head 2024-09-16T05:08:52.213 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:c4fdafeb:::6:head -> 6 2024-09-16T05:08:52.213 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=3 cursor=9:cfc208b3:::3:head 2024-09-16T05:08:52.213 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:cfc208b3:::3:head 2024-09-16T05:08:52.213 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:cfc208b3:::3:head -> 3 2024-09-16T05:08:52.213 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=4 cursor=9:d83876eb:::4:head 2024-09-16T05:08:52.213 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:d83876eb:::4:head 2024-09-16T05:08:52.214 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:d83876eb:::4:head -> 4 2024-09-16T05:08:52.214 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=12 cursor=9:de5d7c5f:::12:head 2024-09-16T05:08:52.214 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:de5d7c5f:::12:head 2024-09-16T05:08:52.214 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:de5d7c5f:::12:head -> 12 2024-09-16T05:08:52.214 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=9 cursor=9:e960b815:::9:head 2024-09-16T05:08:52.214 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:e960b815:::9:head 2024-09-16T05:08:52.214 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:e960b815:::9:head -> 9 2024-09-16T05:08:52.214 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : oid=2 cursor=9:f905c69b:::2:head 2024-09-16T05:08:52.214 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:f905c69b:::2:head 2024-09-16T05:08:52.214 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:f905c69b:::2:head -> 2 2024-09-16T05:08:52.214 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:e960b815:::9:head 2024-09-16T05:08:52.214 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cursor()=9:e960b815:::9:head expected=9:e960b815:::9:head 2024-09-16T05:08:52.214 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:e960b815:::9:head -> 9 2024-09-16T05:08:52.214 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=9 expected=9 2024-09-16T05:08:52.215 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:de5d7c5f:::12:head 2024-09-16T05:08:52.215 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cursor()=9:de5d7c5f:::12:head expected=9:de5d7c5f:::12:head 2024-09-16T05:08:52.215 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:de5d7c5f:::12:head -> 12 2024-09-16T05:08:52.215 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=12 expected=12 2024-09-16T05:08:52.215 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:f905c69b:::2:head 2024-09-16T05:08:52.215 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cursor()=9:f905c69b:::2:head expected=9:f905c69b:::2:head 2024-09-16T05:08:52.215 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:f905c69b:::2:head -> 2 2024-09-16T05:08:52.215 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=2 expected=2 2024-09-16T05:08:52.215 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:d83876eb:::4:head 2024-09-16T05:08:52.215 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cursor()=9:d83876eb:::4:head expected=9:d83876eb:::4:head 2024-09-16T05:08:52.215 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:d83876eb:::4:head -> 4 2024-09-16T05:08:52.215 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=4 expected=4 2024-09-16T05:08:52.215 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:cfc208b3:::3:head 2024-09-16T05:08:52.216 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cursor()=9:cfc208b3:::3:head expected=9:cfc208b3:::3:head 2024-09-16T05:08:52.216 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:cfc208b3:::3:head -> 3 2024-09-16T05:08:52.216 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=3 expected=3 2024-09-16T05:08:52.216 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:c4fdafeb:::6:head 2024-09-16T05:08:52.216 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cursor()=9:c4fdafeb:::6:head expected=9:c4fdafeb:::6:head 2024-09-16T05:08:52.216 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:c4fdafeb:::6:head -> 6 2024-09-16T05:08:52.216 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=6 expected=6 2024-09-16T05:08:52.216 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:bd63b0f1:::8:head 2024-09-16T05:08:52.216 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cursor()=9:bd63b0f1:::8:head expected=9:bd63b0f1:::8:head 2024-09-16T05:08:52.216 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:bd63b0f1:::8:head -> 8 2024-09-16T05:08:52.216 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=8 expected=8 2024-09-16T05:08:52.216 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:b29083e3:::5:head 2024-09-16T05:08:52.216 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cursor()=9:b29083e3:::5:head expected=9:b29083e3:::5:head 2024-09-16T05:08:52.216 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:b29083e3:::5:head -> 5 2024-09-16T05:08:52.216 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=5 expected=5 2024-09-16T05:08:52.217 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:89d3ae78:::11:head 2024-09-16T05:08:52.521 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cur api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-09-16T05:08:52.521 INFO:tasks.workunit.client.0.smithi149.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-09-16T05:08:52.521 INFO:tasks.workunit.client.0.smithi149.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-09-16T05:08:52.521 INFO:tasks.workunit.client.0.smithi149.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-09-16T05:08:52.521 INFO:tasks.workunit.client.0.smithi149.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (3005 ms) 2024-09-16T05:08:52.521 INFO:tasks.workunit.client.0.smithi149.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (3005 ms total) 2024-09-16T05:08:52.521 INFO:tasks.workunit.client.0.smithi149.stdout: api_cls_remote_reads: 2024-09-16T05:08:52.522 INFO:tasks.workunit.client.0.smithi149.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-09-16T05:08:52.522 INFO:tasks.workunit.client.0.smithi149.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (3005 ms total) 2024-09-16T05:08:52.522 INFO:tasks.workunit.client.0.smithi149.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-09-16T05:08:52.547 INFO:tasks.workunit.client.0.smithi149.stdout: handler_error: Running main() from gmock_main.cc 2024-09-16T05:08:52.547 INFO:tasks.workunit.client.0.smithi149.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-09-16T05:08:52.547 INFO:tasks.workunit.client.0.smithi149.stdout: handler_error: [----------] Global test environment set-up. 2024-09-16T05:08:52.547 INFO:tasks.workunit.client.0.smithi149.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-09-16T05:08:52.547 INFO:tasks.workunit.client.0.smithi149.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-09-16T05:08:52.547 INFO:tasks.workunit.client.0.smithi149.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (2974 ms) 2024-09-16T05:08:52.547 INFO:tasks.workunit.client.0.smithi149.stdout: handler_error: [----------] 1 test from neocls_handler_error (2974 ms total) 2024-09-16T05:08:52.548 INFO:tasks.workunit.client.0.smithi149.stdout: handler_error: 2024-09-16T05:08:52.548 INFO:tasks.workunit.client.0.smithi149.stdout: handler_error: [----------] Global test environment tear-down 2024-09-16T05:08:52.548 INFO:tasks.workunit.client.0.smithi149.stdout: handler_error: [==========] 1 test from 1 test suite ran. (2974 ms total) 2024-09-16T05:08:52.548 INFO:tasks.workunit.client.0.smithi149.stdout: handler_error: [ PASSED ] 1 test. 2024-09-16T05:08:52.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-16T05:08:52.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-09-16T05:08:52.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-16T05:08:52.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-09-16T05:08:52.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-16T05:08:52.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-16T05:08:52.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-16T05:08:52.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (258 ms) 2024-09-16T05:08:52.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-16T05:08:52.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (15 ms) 2024-09-16T05:08:52.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-16T05:08:52.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (10 ms) 2024-09-16T05:08:52.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-16T05:08:52.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (67 ms) 2024-09-16T05:08:52.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-16T05:08:52.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (16 ms) 2024-09-16T05:08:52.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-16T05:08:52.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (10 ms) 2024-09-16T05:08:52.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-16T05:08:52.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (7 ms) 2024-09-16T05:08:52.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-16T05:08:52.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (11 ms) 2024-09-16T05:08:52.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-16T05:08:52.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (5 ms) 2024-09-16T05:08:52.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-16T05:08:52.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (9 ms) 2024-09-16T05:08:52.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-16T05:08:52.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (11 ms) 2024-09-16T05:08:52.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-09-16T05:08:52.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (12 ms) 2024-09-16T05:08:52.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-16T05:08:52.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (33 ms) 2024-09-16T05:08:52.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-16T05:08:52.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (36 ms) 2024-09-16T05:08:52.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-16T05:08:52.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (18 ms) 2024-09-16T05:08:52.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-16T05:08:52.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (7 ms) 2024-09-16T05:08:52.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (525 ms total) 2024-09-16T05:08:52.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: 2024-09-16T05:08:52.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-16T05:08:52.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (3050 ms total) 2024-09-16T05:08:52.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-09-16T05:08:52.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-16T05:08:52.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-16T05:08:52.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-16T05:08:52.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-16T05:08:52.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-16T05:08:52.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (143 ms) 2024-09-16T05:08:52.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-16T05:08:52.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (53 ms) 2024-09-16T05:08:52.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-16T05:08:52.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2897 ms) 2024-09-16T05:08:52.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-16T05:08:52.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.372199+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi149-30236-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T05:08:52.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.372451+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi149-29776-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.372750+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi149-29485-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.372986+0000 mon.a [INF] from='client.? 172.21.15.149:0/3985318423' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi149-29453-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.373183+0000 mon.a [INF] from='client.? 172.21.15.149:0/4240059448' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi149-29744-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.373341+0000 mon.a [INF] from='client.? 172.21.15.149:0/92817236' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi149-29490-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.373517+0000 mon.a [INF] from='client.? 172.21.15.149:0/666353614' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi149-29688-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.373667+0000 mon.a [INF] from='client.? 172.21.15.149:0/279885740' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi149-29922-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.373827+0000 mon.a [INF] from='client.? 172.21.15.149:0/2377866128' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi149-29944-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.373978+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi149-29631-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.374179+0000 mon.a [INF] from='client.? 172.21.15.149:0/198508757' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi149-29445-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.374376+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi149-29623-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.374574+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi149-29675-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.374794+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi149-29736-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.375032+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi149-29712-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.375254+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi149-29587-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.375465+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi149-29966-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.375679+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi149-29461-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.375907+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi149-29469-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.376129+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi149-29498-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:52.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.376383+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi149-29771-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T05:08:52.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.384583+0000 mon.b [INF] from='client.? 172.21.15.149:0/2121721883' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi149-30236-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi149-30236-1"}]: dispatch 2024-09-16T05:08:52.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.385462+0000 mon.b [INF] from='client.? 172.21.15.149:0/1799357224' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi149-29771-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi149-29771-1"}]: dispatch 2024-09-16T05:08:52.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.395560+0000 mon.b [INF] from='client.? 172.21.15.149:0/2145916445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi149-29787-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:08:52.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.441006+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi149-30236-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi149-30236-1"}]: dispatch 2024-09-16T05:08:52.913 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.455047+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi149-29771-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi149-29771-1"}]: dispatch 2024-09-16T05:08:53.061 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:51.456209+0000 mon.a [INF] from='client.? ' list_parallel: process_1_[29986]: starting. 2024-09-16T05:08:53.061 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_1_[29986]: creating pool ceph_test_rados_list_parallel.smithi149-29955 2024-09-16T05:08:53.061 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_1_[29986]: created object 0... 2024-09-16T05:08:53.061 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_1_[29986]: created object 25... 2024-09-16T05:08:53.061 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_1_[29986]: created object 49... 2024-09-16T05:08:53.062 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_1_[29986]: finishing. 2024-09-16T05:08:53.062 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_1_[29986]: shutting down. 2024-09-16T05:08:53.062 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_2_[29987]: starting. 2024-09-16T05:08:53.062 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_2_[29987]: listing objects. 2024-09-16T05:08:53.062 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_2_[29987]: listed object 0... 2024-09-16T05:08:53.062 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_2_[29987]: listed object 25... 2024-09-16T05:08:53.062 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_2_[29987]: saw 50 objects 2024-09-16T05:08:53.062 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_2_[29987]: shutting down. 2024-09-16T05:08:53.062 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.062 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_3_[30597]: starting. 2024-09-16T05:08:53.062 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_3_[30597]: creating pool ceph_test_rados_list_parallel.smithi149-29955 2024-09-16T05:08:53.062 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_3_[30597]: created object 0... 2024-09-16T05:08:53.062 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_3_[30597]: created object 25... 2024-09-16T05:08:53.062 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_3_[30597]: created object 49... 2024-09-16T05:08:53.063 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_3_[30597]: finishing. 2024-09-16T05:08:53.063 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_3_[30597]: shutting down. 2024-09-16T05:08:53.063 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.063 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_4_[30598]: starting. 2024-09-16T05:08:53.063 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_4_[30598]: listing objects. 2024-09-16T05:08:53.063 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_4_[30598]: listed object 0... 2024-09-16T05:08:53.063 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_4_[30598]: listed object 25... 2024-09-16T05:08:53.063 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_4_[30598]: saw 45 objects 2024-09-16T05:08:53.063 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_4_[30598]: shutting down. 2024-09-16T05:08:53.063 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.063 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_5_[30599]: starting. 2024-09-16T05:08:53.063 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_5_[30599]: removed 25 objects... 2024-09-16T05:08:53.063 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_5_[30599]: removed half of the objects 2024-09-16T05:08:53.064 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_5_[30599]: removed 50 objects... 2024-09-16T05:08:53.064 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_5_[30599]: removed 50 objects 2024-09-16T05:08:53.064 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_5_[30599]: shutting down. 2024-09-16T05:08:53.064 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.064 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.064 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_6_[30648]: starting. 2024-09-16T05:08:53.064 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_6_[30648]: creating pool ceph_test_rados_list_parallel.smithi149-29955 2024-09-16T05:08:53.064 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_6_[30648]: created object 0... 2024-09-16T05:08:53.064 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_6_[30648]: created object 25... 2024-09-16T05:08:53.064 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_6_[30648]: created object 49... 2024-09-16T05:08:53.064 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_6_[30648]: finishing. 2024-09-16T05:08:53.064 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_6_[30648]: shutting down. 2024-09-16T05:08:53.065 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.065 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.065 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_7_[30649]: starting. 2024-09-16T05:08:53.065 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_7_[30649]: listing objects. 2024-09-16T05:08:53.065 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_7_[30649]: listed object 0... 2024-09-16T05:08:53.065 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_7_[30649]: listed object 25... 2024-09-16T05:08:53.065 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_7_[30649]: listed object 50... 2024-09-16T05:08:53.065 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_7_[30649]: saw 52 objects 2024-09-16T05:08:53.065 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_7_[30649]: shutting down. 2024-09-16T05:08:53.065 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.065 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.065 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_8_[30650]: starting. 2024-09-16T05:08:53.065 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_8_[30650]: added 25 objects... 2024-09-16T05:08:53.065 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_8_[30650]: added half of the objects 2024-09-16T05:08:53.066 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_8_[30650]: added 50 objects... 2024-09-16T05:08:53.066 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_8_[30650]: added 50 objects 2024-09-16T05:08:53.066 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_8_[30650]: shutting down. 2024-09-16T05:08:53.066 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.066 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.402 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: **********************entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi149-29787-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:08:53.402 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.364131+0000 mon.b [INF] from='client.? 172.21.15.149:0/3222268231' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi149-29485-1","var":"pg_num","val":"11"}]: dispatch 2024-09-16T05:08:53.402 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.365463+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi149-29485-1","var":"pg_num","val":"11"}]: dispatch 2024-09-16T05:08:53.402 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.372129+0000 mon.a [WRN] Health check failed: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T05:08:53.402 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.390727+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi149-30197-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi149-30197-1"}]': finished 2024-09-16T05:08:53.403 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.391042+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi149-29787-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:53.403 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.391290+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi149-29485-1","var":"pg_num","val":"11"}]': finished 2024-09-16T05:08:53.403 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.561414+0000 mon.b [INF] from='client.? 172.21.15.149:0/3124130221' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi149-29623-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:08:53.403 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.562915+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi149-29623-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:08:53.403 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.650666+0000 mon.b [INF] from='client.? 172.21.15.149:0/355934775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi149-29736-7"}]: dispatch 2024-09-16T05:08:53.403 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.663572+0000 mon.b [INF] from='client.? 172.21.15.149:0/2238391616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi149-29712-7"}]: dispatch 2024-09-16T05:08:53.403 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.723324+0000 mon.b [INF] from='client.? 172.21.15.149:0/2900595159' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi149-29469-22"}]: dispatch 2024-09-16T05:08:53.403 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.726327+0000 mon.b [INF] from='client.? 172.21.15.149:0/1432834730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi149-29461-16"}]: dispatch 2024-09-16T05:08:53.403 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.753582+0000 client.admin [INF] onexx 2024-09-16T05:08:53.403 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.786451+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi149-29736-7"}]: dispatch 2024-09-16T05:08:53.403 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.786478+0000 mon.b [INF] from='client.? 172.21.15.149:0/355934775' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi149-29736-7"}]: dispatch 2024-09-16T05:08:53.403 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.911393+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi149-29712-7"}]: dispatch 2024-09-16T05:08:53.404 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.911861+0000 mon.a [INF] from='client.? 172.21.15.149:0/897142304' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-16T05:08:53.404 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.911875+0000 mon.b [INF] from='client.? 172.21.15.149:0/2238391616' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi149-29712-7"}]: dispatch 2024-09-16T05:08:53.404 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.947236+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi149-29469-22"}]: dispatch 2024-09-16T05:08:53.404 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.947270+0000 mon.b [INF] from='client.? 172.21.15.149:0/2900595159' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi149-29469-22"}]: dispatch 2024-09-16T05:08:53.404 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.989059+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi149-29461-16"}]: dispatch 2024-09-16T05:08:53.404 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.989124+0000 mon.b [INF] from='client.? 172.21.15.149:0/1432834730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi149-29461-16"}]: dispatch 2024-09-16T05:08:53.404 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.989320+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi149-29736-7"}]: dispatch 2024-09-16T05:08:53.404 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.989372+0000 mon.b [INF] from='client.? 172.21.15.149:0/355934775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi149-29736-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T05:08:53.404 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.989543+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi149-29712-7"}]: dispatch 2024-09-16T05:08:53.404 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.989563+0000 mon.b [INF] from='client.? 172.21.15.149:0/2238391616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi149-29712-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T05:08:53.404 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.989736+0000 mon.b [INF] from='client.? 172.21.15.149:0/2900595159' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi149-29469-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T05:08:53.404 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.989768+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi149-29469-22"}]: dispatch 2024-09-16T05:08:53.405 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.989898+0000 mon.b [INF] from='client.? 172.21.15.149:0/1432834730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi149-29461-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T05:08:53.405 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.990107+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi149-29461-16"}]: dispatch 2024-09-16T05:08:53.405 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.990458+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi149-29736-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T05:08:53.405 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.991161+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi149-29712-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T05:08:53.622 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.991457+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi149-29469-22", "profile": [ "k=********* 2024-09-16T05:08:53.623 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_9_[30993]: starting. 2024-09-16T05:08:53.623 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_9_[30993]: creating pool ceph_test_rados_list_parallel.smithi149-29955 2024-09-16T05:08:53.623 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_9_[30993]: created object 0... 2024-09-16T05:08:53.623 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_9_[30993]: created object 25... 2024-09-16T05:08:53.623 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_9_[30993]: created object 49... 2024-09-16T05:08:53.623 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_9_[30993]: finishing. 2024-09-16T05:08:53.623 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_9_[30993]: shutting down. 2024-09-16T05:08:53.623 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.623 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.623 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_10_[30994]: starting. 2024-09-16T05:08:53.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_10_[30994]: listing objects. 2024-09-16T05:08:53.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_10_[30994]: listed object 0... 2024-09-16T05:08:53.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_10_[30994]: listed object 25... 2024-09-16T05:08:53.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_10_[30994]: listed object 50... 2024-09-16T05:08:53.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_10_[30994]: listed object 75... 2024-09-16T05:08:53.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_10_[30994]: saw 100 objects 2024-09-16T05:08:53.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_10_[30994]: shutting down. 2024-09-16T05:08:53.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_13_[30997]: starting. 2024-09-16T05:08:53.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_13_[30997]: removed 25 objects... 2024-09-16T05:08:53.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_13_[30997]: removed half of the objects 2024-09-16T05:08:53.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_13_[30997]: removed 50 objects... 2024-09-16T05:08:53.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_13_[30997]: removed 50 objects 2024-09-16T05:08:53.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_13_[30997]: shutting down. 2024-09-16T05:08:53.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_12_[30996]: starting. 2024-09-16T05:08:53.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_12_[30996]: added 25 objects... 2024-09-16T05:08:53.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_12_[30996]: added half of the objects 2024-09-16T05:08:53.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_12_[30996]: added 50 objects... 2024-09-16T05:08:53.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_12_[30996]: added 50 objects 2024-09-16T05:08:53.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_12_[30996]: shutting down. 2024-09-16T05:08:53.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_11_[30995]: starting. 2024-09-16T05:08:53.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_11_[30995]: added 25 objects... 2024-09-16T05:08:53.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_11_[30995]: added half of the objects 2024-09-16T05:08:53.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_11_[30995]: added 50 objects... 2024-09-16T05:08:53.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_11_[30995]: added 50 objects 2024-09-16T05:08:53.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_11_[30995]: shutting down. 2024-09-16T05:08:53.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.627 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.627 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.627 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.627 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_14_[31078]: starting. 2024-09-16T05:08:53.627 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_14_[31078]: creating pool ceph_test_rados_list_parallel.smithi149-29955 2024-09-16T05:08:53.627 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_14_[31078]: created object 0... 2024-09-16T05:08:53.627 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_14_[31078]: created object 25... 2024-09-16T05:08:53.627 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_14_[31078]: created object 49... 2024-09-16T05:08:53.627 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_14_[31078]: finishing. 2024-09-16T05:08:53.627 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_14_[31078]: shutting down. 2024-09-16T05:08:53.627 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.627 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.627 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.627 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:53.628 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_15_[31079]: starting. 2024-09-16T05:08:53.628 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_15_[31079]: listing objects. 2024-09-16T05:08:53.628 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_15_[31079]: listed object 0... 2024-09-16T05:08:54.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_15_[31079]: listed object 25... 2024-09-16T05:08:54.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_15_[31079]: listed object 50... 2024-09-16T05:08:54.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_15_[31079]: listed object 75... 2024-09-16T05:08:54.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_15_[31079]: listed object 100... 2024-09-16T05:08:54.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_15_[31079]: listed object 125... 2024-09-16T05:08:54.624 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_15_[31079]: saw 150 objects 2024-09-16T05:08:54.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_15_[31079]: shutting down. 2024-09-16T05:08:54.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:54.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:54.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:54.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:54.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_16_[31080]: starting. 2024-09-16T05:08:54.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_16_[31080]: added 25 objects... 2024-09-16T05:08:54.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_16_[31080]: added half of the objects 2024-09-16T05:08:54.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_16_[31080]: added 50 objects... 2024-09-16T05:08:54.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_16_[31080]: added 50 objects 2024-09-16T05:08:54.625 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: process_16_[31080]: shutting down. 2024-09-16T05:08:54.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:54.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:54.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:54.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:54.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******************************* 2024-09-16T05:08:54.626 INFO:tasks.workunit.client.0.smithi149.stdout: list_parallel: ******* SUCCESS ********** 2024-09-16T05:08:54.626 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-09-16T05:08:54.626 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-16T05:08:54.626 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-16T05:08:54.626 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-16T05:08:54.626 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-16T05:08:54.626 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (434 ms) 2024-09-16T05:08:54.627 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-16T05:08:54.627 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (22 ms) 2024-09-16T05:08:54.627 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-16T05:08:54.627 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (4721 ms) 2024-09-16T05:08:54.627 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (5177 ms total) 2024-09-16T05:08:54.627 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd_pp: 2024-09-16T05:08:54.627 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-16T05:08:54.627 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (5177 ms total) 2024-09-16T05:08:54.627 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-16T05:08:55.489 INFO:tasks.workunit.client.0.smithi149.stdout: cls: Running main() from gmock_main.cc 2024-09-16T05:08:55.490 INFO:tasks.workunit.client.0.smithi149.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-09-16T05:08:55.490 INFO:tasks.workunit.client.0.smithi149.stdout: cls: [----------] Global test environment set-up. 2024-09-16T05:08:55.490 INFO:tasks.workunit.client.0.smithi149.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-09-16T05:08:55.490 INFO:tasks.workunit.client.0.smithi149.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-09-16T05:08:55.490 INFO:tasks.workunit.client.0.smithi149.stdout: cls: [ OK ] NeoRadosCls.DNE (2985 ms) 2024-09-16T05:08:55.490 INFO:tasks.workunit.client.0.smithi149.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-09-16T05:08:55.490 INFO:tasks.workunit.client.0.smithi149.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (2946 ms) 2024-09-16T05:08:55.490 INFO:tasks.workunit.client.0.smithi149.stdout: cls: [----------] 2 tests from NeoRadosCls (5931 ms total) 2024-09-16T05:08:55.490 INFO:tasks.workunit.client.0.smithi149.stdout: cls: 2024-09-16T05:08:55.490 INFO:tasks.workunit.client.0.smithi149.stdout: cls: [----------] Global test environment tear-down 2024-09-16T05:08:55.490 INFO:tasks.workunit.client.0.smithi149.stdout: cls: [==========] 2 tests from 1 test suite ran. (5931 ms total) 2024-09-16T05:08:55.490 INFO:tasks.workunit.client.0.smithi149.stdout: cls: [ PASSED ] 2 tests. 2024-09-16T05:08:55.500 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_1_[30021]: starting. 2024-09-16T05:08:55.500 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_1_[30021]: creating pool ceph_test_rados_delete_pools_parallel.smithi149-29989 2024-09-16T05:08:55.500 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_1_[30021]: created object 0... 2024-09-16T05:08:55.500 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_1_[30021]: created object 25... 2024-09-16T05:08:55.500 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_1_[30021]: created object 49... 2024-09-16T05:08:55.500 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_1_[30021]: finishing. 2024-09-16T05:08:55.500 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_1_[30021]: shutting down. 2024-09-16T05:08:55.500 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_2_[30022]: starting. 2024-09-16T05:08:55.500 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_2_[30022]: deleting pool ceph_test_rados_delete_pools_parallel.smithi149-29989 2024-09-16T05:08:55.500 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_2_[30022]: shutting down. 2024-09-16T05:08:55.500 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: ******************************* 2024-09-16T05:08:55.501 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_3_[30699]: starting. 2024-09-16T05:08:55.501 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_3_[30699]: creating pool ceph_test_rados_delete_pools_parallel.smithi149-29989 2024-09-16T05:08:55.501 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_3_[30699]: created object 0... 2024-09-16T05:08:55.501 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_3_[30699]: created object 25... 2024-09-16T05:08:55.501 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_3_[30699]: created object 49... 2024-09-16T05:08:55.501 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_3_[30699]: finishing. 2024-09-16T05:08:55.501 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_3_[30699]: shutting down. 2024-09-16T05:08:55.501 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: ******************************* 2024-09-16T05:08:55.501 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_5_[30701]: starting. 2024-09-16T05:08:55.501 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_5_[30701]: listing objects. 2024-09-16T05:08:55.501 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_5_[30701]: listed object 0... 2024-09-16T05:08:55.501 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_5_[30701]: listed object 25... 2024-09-16T05:08:55.501 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_5_[30701]: saw 50 objects 2024-09-16T05:08:55.501 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_5_[30701]: shutting down. 2024-09-16T05:08:55.501 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: ******************************* 2024-09-16T05:08:55.502 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_4_[30700]: starting. 2024-09-16T05:08:55.502 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_4_[30700]: deleting pool ceph_test_rados_delete_pools_parallel.smithi149-29989 2024-09-16T05:08:55.502 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: process_4_[30700]: shutting down. 2024-09-16T05:08:55.502 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: ******************************* 2024-09-16T05:08:55.502 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: ******************************* 2024-09-16T05:08:55.502 INFO:tasks.workunit.client.0.smithi149.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-16T05:08:55.502 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_1_[29993]: starting. 2024-09-16T05:08:55.502 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_1_[29993]: creating pool ceph_test_rados_open_pools_parallel.smithi149-29983 2024-09-16T05:08:55.502 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_1_[29993]: created object 0... 2024-09-16T05:08:55.503 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_1_[29993]: created object 25... 2024-09-16T05:08:55.503 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_1_[29993]: created object 49... 2024-09-16T05:08:55.503 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_1_[29993]: finishing. 2024-09-16T05:08:55.503 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_1_[29993]: shutting down. 2024-09-16T05:08:55.503 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_2_[29997]: starting. 2024-09-16T05:08:55.503 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_2_[29997]: rados_pool_create. 2024-09-16T05:08:55.503 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_2_[29997]: rados_ioctx_create. 2024-09-16T05:08:55.503 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_2_[29997]: shutting down. 2024-09-16T05:08:55.503 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: ******************************* 2024-09-16T05:08:55.503 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_3_[30702]: starting. 2024-09-16T05:08:55.503 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_3_[30702]: creating pool ceph_test_rados_open_pools_parallel.smithi149-29983 2024-09-16T05:08:55.503 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_3_[30702]: created object 0... 2024-09-16T05:08:55.503 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_3_[30702]: created object 25... 2024-09-16T05:08:55.504 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_3_[30702]: created object 49... 2024-09-16T05:08:55.504 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_3_[30702]: finishing. 2024-09-16T05:08:55.504 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_3_[30702]: shutting down. 2024-09-16T05:08:55.504 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: ******************************* 2024-09-16T05:08:55.504 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_4_[30703]: starting. 2024-09-16T05:08:55.504 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_4_[30703]: rados_pool_create. 2024-09-16T05:08:55.504 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_4_[30703]: rados_ioctx_create. 2024-09-16T05:08:55.504 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: process_4_[30703]: shutting down. 2024-09-16T05:08:55.504 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: ******************************* 2024-09-16T05:08:55.504 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: ******************************* 2024-09-16T05:08:55.504 INFO:tasks.workunit.client.0.smithi149.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-16T05:08:56.595 INFO:tasks.workunit.client.0.smithi149.stdout: cmd: Running main() from gmock_main.cc 2024-09-16T05:08:56.595 INFO:tasks.workunit.client.0.smithi149.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-09-16T05:08:56.595 INFO:tasks.workunit.client.0.smithi149.stdout: cmd: [----------] Global test environment set-up. 2024-09-16T05:08:56.595 INFO:tasks.workunit.client.0.smithi149.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-09-16T05:08:56.595 INFO:tasks.workunit.client.0.smithi149.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-09-16T05:08:56.596 INFO:tasks.workunit.client.0.smithi149.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (1897 ms) 2024-09-16T05:08:56.596 INFO:tasks.workunit.client.0.smithi149.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-09-16T05:08:56.596 INFO:tasks.workunit.client.0.smithi149.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (1987 ms) 2024-09-16T05:08:56.596 INFO:tasks.workunit.client.0.smithi149.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-09-16T05:08:56.596 INFO:tasks.workunit.client.0.smithi149.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (3159 ms) 2024-09-16T05:08:56.596 INFO:tasks.workunit.client.0.smithi149.stdout: cmd: [----------] 3 tests from NeoRadosCmd (7043 ms total) 2024-09-16T05:08:56.596 INFO:tasks.workunit.client.0.smithi149.stdout: cmd: 2024-09-16T05:08:56.596 INFO:tasks.workunit.client.0.smithi149.stdout: cmd: [----------] Global test environment tear-down 2024-09-16T05:08:56.596 INFO:tasks.workunit.client.0.smithi149.stdout: cmd: [==========] 3 tests from 1 test suite ran. (7044 ms total) 2024-09-16T05:08:56.596 INFO:tasks.workunit.client.0.smithi149.stdout: cmd: [ PASSED ] 3 tests. 2024-09-16T05:08:56.722 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: Running main() from gmock_main.cc 2024-09-16T05:08:56.722 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-09-16T05:08:56.722 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-16T05:08:56.722 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-09-16T05:08:56.722 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: seed 29469 2024-09-16T05:08:56.722 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-16T05:08:56.722 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-09-16T05:08:56.722 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-16T05:08:56.722 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (290 ms) 2024-09-16T05:08:56.722 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-16T05:08:56.722 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (10 ms) 2024-09-16T05:08:56.723 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-16T05:08:56.723 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (33 ms) 2024-09-16T05:08:56.723 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-16T05:08:56.723 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (6 ms) 2024-09-16T05:08:56.723 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-16T05:08:56.723 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (8 ms) 2024-09-16T05:08:56.723 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-16T05:08:56.723 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (7 ms) 2024-09-16T05:08:56.723 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-16T05:08:56.723 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (5 ms) 2024-09-16T05:08:56.723 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-16T05:08:56.723 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (15 ms) 2024-09-16T05:08:56.723 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-16T05:08:56.724 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (6 ms) 2024-09-16T05:08:56.724 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-16T05:08:56.724 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (6 ms) 2024-09-16T05:08:56.724 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-16T05:08:56.724 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (11 ms) 2024-09-16T05:08:56.724 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-16T05:08:56.724 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (10 ms) 2024-09-16T05:08:56.724 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-16T05:08:56.724 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (9 ms) 2024-09-16T05:08:56.724 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-16T05:08:56.724 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (9 ms) 2024-09-16T05:08:56.724 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-16T05:08:56.724 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (27 ms) 2024-09-16T05:08:56.724 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-16T05:08:56.725 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (15 ms) 2024-09-16T05:08:56.725 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-16T05:08:56.725 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (6 ms) 2024-09-16T05:08:56.725 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-16T05:08:56.725 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (8 ms) 2024-09-16T05:08:56.725 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-16T05:08:56.725 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (11 ms) 2024-09-16T05:08:56.725 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (493 ms total) 2024-09-16T05:08:56.725 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: 2024-09-16T05:08:56.725 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-09-16T05:08:56.725 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-16T05:08:56.725 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1162 ms) 2024-09-16T05:08:56.725 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-16T05:08:56.726 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (37 ms) 2024-09-16T05:08:56.726 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-16T05:08:56.726 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (21 ms) 2024-09-16T05:08:56.726 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-16T05:08:56.726 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (7 ms) 2024-09-16T05:08:56.726 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-16T05:08:56.726 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (5 ms) 2024-09-16T05:08:57.406 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T05:08:57.406 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:52.991755+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi149-29461-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T05:08:57.406 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.405386+0000 mon.a [INF] from='client.? 172.21.15.149:0/897142304' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-16T05:08:57.406 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.437370+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi149-29736-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi149-29736-7"}]': finished 2024-09-16T05:08:57.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.437624+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi149-29712-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi149-29712-7"}]': finished 2024-09-16T05:08:57.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.437835+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi149-29461-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi149-29461-16"}]': finished 2024-09-16T05:08:57.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.438102+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi149-29469-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi149-29469-22"}]': finished 2024-09-16T05:08:57.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.438301+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi149-30197-1"}]': finished 2024-09-16T05:08:57.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.438490+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi149-29623-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:57.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.438660+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi149-29490-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T05:08:57.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.438823+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi149-29498-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T05:08:57.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.461883+0000 mon.b [INF] from='client.? 172.21.15.149:0/916294086' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi149-29498-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi149-29498-10"}]: dispatch 2024-09-16T05:08:57.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.465120+0000 mon.b [INF] from='client.? 172.21.15.149:0/2220205956' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi149-29490-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi149-29490-10"}]: dispatch 2024-09-16T05:08:57.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.465982+0000 mon.b [INF] from='client.? 172.21.15.149:0/1068055236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi149-29453-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:08:57.491 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.487087+0000 mon.b [INF] from='client.? 172.21.15.149:0/1799357224' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi149-29771-1"}]: dispatch 2024-09-16T05:08:57.491 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.487777+0000 mon.b [INF] from='client.? 172.21.15.149:0/2121721883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi149-30236-1"}]: dispatch 2024-09-16T05:08:57.491 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.491205+0000 mon.b [INF] from='client.? 172.21.15.149:0/3124130221' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi149-29623-4", "tierpool": "test-rados-api-smithi149-29623-4-cache"}]: dispatch 2024-09-16T05:08:57.491 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.492633+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi149-29498-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi149-29498-10"}]: dispatch 2024-09-16T05:08:57.491 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.493633+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi149-29490-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi149-29490-10"}]: dispatch 2024-09-16T05:08:57.491 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.494492+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi149-29453-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:08:57.491 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.521698+0000 mon.b [INF] from='client.? 172.21.15.149:0/1239933802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi149-30197-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T05:08:57.491 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.563905+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi149-29771-1"}]: dispatch 2024-09-16T05:08:57.491 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.564368+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi149-30236-1"}]: dispatch 2024-09-16T05:08:57.491 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.565222+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi149-29623-4", "tierpool": "test-rados-api-smithi149-29623-4-cache"}]: dispatch 2024-09-16T05:08:57.491 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:55.566425+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi149-30197-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T05:08:57.492 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.480373+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi149-29453-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T05:08:57.492 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.480660+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi149-29771-1"}]': finished 2024-09-16T05:08:57.492 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.480892+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi149-30236-1"}]': finished 2024-09-16T05:08:57.492 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.481105+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi149-29623-4", "tierpool": "test-rados-api-smithi149-29623-4-cache"}]': finished 2024-09-16T05:08:57.492 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.481240+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi149-30197-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T05:08:57.492 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.491051+0000 mon.b [INF] from='client.? 172.21.15.149:0/1799357224' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi149-29771-1"}]: dispatch 2024-09-16T05:08:57.492 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.492083+0000 mon.b [INF] from='client.? 172.21.15.149:0/1239933802' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi149-30197-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi149-30197-2"}]: dispatch 2024-09-16T05:08:57.492 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.492735+0000 mon.b [INF] from='client.? 172.21.15.149:0/1068055236' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi149-29453-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-16T05:08:57.492 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.501992+0000 mon.b [INF] from='client.? 172.21.15.149:0/2121721883' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi149-30236-1"}]: dispatch 2024-09-16T05:08:57.492 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.502626+0000 mon.b [INF] from='client.? 172.21.15.149:0/3124130221' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi149-29623-4", "overlaypool": "test-rados-api-smithi149-29623-4-cache"}]: dispatch 2024-09-16T05:08:57.492 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.581111+0000 mon.b [INF] from='client.? 172.21.15.149:0/1492780092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi149-29631-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:08:57.492 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.581317+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi149-29771-1"}]: dispatch 2024-09-16T05:08:57.492 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.581796+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi149-30197-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi149-30197-2"}]: dispatch 2024-09-16T05:08:57.493 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.582401+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi149-29453-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-16T05:08:57.493 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.582666+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi149-30236-1"}]: dispatch 2024-09-16T05:08:57.493 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.582980+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi149-29623-4", "overlaypool": "test-rados-api-smithi149-29623-4-cache"}]: dispatch 2024-09-16T05:08:57.493 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.588678+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi149-29631-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:08:57.493 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.596787+0000 mon.b [INF] from='client.? 172.21.15.149:0/3253034931' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi149-29445-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:08:57.493 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.597201+0000 mon.b [INF] from='client.? 172.21.15.149:0/61074850' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi149-29922-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:08:57.493 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.600612+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi149-29445-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:08:59.549 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.602059+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool" api_io: Running main() from gmock_main.cc 2024-09-16T05:08:59.549 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-16T05:08:59.549 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [----------] Global test environment set-up. 2024-09-16T05:08:59.550 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-16T05:08:59.550 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-16T05:08:59.550 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (303 ms) 2024-09-16T05:08:59.550 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-16T05:08:59.550 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-16T05:08:59.550 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-16T05:08:59.550 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: no timeout :/ 2024-09-16T05:08:59.550 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: no timeout :/ 2024-09-16T05:08:59.550 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: no timeout :/ 2024-09-16T05:08:59.550 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: no timeout :/ 2024-09-16T05:08:59.550 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: no timeout :/ 2024-09-16T05:08:59.550 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (46 ms) 2024-09-16T05:08:59.550 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-16T05:08:59.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (15 ms) 2024-09-16T05:08:59.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-16T05:08:59.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIo.Checksum (207 ms) 2024-09-16T05:08:59.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-16T05:08:59.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (6 ms) 2024-09-16T05:08:59.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-16T05:08:59.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (5 ms) 2024-09-16T05:08:59.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-16T05:08:59.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (7 ms) 2024-09-16T05:08:59.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-16T05:08:59.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (6 ms) 2024-09-16T05:08:59.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-16T05:08:59.551 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIo.TruncTest (11 ms) 2024-09-16T05:08:59.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-16T05:08:59.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (8 ms) 2024-09-16T05:08:59.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-16T05:08:59.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (12 ms) 2024-09-16T05:08:59.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-16T05:08:59.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIo.RmXattr (27 ms) 2024-09-16T05:08:59.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-16T05:08:59.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIo.XattrIter (12 ms) 2024-09-16T05:08:59.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [----------] 14 tests from LibRadosIo (665 ms total) 2024-09-16T05:08:59.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: 2024-09-16T05:08:59.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-16T05:08:59.552 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-16T05:08:59.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1180 ms) 2024-09-16T05:08:59.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-16T05:08:59.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (21 ms) 2024-09-16T05:08:59.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-16T05:08:59.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (22 ms) 2024-09-16T05:08:59.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-16T05:08:59.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (9 ms) 2024-09-16T05:08:59.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-16T05:08:59.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (26 ms) 2024-09-16T05:08:59.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-16T05:08:59.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (10 ms) 2024-09-16T05:08:59.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-16T05:08:59.553 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (4 ms) 2024-09-16T05:08:59.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-16T05:08:59.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (6 ms) 2024-09-16T05:08:59.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-16T05:08:59.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (37 ms) 2024-09-16T05:08:59.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-16T05:08:59.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (16 ms) 2024-09-16T05:08:59.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1331 ms total) 2024-09-16T05:08:59.554 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: 2024-09-16T05:08:59.559 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [----------] Global test environment tear-down 2024-09-16T05:08:59.559 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [==========] 24 tests from 2 test suites ran. (10253 ms total) 2024-09-16T05:08:59.559 INFO:tasks.workunit.client.0.smithi149.stdout: api_io: [ PASSED ] 24 tests. 2024-09-16T05:08:59.569 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-16T05:08:59.569 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-16T05:08:59.569 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-16T05:08:59.570 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-16T05:08:59.570 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: seed 29736 2024-09-16T05:08:59.570 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-16T05:08:59.570 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (250 ms) 2024-09-16T05:08:59.570 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-16T05:08:59.570 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (15 ms) 2024-09-16T05:08:59.570 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-16T05:08:59.570 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-09-16T05:08:59.570 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-16T05:08:59.570 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (69 ms) 2024-09-16T05:08:59.570 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-16T05:08:59.570 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (217 ms) 2024-09-16T05:08:59.571 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (552 ms total) 2024-09-16T05:08:59.571 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: 2024-09-16T05:08:59.571 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-16T05:08:59.571 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-16T05:08:59.571 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1174 ms) 2024-09-16T05:08:59.571 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-16T05:08:59.571 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1 ms) 2024-09-16T05:08:59.571 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-16T05:08:59.571 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (13 ms) 2024-09-16T05:08:59.571 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-16T05:08:59.571 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (26 ms) 2024-09-16T05:08:59.571 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1214 ms total) 2024-09-16T05:08:59.571 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: 2024-09-16T05:08:59.571 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-16T05:08:59.571 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (10134 ms total) 2024-09-16T05:08:59.572 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-16T05:08:59.577 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: Running main() from gmock_main.cc 2024-09-16T05:08:59.577 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-09-16T05:08:59.577 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [----------] Global test environment set-up. 2024-09-16T05:08:59.577 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-09-16T05:08:59.577 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-16T05:08:59.577 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ OK ] LibRadosStat.Stat (287 ms) 2024-09-16T05:08:59.577 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-09-16T05:08:59.577 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (35 ms) 2024-09-16T05:08:59.577 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-16T05:08:59.577 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ OK ] LibRadosStat.StatNS (17 ms) 2024-09-16T05:08:59.577 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-16T05:08:59.577 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-09-16T05:08:59.578 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-16T05:08:59.578 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (3 ms) 2024-09-16T05:08:59.578 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [----------] 5 tests from LibRadosStat (342 ms total) 2024-09-16T05:08:59.578 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: 2024-09-16T05:08:59.578 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-16T05:08:59.578 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-16T05:08:59.578 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1176 ms) 2024-09-16T05:08:59.578 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-16T05:08:59.578 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (38 ms) 2024-09-16T05:08:59.578 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-16T05:08:59.578 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-09-16T05:08:59.578 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-16T05:08:59.578 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (6 ms) 2024-09-16T05:08:59.578 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1220 ms total) 2024-09-16T05:08:59.579 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: 2024-09-16T05:08:59.579 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [----------] Global test environment tear-down 2024-09-16T05:08:59.579 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (10143 ms total) 2024-09-16T05:08:59.579 INFO:tasks.workunit.client.0.smithi149.stdout: api_stat: [ PASSED ] 9 tests. 2024-09-16T05:08:59.591 INFO:tasks.workunit.client.0.smithi149.stdout: ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-16T05:08:59.591 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (20 ms) 2024-09-16T05:08:59.591 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-16T05:08:59.591 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (5 ms) 2024-09-16T05:08:59.591 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-16T05:08:59.591 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (6 ms) 2024-09-16T05:08:59.591 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-16T05:08:59.591 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (8 ms) 2024-09-16T05:08:59.591 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-16T05:08:59.591 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (8 ms) 2024-09-16T05:08:59.591 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-16T05:08:59.592 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (9 ms) 2024-09-16T05:08:59.592 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-16T05:08:59.592 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (9 ms) 2024-09-16T05:08:59.592 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-16T05:08:59.592 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (35 ms) 2024-09-16T05:08:59.592 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-16T05:08:59.593 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (7 ms) 2024-09-16T05:08:59.593 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-16T05:08:59.593 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (9 ms) 2024-09-16T05:08:59.593 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-16T05:08:59.593 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (8 ms) 2024-09-16T05:08:59.593 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-16T05:08:59.593 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (11 ms) 2024-09-16T05:08:59.593 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1367 ms total) 2024-09-16T05:08:59.593 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: 2024-09-16T05:08:59.593 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-16T05:08:59.593 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (10246 ms total) 2024-09-16T05:08:59.593 INFO:tasks.workunit.client.0.smithi149.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-09-16T05:09:00.565 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: Running main() from gmock_main.cc 2024-09-16T05:09:00.565 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-09-16T05:09:00.565 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: [----------] Global test environment set-up. 2024-09-16T05:09:00.565 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-09-16T05:09:00.565 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-09-16T05:09:00.565 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: handle_notify cookie 94489310923264 notify_id 94489280512 notifier_gid 4630 2024-09-16T05:09:00.565 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (3782 ms) 2024-09-16T05:09:00.565 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-09-16T05:09:00.566 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: Trying... 2024-09-16T05:09:00.566 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: handle_notify cookie 94489324159440 notify_id 107374182400 notifier_gid 4885 2024-09-16T05:09:00.566 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: Waiting for 3.000000000s 2024-09-16T05:09:00.566 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: Timed out. 2024-09-16T05:09:00.566 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: Flushing... 2024-09-16T05:09:00.566 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: Flushed... 2024-09-16T05:09:00.566 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7127 ms) 2024-09-16T05:09:00.566 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (10909 ms total) 2024-09-16T05:09:00.566 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: 2024-09-16T05:09:00.566 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: [----------] Global test environment tear-down 2024-09-16T05:09:00.566 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (10909 ms total) 2024-09-16T05:09:00.567 INFO:tasks.workunit.client.0.smithi149.stdout: watch_notify: [ PASSED ] 2 tests. 2024-09-16T05:09:00.567 INFO:tasks.workunit.client.0.smithi149.stdout: list: Running main() from gmock_main.cc 2024-09-16T05:09:00.567 INFO:tasks.workunit.client.0.smithi149.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-09-16T05:09:00.567 INFO:tasks.workunit.client.0.smithi149.stdout: list: [----------] Global test environment set-up. 2024-09-16T05:09:00.567 INFO:tasks.workunit.client.0.smithi149.stdout: list: [----------] 3 tests from NeoradosList 2024-09-16T05:09:00.567 INFO:tasks.workunit.client.0.smithi149.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-09-16T05:09:00.567 INFO:tasks.workunit.client.0.smithi149.stdout: list: [ OK ] NeoradosList.ListObjects (3828 ms) 2024-09-16T05:09:00.567 INFO:tasks.workunit.client.0.smithi149.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-09-16T05:09:00.567 INFO:tasks.workunit.client.0.smithi149.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3157 ms) 2024-09-16T05:09:00.567 INFO:tasks.workunit.client.0.smithi149.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-09-16T05:09:00.568 INFO:tasks.workunit.client.0.smithi149.stdout: list: [ OK ] NeoradosList.ListObjectsMany (3966 ms) 2024-09-16T05:09:00.568 INFO:tasks.workunit.client.0.smithi149.stdout: list: [----------] 3 tests from NeoradosList (10951 ms total) 2024-09-16T05:09:00.568 INFO:tasks.workunit.client.0.smithi149.stdout: list: 2024-09-16T05:09:00.568 INFO:tasks.workunit.client.0.smithi149.stdout: list: [----------] Global test environment tear-down 2024-09-16T05:09:00.568 INFO:tasks.workunit.client.0.smithi149.stdout: list: [==========] 3 tests from 1 test suite ran. (10951 ms total) 2024-09-16T05:09:00.568 INFO:tasks.workunit.client.0.smithi149.stdout: list: [ PASSED ] 3 tests. 2024-09-16T05:09:01.501 INFO:tasks.workunit.client.0.smithi149.stdout:: "test-rados-api-smithi149-29922-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:09:01.501 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.733909+0000 mon.b [INF] from='client.? 172.21.15.149:0/3648262308' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.0"}]: dispatch 2024-09-16T05:09:01.501 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.734725+0000 mon.b [INF] from='client.? 172.21.15.149:0/3648262308' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.1"}]: dispatch 2024-09-16T05:09:01.502 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.735286+0000 mon.b [INF] from='client.? 172.21.15.149:0/3648262308' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.2"}]: dispatch 2024-09-16T05:09:01.502 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.735805+0000 mon.b [INF] from='client.? 172.21.15.149:0/3648262308' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.3"}]: dispatch 2024-09-16T05:09:01.502 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.736342+0000 mon.b [INF] from='client.? 172.21.15.149:0/3648262308' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.4"}]: dispatch 2024-09-16T05:09:01.502 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.737082+0000 mon.b [INF] from='client.? 172.21.15.149:0/3648262308' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.5"}]: dispatch 2024-09-16T05:09:01.502 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.737759+0000 mon.b [INF] from='client.? 172.21.15.149:0/3648262308' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.6"}]: dispatch 2024-09-16T05:09:01.502 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.738238+0000 mon.b [INF] from='client.? 172.21.15.149:0/3648262308' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.7"}]: dispatch 2024-09-16T05:09:01.502 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.738663+0000 mon.b [INF] from='client.? 172.21.15.149:0/3648262308' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.8"}]: dispatch 2024-09-16T05:09:01.502 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:56.739218+0000 mon.b [INF] from='client.? 172.21.15.149:0/3648262308' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.9"}]: dispatch 2024-09-16T05:09:01.502 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:57.033388+0000 mon.a [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-16T05:09:01.502 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:57.406401+0000 client.admin [INF] threexx 2024-09-16T05:09:01.502 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: got: 2024-09-16T05:08:57.407048+0000 mon.a [INF] from='client.? 172.21.15.149:0/897142304' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-16T05:09:01.502 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (8965 ms) 2024-09-16T05:09:01.503 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (12058 ms total) 2024-09-16T05:09:01.503 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: 2024-09-16T05:09:01.503 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-16T05:09:01.503 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (12058 ms total) 2024-09-16T05:09:01.503 INFO:tasks.workunit.client.0.smithi149.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-16T05:09:02.056 INFO:tasks.workunit.client.0.smithi149.stdout: ec_io: Running main() from gmock_main.cc 2024-09-16T05:09:02.056 INFO:tasks.workunit.client.0.smithi149.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-09-16T05:09:02.056 INFO:tasks.workunit.client.0.smithi149.stdout: ec_io: [----------] Global test environment set-up. 2024-09-16T05:09:02.056 INFO:tasks.workunit.client.0.smithi149.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-09-16T05:09:02.056 INFO:tasks.workunit.client.0.smithi149.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-09-16T05:09:02.056 INFO:tasks.workunit.client.0.smithi149.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (5856 ms) 2024-09-16T05:09:02.057 INFO:tasks.workunit.client.0.smithi149.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-09-16T05:09:02.057 INFO:tasks.workunit.client.0.smithi149.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (6598 ms) 2024-09-16T05:09:02.057 INFO:tasks.workunit.client.0.smithi149.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (12454 ms total) 2024-09-16T05:09:02.057 INFO:tasks.workunit.client.0.smithi149.stdout: ec_io: 2024-09-16T05:09:02.057 INFO:tasks.workunit.client.0.smithi149.stdout: ec_io: [----------] Global test environment tear-down 2024-09-16T05:09:02.057 INFO:tasks.workunit.client.0.smithi149.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (12454 ms total) 2024-09-16T05:09:02.057 INFO:tasks.workunit.client.0.smithi149.stdout: ec_io: [ PASSED ] 2 tests. 2024-09-16T05:09:03.092 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-09-16T05:09:03.092 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-16T05:09:03.092 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-16T05:09:03.092 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-16T05:09:03.092 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: seed 29498 2024-09-16T05:09:03.092 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-16T05:09:03.092 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (244 ms) 2024-09-16T05:09:03.092 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-16T05:09:03.093 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (84 ms) 2024-09-16T05:09:03.093 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-16T05:09:03.093 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1026 ms) 2024-09-16T05:09:03.093 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-16T05:09:03.093 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1005 ms) 2024-09-16T05:09:03.093 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-16T05:09:03.093 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (5 ms) 2024-09-16T05:09:03.093 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-16T05:09:03.093 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (5 ms) 2024-09-16T05:09:03.093 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-16T05:09:03.093 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (9 ms) 2024-09-16T05:09:03.093 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-16T05:09:03.093 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (8 ms) 2024-09-16T05:09:03.094 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2386 ms total) 2024-09-16T05:09:03.094 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: 2024-09-16T05:09:03.094 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-16T05:09:03.094 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-16T05:09:03.094 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1125 ms) 2024-09-16T05:09:03.094 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-16T05:09:03.094 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (18 ms) 2024-09-16T05:09:03.094 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-16T05:09:03.094 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1092 ms) 2024-09-16T05:09:03.094 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-16T05:09:03.094 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1040 ms) 2024-09-16T05:09:03.094 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-16T05:09:03.094 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (11 ms) 2024-09-16T05:09:03.095 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-16T05:09:03.095 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (13 ms) 2024-09-16T05:09:03.095 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-16T05:09:03.095 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (12 ms) 2024-09-16T05:09:03.095 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-16T05:09:03.095 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (9 ms) 2024-09-16T05:09:03.095 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3320 ms total) 2024-09-16T05:09:03.095 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: 2024-09-16T05:09:03.095 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-16T05:09:03.095 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (13761 ms total) 2024-09-16T05:09:03.095 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-16T05:09:03.101 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: Running main() from gmock_main.cc 2024-09-16T05:09:03.102 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-16T05:09:03.102 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [----------] Global test environment set-up. 2024-09-16T05:09:03.102 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-16T05:09:03.102 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-16T05:09:03.102 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (270 ms) 2024-09-16T05:09:03.102 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-16T05:09:03.102 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLock.LockShared (12 ms) 2024-09-16T05:09:03.102 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-16T05:09:03.102 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1010 ms) 2024-09-16T05:09:03.102 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-16T05:09:03.102 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1011 ms) 2024-09-16T05:09:03.102 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-16T05:09:03.102 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (7 ms) 2024-09-16T05:09:03.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-16T05:09:03.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLock.Unlock (9 ms) 2024-09-16T05:09:03.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-16T05:09:03.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (14 ms) 2024-09-16T05:09:03.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-16T05:09:03.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (4 ms) 2024-09-16T05:09:03.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [----------] 8 tests from LibRadosLock (2337 ms total) 2024-09-16T05:09:03.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: 2024-09-16T05:09:03.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-16T05:09:03.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-16T05:09:03.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1152 ms) 2024-09-16T05:09:03.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-16T05:09:03.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (13 ms) 2024-09-16T05:09:03.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-16T05:09:03.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1088 ms) 2024-09-16T05:09:03.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-16T05:09:03.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1043 ms) 2024-09-16T05:09:03.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-16T05:09:03.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (13 ms) 2024-09-16T05:09:03.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-16T05:09:03.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (11 ms) 2024-09-16T05:09:03.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-16T05:09:03.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (14 ms) 2024-09-16T05:09:03.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-16T05:09:03.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (10 ms) 2024-09-16T05:09:03.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3344 ms total) 2024-09-16T05:09:03.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: 2024-09-16T05:09:03.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [----------] Global test environment tear-down 2024-09-16T05:09:03.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (13774 ms total) 2024-09-16T05:09:03.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-16T05:09:04.083 INFO:tasks.workunit.client.0.smithi149.stdout: pool: Running main() from gmock_main.cc 2024-09-16T05:09:04.083 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-09-16T05:09:04.083 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [----------] Global test environment set-up. 2024-09-16T05:09:04.083 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-09-16T05:09:04.083 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-09-16T05:09:04.083 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [ OK ] NeoRadosPools.PoolList (2929 ms) 2024-09-16T05:09:04.083 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-09-16T05:09:04.084 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (2086 ms) 2024-09-16T05:09:04.084 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-09-16T05:09:04.084 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (1954 ms) 2024-09-16T05:09:04.084 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-09-16T05:09:04.084 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (3969 ms) 2024-09-16T05:09:04.084 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-09-16T05:09:04.084 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (1512 ms) 2024-09-16T05:09:04.084 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-09-16T05:09:04.084 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2014 ms) 2024-09-16T05:09:04.084 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [----------] 6 tests from NeoRadosPools (14465 ms total) 2024-09-16T05:09:04.084 INFO:tasks.workunit.client.0.smithi149.stdout: pool: 2024-09-16T05:09:04.084 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [----------] Global test environment tear-down 2024-09-16T05:09:04.084 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [==========] 6 tests from 1 test suite ran. (14465 ms total) 2024-09-16T05:09:04.085 INFO:tasks.workunit.client.0.smithi149.stdout: pool: [ PASSED ] 6 tests. 2024-09-16T05:09:06.287 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-16T05:09:06.287 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-16T05:09:06.287 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-16T05:09:06.287 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-16T05:09:06.287 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-16T05:09:06.287 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: notify 2024-09-16T05:09:06.287 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1326 ms) 2024-09-16T05:09:06.287 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-16T05:09:06.288 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (17 ms) 2024-09-16T05:09:06.288 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1343 ms total) 2024-09-16T05:09:06.288 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: 2024-09-16T05:09:06.288 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-16T05:09:06.288 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-16T05:09:06.288 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: notify 2024-09-16T05:09:06.288 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (142 ms) 2024-09-16T05:09:06.288 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-16T05:09:06.288 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: notify 2024-09-16T05:09:06.288 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3529 ms) 2024-09-16T05:09:06.288 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-16T05:09:06.288 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (10 ms) 2024-09-16T05:09:06.288 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-16T05:09:06.289 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (10 ms) 2024-09-16T05:09:06.289 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-16T05:09:06.289 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: handle_notify cookie 94401556118912 notify_id 141733920768 notifier_gid 5146 2024-09-16T05:09:06.289 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (8 ms) 2024-09-16T05:09:06.289 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-16T05:09:06.289 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: handle_notify cookie 94401556118912 notify_id 141733920772 notifier_gid 5146 2024-09-16T05:09:06.289 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (6 ms) 2024-09-16T05:09:06.289 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-16T05:09:06.289 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: handle_notify cookie 94401556118912 notify_id 141733920772 notifier_gid 5146 2024-09-16T05:09:06.289 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (6 ms) 2024-09-16T05:09:06.289 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-16T05:09:06.289 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: handle_notify cookie 94401556118912 notify_id 141733920773 notifier_gid 5146 2024-09-16T05:09:06.289 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (7 ms) 2024-09-16T05:09:06.290 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-16T05:09:06.290 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: handle_notify cookie 94401556118912 notify_id 141733920769 notifier_gid 5146 2024-09-16T05:09:06.290 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (10 ms) 2024-09-16T05:09:06.290 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-16T05:09:06.290 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: handle_notify cookie 94401556106960 notify_id 141733920770 notifier_gid 5146 2024-09-16T05:09:06.290 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (8 ms) 2024-09-16T05:09:06.290 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-16T05:09:06.290 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: trying... 2024-09-16T05:09:06.290 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: handle_notify cookie 94401556106960 notify_id 141733920771 notifier_gid 5146 2024-09-16T05:09:06.290 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: timed out 2024-09-16T05:09:06.290 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: flushing 2024-09-16T05:09:10.162 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_n api_watch_notify: Running main() from gmock_main.cc 2024-09-16T05:09:10.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-16T05:09:10.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-16T05:09:10.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-16T05:09:10.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-16T05:09:10.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: watch_notify_test_cb 2024-09-16T05:09:10.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (265 ms) 2024-09-16T05:09:10.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-16T05:09:10.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-16T05:09:10.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94657263724416 err -107 2024-09-16T05:09:10.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1019 ms) 2024-09-16T05:09:10.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-16T05:09:10.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-16T05:09:10.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94657263724416 err -107 2024-09-16T05:09:10.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1010 ms) 2024-09-16T05:09:10.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-16T05:09:10.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: watch_notify2_test_cb from 4506 notify_id 98784247808 cookie 94657263743424 2024-09-16T05:09:10.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (7 ms) 2024-09-16T05:09:10.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-16T05:09:10.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: watch_notify2_test_cb from 4506 notify_id 98784247809 cookie 94657263750096 2024-09-16T05:09:10.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (8 ms) 2024-09-16T05:09:10.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-16T05:09:10.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: watch_notify2_test_cb from 4506 notify_id 98784247809 cookie 94657263764400 2024-09-16T05:09:10.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (12 ms) 2024-09-16T05:09:10.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-16T05:09:10.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: watch_notify2_test_cb from 4506 notify_id 98784247810 cookie 94657263749904 2024-09-16T05:09:10.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: watch_notify2_test_cb from 4506 notify_id 98784247810 cookie 94657263764400 2024-09-16T05:09:10.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (21 ms) 2024-09-16T05:09:10.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-16T05:09:10.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: watch_notify2_test_cb from 4506 notify_id 98784247809 cookie 94657263764400 2024-09-16T05:09:10.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: watch_notify2_test_cb from 4506 notify_id 103079215106 cookie 94657263764400 2024-09-16T05:09:10.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3010 ms) 2024-09-16T05:09:10.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-16T05:09:10.166 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-16T05:09:10.166 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94657263780336 err -107 2024-09-16T05:09:10.166 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: watch_notify2_test_cb from 4506 notify_id 133143986177 cookie 94657263783520 2024-09-16T05:09:10.166 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5022 ms) 2024-09-16T05:09:10.166 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-16T05:09:10.166 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-16T05:09:10.166 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94657263783520 err -107 2024-09-16T05:09:10.166 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1014 ms) 2024-09-16T05:09:10.166 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11388 ms total) 2024-09-16T05:09:10.166 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: 2024-09-16T05:09:10.166 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-16T05:09:10.166 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-16T05:09:10.166 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: watch_notify_test_cb 2024-09-16T05:09:10.167 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1158 ms) 2024-09-16T05:09:10.167 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1158 ms total) 2024-09-16T05:09:10.167 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: 2024-09-16T05:09:10.167 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-16T05:09:10.167 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (20741 ms total) 2024-09-16T05:09:10.167 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-16T05:09:15.124 INFO:tasks.workunit.client.0.smithi149.stdout: ec_list: Running main() from gmock_main.cc 2024-09-16T05:09:15.125 INFO:tasks.workunit.client.0.smithi149.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-09-16T05:09:15.125 INFO:tasks.workunit.client.0.smithi149.stdout: ec_list: [----------] Global test environment set-up. 2024-09-16T05:09:15.125 INFO:tasks.workunit.client.0.smithi149.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-09-16T05:09:15.125 INFO:tasks.workunit.client.0.smithi149.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-09-16T05:09:15.125 INFO:tasks.workunit.client.0.smithi149.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (7903 ms) 2024-09-16T05:09:15.125 INFO:tasks.workunit.client.0.smithi149.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-09-16T05:09:15.125 INFO:tasks.workunit.client.0.smithi149.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (6583 ms) 2024-09-16T05:09:15.125 INFO:tasks.workunit.client.0.smithi149.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-09-16T05:09:15.125 INFO:tasks.workunit.client.0.smithi149.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (11030 ms) 2024-09-16T05:09:15.125 INFO:tasks.workunit.client.0.smithi149.stdout: ec_list: [----------] 3 tests from NeoradosECList (25516 ms total) 2024-09-16T05:09:15.126 INFO:tasks.workunit.client.0.smithi149.stdout: ec_list: 2024-09-16T05:09:15.126 INFO:tasks.workunit.client.0.smithi149.stdout: ec_list: [----------] Global test environment tear-down 2024-09-16T05:09:15.126 INFO:tasks.workunit.client.0.smithi149.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (25516 ms total) 2024-09-16T05:09:15.126 INFO:tasks.workunit.client.0.smithi149.stdout: ec_list: [ PASSED ] 3 tests. 2024-09-16T05:09:16.123 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: Running main() from gmock_main.cc 2024-09-16T05:09:16.123 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-09-16T05:09:16.123 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [----------] Global test environment set-up. 2024-09-16T05:09:16.123 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-09-16T05:09:16.123 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-09-16T05:09:16.123 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (3766 ms) 2024-09-16T05:09:16.123 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-09-16T05:09:16.123 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3157 ms) 2024-09-16T05:09:16.123 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-09-16T05:09:16.123 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3009 ms) 2024-09-16T05:09:16.124 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-09-16T05:09:16.124 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (2466 ms) 2024-09-16T05:09:16.124 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-09-16T05:09:16.124 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3089 ms) 2024-09-16T05:09:16.124 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-09-16T05:09:16.124 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (2992 ms) 2024-09-16T05:09:16.124 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-09-16T05:09:16.124 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (7975 ms) 2024-09-16T05:09:16.124 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (26454 ms total) 2024-09-16T05:09:16.124 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: 2024-09-16T05:09:16.124 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [----------] Global test environment tear-down 2024-09-16T05:09:16.124 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (26455 ms total) 2024-09-16T05:09:16.125 INFO:tasks.workunit.client.0.smithi149.stdout: write_operations: [ PASSED ] 7 tests. 2024-09-16T05:09:16.190 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-16T05:09:16.190 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-16T05:09:16.191 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-16T05:09:16.191 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-16T05:09:16.191 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-16T05:09:16.191 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-16T05:09:16.191 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-16T05:09:16.191 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: 2024-09-16T05:09:16.191 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-16T05:09:16.191 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-16T05:09:16.191 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (3031 ms) 2024-09-16T05:09:16.191 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-16T05:09:16.191 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (2947 ms) 2024-09-16T05:09:16.191 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-16T05:09:16.191 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3057 ms) 2024-09-16T05:09:16.192 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-16T05:09:16.192 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3071 ms) 2024-09-16T05:09:16.192 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-16T05:09:16.192 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (2481 ms) 2024-09-16T05:09:16.192 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-16T05:09:16.192 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3045 ms) 2024-09-16T05:09:16.192 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-16T05:09:16.192 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (9058 ms) 2024-09-16T05:09:16.192 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (26690 ms total) 2024-09-16T05:09:16.192 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: 2024-09-16T05:09:16.192 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-16T05:09:16.192 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (26690 ms total) 2024-09-16T05:09:16.192 INFO:tasks.workunit.client.0.smithi149.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-16T05:09:17.160 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-09-16T05:09:17.160 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: [----------] Global test environment set-up. 2024-09-16T05:09:17.160 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-16T05:09:17.160 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-16T05:09:17.160 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5148 ms) 2024-09-16T05:09:17.160 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-16T05:09:17.160 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: [ OK ] LibRadosService.RegisterLate (57 ms) 2024-09-16T05:09:17.161 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-16T05:09:17.161 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: cluster: 2024-09-16T05:09:17.161 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: id: c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:09:17.161 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: health: HEALTH_WARN 2024-09-16T05:09:17.161 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 16 pool(s) do not have an application enabled 2024-09-16T05:09:17.161 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 2024-09-16T05:09:17.161 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: services: 2024-09-16T05:09:17.161 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: mon: 3 daemons, quorum a,b (age 39s), out of quorum: c 2024-09-16T05:09:17.161 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: mgr: x(active, since 82s) 2024-09-16T05:09:17.161 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: osd: 6 osds: 6 up (since 75s), 6 in (since 85s) 2024-09-16T05:09:17.161 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-16T05:09:17.161 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 2024-09-16T05:09:17.161 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: data: 2024-09-16T05:09:17.162 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: pools: 28 pools, 793 pgs 2024-09-16T05:09:17.162 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: objects: 59 objects, 584 KiB 2024-09-16T05:09:17.162 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: usage: 165 MiB used, 540 GiB / 540 GiB avail 2024-09-16T05:09:17.162 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: pgs: 80.580% pgs unknown 2024-09-16T05:09:17.162 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 3.405% pgs not active 2024-09-16T05:09:17.162 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 639 unknown 2024-09-16T05:09:17.162 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 127 active+clean 2024-09-16T05:09:17.162 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 27 creating+peering 2024-09-16T05:09:17.162 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 2024-09-16T05:09:17.162 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: io: 2024-09-16T05:09:17.162 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: client: 1.7 KiB/s rd, 39 KiB/s wr, 87 op/s rd, 139 op/s wr 2024-09-16T05:09:17.162 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 2024-09-16T05:09:17.162 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 2024-09-16T05:09:17.162 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: cluster: 2024-09-16T05:09:17.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: id: c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:09:17.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: health: HEALTH_WARN 2024-09-16T05:09:17.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 14 pool(s) do not have an application enabled 2024-09-16T05:09:17.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 2024-09-16T05:09:17.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: services: 2024-09-16T05:09:17.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: mon: 3 daemons, quorum a,b (age 41s), out of quorum: c 2024-09-16T05:09:17.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: mgr: x(active, since 84s) 2024-09-16T05:09:17.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: osd: 6 osds: 6 up (since 77s), 6 in (since 87s) 2024-09-16T05:09:17.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-09-16T05:09:17.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-16T05:09:17.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 2024-09-16T05:09:17.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: data: 2024-09-16T05:09:17.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: pools: 25 pools, 649 pgs 2024-09-16T05:09:17.163 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: objects: 51 objects, 581 KiB 2024-09-16T05:09:17.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: usage: 183 MiB used, 540 GiB / 540 GiB avail 2024-09-16T05:09:17.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: pgs: 48.690% pgs unknown 2024-09-16T05:09:17.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 11.402% pgs not active 2024-09-16T05:09:17.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 316 unknown 2024-09-16T05:09:17.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 259 active+clean 2024-09-16T05:09:17.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 74 creating+peering 2024-09-16T05:09:17.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 2024-09-16T05:09:17.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: io: 2024-09-16T05:09:17.164 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: client: 1.7 KiB/s rd, 23 KiB/s wr, 87 op/s rd, 121 op/s wr 2024-09-16T05:09:17.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 2024-09-16T05:09:17.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 2024-09-16T05:09:17.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: [ OK ] LibRadosService.StatusFormat (2348 ms) 2024-09-16T05:09:17.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-16T05:09:17.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: [ OK ] LibRadosService.Status (20094 ms) 2024-09-16T05:09:17.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: [----------] 4 tests from LibRadosService (27647 ms total) 2024-09-16T05:09:17.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: 2024-09-16T05:09:17.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: [----------] Global test environment tear-down 2024-09-16T05:09:17.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: [==========] 4 tests from 1 test suite ran. (27647 ms total) 2024-09-16T05:09:17.165 INFO:tasks.workunit.client.0.smithi149.stdout: api_service: [ PASSED ] 4 tests. 2024-09-16T05:09:19.094 INFO:tasks.workunit.client.0.smithi149.stdout:otify_pp: flushed 2024-09-16T05:09:19.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3008 ms) 2024-09-16T05:09:19.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-16T05:09:19.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: trying... 2024-09-16T05:09:19.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: handle_notify cookie 94401556106960 notify_id 154618822660 notifier_gid 5146 2024-09-16T05:09:19.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: timed out 2024-09-16T05:09:19.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: flushing 2024-09-16T05:09:19.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: flushed 2024-09-16T05:09:19.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3009 ms) 2024-09-16T05:09:19.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-16T05:09:19.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: List watches 2024-09-16T05:09:19.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: notify2 2024-09-16T05:09:19.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: handle_notify cookie 94401556106960 notify_id 167503724549 notifier_gid 5146 2024-09-16T05:09:19.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: notify2 done 2024-09-16T05:09:19.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: watch_check 2024-09-16T05:09:19.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: unwatch2 2024-09-16T05:09:19.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: flushing 2024-09-16T05:09:19.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: done 2024-09-16T05:09:19.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3025 ms) 2024-09-16T05:09:19.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-16T05:09:19.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: List watches 2024-09-16T05:09:19.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: notify2 2024-09-16T05:09:19.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: handle_notify cookie 94401556106960 notify_id 180388626437 notifier_gid 5146 2024-09-16T05:09:19.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: notify2 done 2024-09-16T05:09:19.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: watch_check 2024-09-16T05:09:19.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: unwatch2 2024-09-16T05:09:19.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: flushing 2024-09-16T05:09:19.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: done 2024-09-16T05:09:19.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3015 ms) 2024-09-16T05:09:19.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (15794 ms total) 2024-09-16T05:09:19.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: 2024-09-16T05:09:19.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-16T05:09:19.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (29654 ms total) 2024-09-16T05:09:19.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-16T05:09:19.901 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-16T05:09:19.901 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-16T05:09:19.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-16T05:09:19.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-16T05:09:19.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5058 ms) 2024-09-16T05:09:19.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-16T05:09:19.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (120 ms) 2024-09-16T05:09:19.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-16T05:09:19.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20051 ms) 2024-09-16T05:09:19.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-16T05:09:19.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: attempt 0 of 20 2024-09-16T05:09:19.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5175 ms) 2024-09-16T05:09:19.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30404 ms total) 2024-09-16T05:09:19.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: 2024-09-16T05:09:19.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-16T05:09:19.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30404 ms total) 2024-09-16T05:09:19.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-16T05:09:23.172 INFO:tasks.workunit.client.0.smithi149.stdout: misc: Running main() from gmock_main.cc 2024-09-16T05:09:23.172 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-09-16T05:09:23.172 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [----------] Global test environment set-up. 2024-09-16T05:09:23.172 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-09-16T05:09:23.172 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-09-16T05:09:23.172 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ OK ] NeoRadosMisc.Version (2922 ms) 2024-09-16T05:09:23.172 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-09-16T05:09:23.172 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (2092 ms) 2024-09-16T05:09:23.173 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-09-16T05:09:23.173 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ OK ] NeoRadosMisc.LongName (2939 ms) 2024-09-16T05:09:23.173 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-09-16T05:09:23.173 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (2990 ms) 2024-09-16T05:09:23.173 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-09-16T05:09:23.173 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (2521 ms) 2024-09-16T05:09:23.173 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-09-16T05:09:23.173 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3073 ms) 2024-09-16T05:09:23.173 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-09-16T05:09:23.173 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ OK ] NeoRadosMisc.Exec (2998 ms) 2024-09-16T05:09:23.173 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-09-16T05:09:23.173 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (2926 ms) 2024-09-16T05:09:23.173 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-09-16T05:09:23.173 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3062 ms) 2024-09-16T05:09:23.174 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-09-16T05:09:23.174 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ OK ] NeoRadosMisc.BigObject (2959 ms) 2024-09-16T05:09:23.174 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-09-16T05:09:23.174 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2004 ms) 2024-09-16T05:09:23.174 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-09-16T05:09:23.174 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3065 ms) 2024-09-16T05:09:23.174 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [----------] 12 tests from NeoRadosMisc (33551 ms total) 2024-09-16T05:09:23.174 INFO:tasks.workunit.client.0.smithi149.stdout: misc: 2024-09-16T05:09:23.174 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [----------] Global test environment tear-down 2024-09-16T05:09:23.174 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [==========] 12 tests from 1 test suite ran. (33551 ms total) 2024-09-16T05:09:23.174 INFO:tasks.workunit.client.0.smithi149.stdout: misc: [ PASSED ] 12 tests. 2024-09-16T05:09:27.070 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: Running main() from gmock_main.cc 2024-09-16T05:09:27.071 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-16T05:09:27.071 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [----------] Global test environment set-up. 2024-09-16T05:09:27.071 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-16T05:09:27.071 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-16T05:09:27.071 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ OK ] LibRadosPools.PoolList (3171 ms) 2024-09-16T05:09:27.071 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-16T05:09:27.071 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (2946 ms) 2024-09-16T05:09:27.071 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-16T05:09:27.071 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3044 ms) 2024-09-16T05:09:27.072 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-16T05:09:27.072 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3062 ms) 2024-09-16T05:09:27.072 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-16T05:09:27.072 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (2504 ms) 2024-09-16T05:09:27.072 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-16T05:09:27.072 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5050 ms) 2024-09-16T05:09:27.072 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-16T05:09:27.072 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (4990 ms) 2024-09-16T05:09:27.072 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-16T05:09:27.072 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5032 ms) 2024-09-16T05:09:27.072 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-16T05:09:27.073 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7896 ms) 2024-09-16T05:09:27.073 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [----------] 9 tests from LibRadosPools (37695 ms total) 2024-09-16T05:09:27.073 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: 2024-09-16T05:09:27.073 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [----------] Global test environment tear-down 2024-09-16T05:09:27.073 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (37695 ms total) 2024-09-16T05:09:27.073 INFO:tasks.workunit.client.0.smithi149.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-16T05:09:31.107 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: Running main() from gmock_main.cc 2024-09-16T05:09:31.108 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-09-16T05:09:31.108 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [----------] Global test environment set-up. 2024-09-16T05:09:31.108 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-09-16T05:09:31.108 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-09-16T05:09:31.108 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (2910 ms) 2024-09-16T05:09:31.108 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-09-16T05:09:31.108 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (2949 ms) 2024-09-16T05:09:31.108 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-09-16T05:09:31.108 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3046 ms) 2024-09-16T05:09:31.108 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-09-16T05:09:31.108 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3050 ms) 2024-09-16T05:09:31.108 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-09-16T05:09:31.109 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (2508 ms) 2024-09-16T05:09:31.109 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-09-16T05:09:31.109 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3037 ms) 2024-09-16T05:09:31.109 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-09-16T05:09:31.109 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3028 ms) 2024-09-16T05:09:31.109 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-09-16T05:09:31.109 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (2923 ms) 2024-09-16T05:09:31.109 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-09-16T05:09:31.109 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3028 ms) 2024-09-16T05:09:31.109 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-09-16T05:09:31.109 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3028 ms) 2024-09-16T05:09:31.109 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-09-16T05:09:31.109 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (2952 ms) 2024-09-16T05:09:31.109 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-09-16T05:09:31.110 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3072 ms) 2024-09-16T05:09:31.110 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-09-16T05:09:31.110 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2925 ms) 2024-09-16T05:09:31.110 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-09-16T05:09:31.110 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3022 ms) 2024-09-16T05:09:31.110 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (41478 ms total) 2024-09-16T05:09:31.110 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: 2024-09-16T05:09:31.110 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [----------] Global test environment tear-down 2024-09-16T05:09:31.110 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (41478 ms total) 2024-09-16T05:09:31.110 INFO:tasks.workunit.client.0.smithi149.stdout: read_operations: [ PASSED ] 14 tests. 2024-09-16T05:09:31.111 INFO:tasks.workunit.client.0.smithi149.stdout: io: Running main() from gmock_main.cc 2024-09-16T05:09:31.111 INFO:tasks.workunit.client.0.smithi149.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-09-16T05:09:31.111 INFO:tasks.workunit.client.0.smithi149.stdout: io: [----------] Global test environment set-up. 2024-09-16T05:09:31.111 INFO:tasks.workunit.client.0.smithi149.stdout: io: [----------] 14 tests from NeoRadosIo 2024-09-16T05:09:31.111 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-09-16T05:09:31.111 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ OK ] NeoRadosIo.Limits (2960 ms) 2024-09-16T05:09:31.111 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-09-16T05:09:31.112 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (2942 ms) 2024-09-16T05:09:31.112 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-09-16T05:09:31.112 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ OK ] NeoRadosIo.ReadOp (3048 ms) 2024-09-16T05:09:31.112 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-09-16T05:09:31.112 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ OK ] NeoRadosIo.SparseRead (3062 ms) 2024-09-16T05:09:31.112 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-09-16T05:09:31.112 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ OK ] NeoRadosIo.RoundTrip (2485 ms) 2024-09-16T05:09:31.112 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-09-16T05:09:31.112 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3049 ms) 2024-09-16T05:09:31.112 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-09-16T05:09:31.112 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3043 ms) 2024-09-16T05:09:31.112 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-09-16T05:09:31.112 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (2910 ms) 2024-09-16T05:09:31.113 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-09-16T05:09:31.113 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3077 ms) 2024-09-16T05:09:31.113 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-09-16T05:09:31.113 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ OK ] NeoRadosIo.Trunc (2985 ms) 2024-09-16T05:09:31.113 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-09-16T05:09:31.113 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ OK ] NeoRadosIo.Remove (2946 ms) 2024-09-16T05:09:31.113 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-09-16T05:09:31.113 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3045 ms) 2024-09-16T05:09:31.113 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-09-16T05:09:31.113 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ OK ] NeoRadosIo.RmXattr (2952 ms) 2024-09-16T05:09:31.113 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-09-16T05:09:31.113 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3026 ms) 2024-09-16T05:09:31.113 INFO:tasks.workunit.client.0.smithi149.stdout: io: [----------] 14 tests from NeoRadosIo (41530 ms total) 2024-09-16T05:09:31.114 INFO:tasks.workunit.client.0.smithi149.stdout: io: 2024-09-16T05:09:31.114 INFO:tasks.workunit.client.0.smithi149.stdout: io: [----------] Global test environment tear-down 2024-09-16T05:09:31.114 INFO:tasks.workunit.client.0.smithi149.stdout: io: [==========] 14 tests from 1 test suite ran. (41530 ms total) 2024-09-16T05:09:31.114 INFO:tasks.workunit.client.0.smithi149.stdout: io: [ PASSED ] 14 tests. 2024-09-16T05:09:31.142 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-16T05:09:31.143 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [----------] Global test environment set-up. 2024-09-16T05:09:31.143 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-16T05:09:31.143 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-16T05:09:31.143 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-16T05:09:31.143 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-16T05:09:31.143 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: 2024-09-16T05:09:31.143 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-16T05:09:31.143 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-16T05:09:31.143 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-16T05:09:31.143 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: 2024-09-16T05:08:49.468+0000 7f8143252880 -1 failed for service _ceph-mon._tcp 2024-09-16T05:09:31.143 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: 2024-09-16T05:08:49.468+0000 7f8143252880 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-16T05:09:31.144 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (109 ms) 2024-09-16T05:09:31.144 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-16T05:09:31.144 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5012 ms) 2024-09-16T05:09:31.144 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5121 ms total) 2024-09-16T05:09:31.144 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: 2024-09-16T05:09:31.144 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-16T05:09:31.144 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-16T05:09:31.144 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: started 0x55dd77fb3d40 2024-09-16T05:09:31.144 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: started 0x55dd78155370 2024-09-16T05:09:31.144 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: started 2 aios 2024-09-16T05:09:31.144 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: waiting 0x55dd77fb3d40 2024-09-16T05:09:31.144 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: waiting 0x55dd78155370 2024-09-16T05:09:31.144 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: done. 2024-09-16T05:09:31.144 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (6077 ms) 2024-09-16T05:09:31.145 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (6077 ms total) 2024-09-16T05:09:31.145 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: 2024-09-16T05:09:31.145 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-16T05:09:31.145 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-16T05:09:31.145 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-09-16T05:09:31.145 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-16T05:09:31.145 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ OK ] LibRadosMisc.Exec (119 ms) 2024-09-16T05:09:31.145 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-16T05:09:31.145 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (11 ms) 2024-09-16T05:09:31.145 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-16T05:09:31.145 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (4 ms) 2024-09-16T05:09:31.145 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-16T05:09:31.145 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4925 ms) 2024-09-16T05:09:31.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-16T05:09:31.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-16T05:09:31.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-16T05:09:31.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-09-16T05:09:31.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-16T05:09:31.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (23173 ms) 2024-09-16T05:09:31.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [----------] 8 tests from LibRadosMisc (28232 ms total) 2024-09-16T05:09:31.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: 2024-09-16T05:09:31.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [----------] Global test environment tear-down 2024-09-16T05:09:31.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (41755 ms total) 2024-09-16T05:09:31.146 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-16T05:09:45.781 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T05:09:45.781 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T05:09:45.781 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T05:09:45.782 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T05:09:45.785 INFO:tasks.ceph.mon.c:Started 2024-09-16T05:09:45.785 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T05:09:45.785 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T05:09:45.786 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T05:10:00.788 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T05:10:00.789 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T05:10:00.790 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:10:00.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:10:00.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: Running main() from gmock_main.cc 2024-09-16T05:10:00.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-09-16T05:10:00.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-16T05:10:00.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-16T05:10:00.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-16T05:10:00.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2043 ms) 2024-09-16T05:10:00.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-16T05:10:00.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2021 ms) 2024-09-16T05:10:00.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-16T05:10:00.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2107 ms) 2024-09-16T05:10:00.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-16T05:10:00.902 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2024 ms) 2024-09-16T05:10:00.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8195 ms total) 2024-09-16T05:10:00.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: 2024-09-16T05:10:00.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-09-16T05:10:00.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-16T05:10:00.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4135 ms) 2024-09-16T05:10:00.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-16T05:10:00.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4097 ms) 2024-09-16T05:10:00.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-09-16T05:10:00.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (5916 ms) 2024-09-16T05:10:00.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (14148 ms total) 2024-09-16T05:10:00.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: 2024-09-16T05:10:00.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-16T05:10:00.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-16T05:10:00.903 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2994 ms) 2024-09-16T05:10:00.904 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-16T05:10:00.904 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2019 ms) 2024-09-16T05:10:00.904 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-16T05:10:00.904 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1931 ms) 2024-09-16T05:10:00.904 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-16T05:10:00.904 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2040 ms) 2024-09-16T05:10:00.904 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8984 ms total) 2024-09-16T05:10:00.904 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: 2024-09-16T05:10:00.904 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-16T05:10:00.904 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-16T05:10:00.904 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4066 ms) 2024-09-16T05:10:00.904 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-16T05:10:00.904 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4022 ms) 2024-09-16T05:10:00.904 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8088 ms total) 2024-09-16T05:10:00.905 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: 2024-09-16T05:10:00.905 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-16T05:10:00.905 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (71504 ms total) 2024-09-16T05:10:00.905 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-09-16T05:10:00.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 10 monclient: get_monmap_and_config 2024-09-16T05:10:00.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 10 monclient: build_initial_monmap 2024-09-16T05:10:00.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 10 monclient: monmap: 2024-09-16T05:10:00.923 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:00.923 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:00.923 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:10:00.921607+0000 2024-09-16T05:10:00.923 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:10:00.921607+0000 2024-09-16T05:10:00.923 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:00.923 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:00.923 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:10:00.923 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:10:00.923 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:10:00.923 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:10:00.923 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:10:00.923 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 1 Processor -- start 2024-09-16T05:10:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 1 -- start start 2024-09-16T05:10:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 10 monclient: init 2024-09-16T05:10:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-16T05:10:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f96b009cb10 0x7f96b009cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 10 monclient(hunting): picked mon.noname-a con 0x7f96b009cb10 addr v2:172.21.15.114:3300/0 2024-09-16T05:10:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f96b0135310 addr v1:172.21.15.149:6789/0 2024-09-16T05:10:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96b00a2570 addr v1:172.21.15.114:6789/0 2024-09-16T05:10:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96b009a5f0 con 0x7f96b00a2570 2024-09-16T05:10:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96b0075d30 con 0x7f96b0135310 2024-09-16T05:10:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f96b00a04b0 con 0x7f96b009cb10 2024-09-16T05:10:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96b695d640 10 monclient(hunting): authenticate will time out at 1085.551639s 2024-09-16T05:10:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96af7fe640 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f96b0135310 0x7f96b00a1e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.114:59996/0 (socket says 172.21.15.114:59996) 2024-09-16T05:10:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96af7fe640 1 -- 172.21.15.114:0/1108831133 learned_addr learned my addr 172.21.15.114:0/1108831133 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:10:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.919+0000 7f96affff640 1 --2- 172.21.15.114:0/1108831133 >> v2:172.21.15.114:3300/0 conn(0x7f96b009cb10 0x7f96b009cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): get_auth_request con 0x7f96b009cb10 auth_method 0 2024-09-16T05:10:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:00.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3686211306 0 0) 0x7f96b0075d30 con 0x7f96b0135310 2024-09-16T05:10:00.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:10:00.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:00.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:00.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): my global_id is 19525 2024-09-16T05:10:00.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9698003530 con 0x7f96b0135310 2024-09-16T05:10:00.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:00.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:00.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:00.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 <== mon.1 v1:172.21.15.149:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2395937927 0 0) 0x7f9698003530 con 0x7f96b0135310 2024-09-16T05:10:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 >> v2:172.21.15.114:3300/0 conn(0x7f96b009cb10 msgr2=0x7f96b009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 --2- 172.21.15.114:0/1108831133 >> v2:172.21.15.114:3300/0 conn(0x7f96b009cb10 0x7f96b009cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 >> v1:172.21.15.114:6789/0 conn(0x7f96b00a2570 legacy=0x7f96b013d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient: _finish_hunting 0 2024-09-16T05:10:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 monclient: found mon.noname-c-legacy 2024-09-16T05:10:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.149:6789/0 2024-09-16T05:10:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 --> v1:172.21.15.149:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b009d430 con 0x7f96b0135310 2024-09-16T05:10:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient: _finish_auth 0 2024-09-16T05:10:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient: _check_auth_tickets 2024-09-16T05:10:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 <== mon.1 v1:172.21.15.149:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f96a4001680 con 0x7f96b0135310 2024-09-16T05:10:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T05:10:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient: dump: 2024-09-16T05:10:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:00.932 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] but i'm connected to v1:172.21.15.149:6789/0, reconnecting 2024-09-16T05:10:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 >> v1:172.21.15.149:6789/0 conn(0x7f96b0135310 legacy=0x7f96b00a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:10:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f96980039c0 0x7f9698003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): picked mon.b con 0x7f96980039c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698004810 0x7f9698004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): picked mon.c con 0x7f9698004810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96980052f0 0x7f969800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): picked mon.a con 0x7f96980052f0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f969800c190 con 0x7f96980052f0 2024-09-16T05:10:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:00.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f969800c340 con 0x7f96980039c0 2024-09-16T05:10:00.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:00.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f969800c540 con 0x7f9698004810 2024-09-16T05:10:00.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:00.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:10:00.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698004810 0x7f9698004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:00.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f96980039c0 0x7f9698003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:00.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient(hunting): get_auth_request con 0x7f9698004810 auth_method 0 2024-09-16T05:10:00.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:00.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:00.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:00.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): get_auth_request con 0x7f96980039c0 auth_method 0 2024-09-16T05:10:00.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:00.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:00.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:00.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:00.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:00.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:10:00.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:00.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:00.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:10:00.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96b4efa640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96980052f0 0x7f969800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:00.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient(hunting): handle_auth_done global_id 19525 payload 293 2024-09-16T05:10:00.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 1 -- 172.21.15.114:0/1108831133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f96980039c0 msgr2=0x7f9698003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:00.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f96980039c0 0x7f9698003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:00.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 1 -- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96980052f0 msgr2=0x7f969800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:00.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96980052f0 0x7f969800b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:00.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:10:00.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 1 monclient: found mon.c 2024-09-16T05:10:00.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:00.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:00.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96b4efa640 1 -- 172.21.15.114:0/1108831133 reap_dead start 2024-09-16T05:10:00.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 1 -- 172.21.15.114:0/1108831133 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9698003530 con 0x7f9698004810 2024-09-16T05:10:00.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient: _finish_auth 0 2024-09-16T05:10:00.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:10:00.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:00.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96b695d640 5 monclient: authenticate success, global_id 19525 2024-09-16T05:10:00.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96b695d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:00.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698004810 0x7f9698004c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f96a4001ec0 tx=0x7f96a4021050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f143966e5387d10e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:00.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 1 -- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698004810 msgr2=0x7f9698004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T05:10:00.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 1 -- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698004810 msgr2=0x7f9698004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:10:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698004810 0x7f9698004c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f96a4001ec0 tx=0x7f96a4021050 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:10:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698004810 0x7f9698004c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f96a4001ec0 tx=0x7f96a4021050 comp rx=0 tx=0).stop 2024-09-16T05:10:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698004810 msgr2=0x7f9698004c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698004810 0x7f9698004c90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:10:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698003de0 0x7f96980099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): picked mon.c con 0x7f9698003de0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9698009fb0 0x7f969800a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): picked mon.b con 0x7f9698009fb0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f969800ab90 0x7f969800b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): picked mon.a con 0x7f969800ab90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:00.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f969800b920 con 0x7f969800ab90 2024-09-16T05:10:00.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:00.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f969800bad0 con 0x7f9698009fb0 2024-09-16T05:10:00.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:00.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f969800bcd0 con 0x7f9698003de0 2024-09-16T05:10:00.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:00.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96aeffd640 10 monclient(hunting): ms_handle_reset hunted mon [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:00.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698003de0 0x7f96980099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:00.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): get_auth_request con 0x7f9698003de0 auth_method 0 2024-09-16T05:10:00.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96b4efa640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f969800ab90 0x7f969800b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:00.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:00.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:00.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:00.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96b4efa640 10 monclient(hunting): get_auth_request con 0x7f969800ab90 auth_method 0 2024-09-16T05:10:00.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96b4efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:00.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96b4efa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:00.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96b4efa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:00.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:00.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:00.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:10:00.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96b4efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:00.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96b4efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:00.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96b4efa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:10:00.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9698009fb0 0x7f969800a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:00.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient(hunting): get_auth_request con 0x7f9698009fb0 auth_method 0 2024-09-16T05:10:00.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:00.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:00.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96af7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:00.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient(hunting): handle_auth_done global_id 19525 payload 293 2024-09-16T05:10:00.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 1 -- 172.21.15.114:0/1108831133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9698009fb0 msgr2=0x7f969800a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:00.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9698009fb0 0x7f969800a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:00.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 1 -- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f969800ab90 msgr2=0x7f969800b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:00.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f969800ab90 0x7f969800b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:00.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient: _finish_hunting 0 2024-09-16T05:10:00.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 1 monclient: found mon.c 2024-09-16T05:10:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 1 -- 172.21.15.114:0/1108831133 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=2+}) -- 0x7f96b0075d30 con 0x7f9698003de0 2024-09-16T05:10:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient: _finish_auth 0 2024-09-16T05:10:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 10 monclient: _check_auth_tickets 2024-09-16T05:10:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.923+0000 7f96affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:00.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698003de0 0x7f96980099c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f96b00a0af0 tx=0x7f96a4007900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f03d50bcd20dd632 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:00.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a00095f0 con 0x7f9698003de0 2024-09-16T05:10:00.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:00.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:00.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96aeffd640 10 monclient: dump: 2024-09-16T05:10:00.961 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:00.961 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:00.961 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:00.961 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:00.961 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:00.962 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:00.962 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:00.962 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:00.962 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:00.962 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:00.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:00.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96aeffd640 1 -- 172.21.15.114:0/1108831133 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96a0004030 con 0x7f9698003de0 2024-09-16T05:10:00.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96aeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:00.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient: get_monmap_and_config success 2024-09-16T05:10:00.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 4 set_mon_vals no callback set 2024-09-16T05:10:00.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient: shutdown 2024-09-16T05:10:00.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:00.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 -- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698003de0 msgr2=0x7f96980099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:00.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698003de0 0x7f96980099c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f96b00a0af0 tx=0x7f96a4007900 comp rx=0 tx=0).stop 2024-09-16T05:10:00.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 -- 172.21.15.114:0/1108831133 shutdown_connections 2024-09-16T05:10:00.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f969800ab90 0x7f969800b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:00.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9698009fb0 0x7f969800a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:00.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 --2- 172.21.15.114:0/1108831133 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698003de0 0x7f96980099c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:00.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 -- 172.21.15.114:0/1108831133 >> 172.21.15.114:0/1108831133 conn(0x7f96b00a2f50 msgr2=0x7f96b00a3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:00.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 -- 172.21.15.114:0/1108831133 shutdown_connections 2024-09-16T05:10:00.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 -- 172.21.15.114:0/1108831133 wait complete. 2024-09-16T05:10:00.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient: build_initial_monmap 2024-09-16T05:10:00.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient: monmap: 2024-09-16T05:10:00.966 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:00.966 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:10:00.966 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:10:00.966 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:10:00.967 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:00.967 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:00.967 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:10:00.967 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:10:00.967 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:10:00.967 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:00.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 librados: starting msgr at 2024-09-16T05:10:00.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 librados: starting objecter 2024-09-16T05:10:00.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 Processor -- start 2024-09-16T05:10:00.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 -- start start 2024-09-16T05:10:00.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 librados: setting wanted keys 2024-09-16T05:10:00.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 librados: calling monclient init 2024-09-16T05:10:00.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient: init 2024-09-16T05:10:00.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:00.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:10:00.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698003de0 0x7f96b006f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:00.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient(hunting): picked mon.noname-c con 0x7f9698003de0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:00.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9698009fb0 0x7f96b006f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:00.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient(hunting): picked mon.noname-a con 0x7f9698009fb0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:00.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f969800ab90 0x7f96b006fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:00.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient(hunting): picked mon.noname-b con 0x7f969800ab90 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:00.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:00.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f96b00a1e60 con 0x7f9698009fb0 2024-09-16T05:10:00.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:00.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f96b00757d0 con 0x7f969800ab90 2024-09-16T05:10:00.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:00.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f96b00a1fe0 con 0x7f9698003de0 2024-09-16T05:10:00.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:00.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient(hunting): authenticate will time out at 1085.558825s 2024-09-16T05:10:00.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698003de0 0x7f96b006f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:00.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698003de0 0x7f96b006f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:43606/0 (socket says 172.21.15.114:43606) 2024-09-16T05:10:00.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 1 -- 172.21.15.114:0/2001184675 learned_addr learned my addr 172.21.15.114:0/2001184675 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:00.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 10 monclient(hunting): get_auth_request con 0x7f9698003de0 auth_method 0 2024-09-16T05:10:00.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:00.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:00.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:00.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96af7fe640 1 --2- 172.21.15.114:0/2001184675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9698009fb0 0x7f96b006f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:00.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:00.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:00.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:00.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96af7fe640 10 monclient(hunting): get_auth_request con 0x7f9698009fb0 auth_method 0 2024-09-16T05:10:00.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:00.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:00.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:00.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b4efa640 1 --2- 172.21.15.114:0/2001184675 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f969800ab90 0x7f96b006fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:00.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b4efa640 10 monclient(hunting): get_auth_request con 0x7f969800ab90 auth_method 0 2024-09-16T05:10:00.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b4efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:00.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b4efa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:00.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b4efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:00.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:00.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:00.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:00.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 10 monclient(hunting): handle_auth_done global_id 54104 payload 824 2024-09-16T05:10:00.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 1 -- 172.21.15.114:0/2001184675 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f969800ab90 msgr2=0x7f96b006fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:00.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 1 --2- 172.21.15.114:0/2001184675 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f969800ab90 0x7f96b006fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:00.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 1 -- 172.21.15.114:0/2001184675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9698009fb0 msgr2=0x7f96b006f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:00.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 1 --2- 172.21.15.114:0/2001184675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9698009fb0 0x7f96b006f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:00.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 10 monclient: _finish_hunting 0 2024-09-16T05:10:00.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 1 monclient: found mon.noname-c 2024-09-16T05:10:00.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:00.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:10:00.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 1 -- 172.21.15.114:0/2001184675 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b0133fa0 con 0x7f9698003de0 2024-09-16T05:10:00.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b4efa640 1 --2- 172.21.15.114:0/2001184675 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f969800ab90 0x7f96b006fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:10:00.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96affff640 1 --2- 172.21.15.114:0/2001184675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698003de0 0x7f96b006f150 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f96a0011690 tx=0x7f96a00046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:00.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96acff9640 10 client.?.objecter ms_handle_connect 0x7f9698003de0 2024-09-16T05:10:00.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96acff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:10:00.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96acff9640 1 -- 172.21.15.114:0/2001184675 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a0003a30 con 0x7f9698003de0 2024-09-16T05:10:00.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:00.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96acff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:10:00.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96acff9640 10 monclient: dump: 2024-09-16T05:10:00.975 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:00.975 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:00.975 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:00.975 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:00.975 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:00.975 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:00.975 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:00.975 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:00.976 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:00.976 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:00.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96acff9640 10 monclient: _finish_auth 0 2024-09-16T05:10:00.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96acff9640 10 monclient: _check_auth_tickets 2024-09-16T05:10:00.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:00.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96acff9640 1 -- 172.21.15.114:0/2001184675 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96a002c070 con 0x7f9698003de0 2024-09-16T05:10:00.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96acff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:00.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96acff9640 1 -- 172.21.15.114:0/2001184675 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a0003e10 con 0x7f9698003de0 2024-09-16T05:10:00.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:00.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96acff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:00.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96acff9640 10 monclient: dump: 2024-09-16T05:10:00.976 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:00.976 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:00.976 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:00.977 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 5 monclient: authenticate success, global_id 54104 2024-09-16T05:10:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient: _renew_subs 2024-09-16T05:10:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96ae7fc640 4 set_mon_vals no callback set 2024-09-16T05:10:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:00.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 -- 172.21.15.114:0/2001184675 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b0072670 con 0x7f9698003de0 2024-09-16T05:10:00.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 client.54104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:10:00.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient: _renew_subs 2024-09-16T05:10:00.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 -- 172.21.15.114:0/2001184675 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b009d640 con 0x7f9698003de0 2024-09-16T05:10:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.927+0000 7f96b695d640 1 librados: init done 2024-09-16T05:10:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.931+0000 7f96b695d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.931+0000 7f96b695d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.931+0000 7f96b695d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.931+0000 7f96acff9640 1 -- 172.21.15.114:0/2001184675 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f96a0012040 con 0x7f9698003de0 2024-09-16T05:10:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.931+0000 7f96b695d640 1 -- 172.21.15.114:0/2001184675 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b01369b0 con 0x7f9698003de0 2024-09-16T05:10:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.931+0000 7f96acff9640 1 --2- 172.21.15.114:0/2001184675 >> v2:172.21.15.149:6812/1185531135 conn(0x7f969c037810 0x7f969c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:00.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.931+0000 7f96acff9640 1 -- 172.21.15.114:0/2001184675 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(79..79 src has 1..79) ==== 7003+0+0 (secure 0 0 0) 0x7f96a004ac40 con 0x7f9698003de0 2024-09-16T05:10:00.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.931+0000 7f96acff9640 10 client.54104.objecter ms_dispatch 0x7f96b0000ba0 osd_map(79..79 src has 1..79) 2024-09-16T05:10:00.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.931+0000 7f96acff9640 3 client.54104.objecter handle_osd_map got epochs [79,79] > 0 2024-09-16T05:10:00.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.931+0000 7f96acff9640 3 client.54104.objecter handle_osd_map decoding full epoch 79 2024-09-16T05:10:00.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.931+0000 7f96acff9640 20 client.54104.objecter dump_active .. 0 homeless 2024-09-16T05:10:00.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.931+0000 7f96af7fe640 1 --2- 172.21.15.114:0/2001184675 >> v2:172.21.15.149:6812/1185531135 conn(0x7f969c037810 0x7f969c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:00.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.931+0000 7f96af7fe640 10 monclient: get_auth_request con 0x7f969c037810 auth_method 0 2024-09-16T05:10:00.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.931+0000 7f96acff9640 10 client.54104.objecter ms_handle_connect 0x7f969c037810 2024-09-16T05:10:00.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.931+0000 7f96af7fe640 1 --2- 172.21.15.114:0/2001184675 >> v2:172.21.15.149:6812/1185531135 conn(0x7f969c037810 0x7f969c039cd0 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7f96a4021090 tx=0x7f96a40ab070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:00.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.935+0000 7f96acff9640 1 -- 172.21.15.114:0/2001184675 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f96a0035020 con 0x7f9698003de0 2024-09-16T05:10:00.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.935+0000 7f96acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:00.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:00.935+0000 7f96acff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:10:01.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.175+0000 7f96b695d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:10:01.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.175+0000 7f96b695d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:10:01.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.175+0000 7f96b695d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:01.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.175+0000 7f96b695d640 1 -- 172.21.15.114:0/2001184675 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f96b00711c0 con 0x7f9698003de0 2024-09-16T05:10:01.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.175+0000 7f96acff9640 1 -- 172.21.15.114:0/2001184675 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f96b0133fa0 con 0x7f9698003de0 2024-09-16T05:10:01.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.175+0000 7f96acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:10:01.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.175+0000 7f96acff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:10:01.180 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:10:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 10 librados: watch_flush enter 2024-09-16T05:10:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 10 librados: watch_flush exit 2024-09-16T05:10:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 20 client.54104.objecter shutdown clearing up homeless session... 2024-09-16T05:10:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 10 client.54104.objecter successfully canceled tick 2024-09-16T05:10:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 1 -- 172.21.15.114:0/2001184675 >> v2:172.21.15.149:6812/1185531135 conn(0x7f969c037810 msgr2=0x7f969c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 1 --2- 172.21.15.114:0/2001184675 >> v2:172.21.15.149:6812/1185531135 conn(0x7f969c037810 0x7f969c039cd0 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7f96a4021090 tx=0x7f96a40ab070 comp rx=0 tx=0).stop 2024-09-16T05:10:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 10 monclient: shutdown 2024-09-16T05:10:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 1 -- 172.21.15.114:0/2001184675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698003de0 msgr2=0x7f96b006f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 1 --2- 172.21.15.114:0/2001184675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698003de0 0x7f96b006f150 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f96a0011690 tx=0x7f96a00046d0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 1 -- 172.21.15.114:0/2001184675 shutdown_connections 2024-09-16T05:10:01.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 1 --2- 172.21.15.114:0/2001184675 >> v2:172.21.15.149:6812/1185531135 conn(0x7f969c037810 0x7f969c039cd0 unknown :-1 s=CLOSED pgs=3382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 1 --2- 172.21.15.114:0/2001184675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9698009fb0 0x7f96b006f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 1 --2- 172.21.15.114:0/2001184675 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f969800ab90 0x7f96b006fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 1 --2- 172.21.15.114:0/2001184675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9698003de0 0x7f96b006f150 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 1 -- 172.21.15.114:0/2001184675 >> 172.21.15.114:0/2001184675 conn(0x7f96b00a2f50 msgr2=0x7f96b0133730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:01.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 1 -- 172.21.15.114:0/2001184675 shutdown_connections 2024-09-16T05:10:01.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 1 -- 172.21.15.114:0/2001184675 wait complete. 2024-09-16T05:10:01.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.179+0000 7f96b695d640 1 librados: shutdown 2024-09-16T05:10:01.197 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:10:01.198 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:10:01.198 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:10:01.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 10 monclient: get_monmap_and_config 2024-09-16T05:10:01.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 10 monclient: build_initial_monmap 2024-09-16T05:10:01.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 10 monclient: monmap: 2024-09-16T05:10:01.314 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:01.314 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.314 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:10:01.312840+0000 2024-09-16T05:10:01.314 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:10:01.312840+0000 2024-09-16T05:10:01.314 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:01.314 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.314 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:10:01.314 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:10:01.314 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:10:01.314 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:10:01.314 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:10:01.314 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 1 Processor -- start 2024-09-16T05:10:01.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 1 -- start start 2024-09-16T05:10:01.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 10 monclient: init 2024-09-16T05:10:01.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:01.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-16T05:10:01.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc274137730 addr v1:172.21.15.149:6789/0 2024-09-16T05:10:01.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc2740a5d00 0x7fc27413c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 10 monclient(hunting): picked mon.noname-a con 0x7fc2740a5d00 addr v2:172.21.15.114:3300/0 2024-09-16T05:10:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc27413ca00 addr v1:172.21.15.114:6789/0 2024-09-16T05:10:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2740a55e0 con 0x7fc27413ca00 2024-09-16T05:10:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2740a3710 con 0x7fc274137730 2024-09-16T05:10:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fc27409c450 con 0x7fc2740a5d00 2024-09-16T05:10:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc278ed9640 10 monclient(hunting): authenticate will time out at 1085.942109s 2024-09-16T05:10:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc2740a5d00 0x7fc27413c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc2740a5d00 0x7fc27413c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59478/0 (socket says 172.21.15.114:59478) 2024-09-16T05:10:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 1 -- 172.21.15.114:0/616409090 learned_addr learned my addr 172.21.15.114:0/616409090 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 10 monclient(hunting): get_auth_request con 0x7fc2740a5d00 auth_method 0 2024-09-16T05:10:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 10 monclient(hunting): handle_auth_done global_id 27528 payload 274 2024-09-16T05:10:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 1 -- 172.21.15.114:0/616409090 >> v1:172.21.15.149:6789/0 conn(0x7fc274137730 legacy=0x7fc274137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 1 -- 172.21.15.114:0/616409090 >> v1:172.21.15.114:6789/0 conn(0x7fc27413ca00 legacy=0x7fc27413edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:10:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 1 monclient: found mon.noname-a 2024-09-16T05:10:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:10:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.311+0000 7fc26bfff640 1 -- 172.21.15.114:0/616409090 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc27409be00 con 0x7fc2740a5d00 2024-09-16T05:10:01.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26bfff640 1 --2- 172.21.15.114:0/616409090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2740a5d00 0x7fc27413c4c0 secure :-1 s=READY pgs=8163 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009870 tx=0x7fc25c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6564df0bed8576c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc271d9b640 1 -- 172.21.15.114:0/616409090 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c0b0070 con 0x7fc2740a5d00 2024-09-16T05:10:01.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc271d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc271d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:10:01.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc271d9b640 10 monclient: dump: 2024-09-16T05:10:01.326 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.326 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.326 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.326 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.326 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.327 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.327 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.327 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.327 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.327 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc271d9b640 10 monclient: _finish_auth 0 2024-09-16T05:10:01.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc271d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:10:01.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc271d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:01.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc271d9b640 1 -- 172.21.15.114:0/616409090 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc25c0ab600 con 0x7fc2740a5d00 2024-09-16T05:10:01.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 5 monclient: authenticate success, global_id 27528 2024-09-16T05:10:01.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:01.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc271d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:01.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 10 monclient: get_monmap_and_config success 2024-09-16T05:10:01.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc271d9b640 1 -- 172.21.15.114:0/616409090 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c0ab9d0 con 0x7fc2740a5d00 2024-09-16T05:10:01.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc271d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 4 set_mon_vals no callback set 2024-09-16T05:10:01.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc271d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:01.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc271d9b640 10 monclient: dump: 2024-09-16T05:10:01.328 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.328 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.328 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.328 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.328 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.328 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.328 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.328 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.328 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.328 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 10 monclient: shutdown 2024-09-16T05:10:01.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:01.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 -- 172.21.15.114:0/616409090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2740a5d00 msgr2=0x7fc27413c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 --2- 172.21.15.114:0/616409090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2740a5d00 0x7fc27413c4c0 secure :-1 s=READY pgs=8163 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009870 tx=0x7fc25c01c440 comp rx=0 tx=0).stop 2024-09-16T05:10:01.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 -- 172.21.15.114:0/616409090 shutdown_connections 2024-09-16T05:10:01.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 --2- 172.21.15.114:0/616409090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2740a5d00 0x7fc27413c4c0 unknown :-1 s=CLOSED pgs=8163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 -- 172.21.15.114:0/616409090 >> 172.21.15.114:0/616409090 conn(0x7fc2740a9c70 msgr2=0x7fc2740aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:01.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 -- 172.21.15.114:0/616409090 shutdown_connections 2024-09-16T05:10:01.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 -- 172.21.15.114:0/616409090 wait complete. 2024-09-16T05:10:01.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 10 monclient: build_initial_monmap 2024-09-16T05:10:01.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 10 monclient: monmap: 2024-09-16T05:10:01.329 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:01.329 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:10:01.329 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:10:01.330 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:10:01.330 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:01.330 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.330 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:10:01.330 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:10:01.330 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:10:01.330 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 librados: starting msgr at 2024-09-16T05:10:01.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 librados: starting objecter 2024-09-16T05:10:01.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 Processor -- start 2024-09-16T05:10:01.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 -- start start 2024-09-16T05:10:01.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 librados: setting wanted keys 2024-09-16T05:10:01.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 librados: calling monclient init 2024-09-16T05:10:01.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 10 monclient: init 2024-09-16T05:10:01.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:01.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:10:01.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2740a5d00 0x7fc274135b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 10 monclient(hunting): picked mon.noname-a con 0x7fc2740a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:01.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc274137730 0x7fc274136090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 10 monclient(hunting): picked mon.noname-c con 0x7fc274137730 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:01.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc27413ca00 0x7fc27409e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 10 monclient(hunting): picked mon.noname-b con 0x7fc27413ca00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:01.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc274142eb0 con 0x7fc2740a5d00 2024-09-16T05:10:01.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fc274142d30 con 0x7fc27413ca00 2024-09-16T05:10:01.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc274143030 con 0x7fc274137730 2024-09-16T05:10:01.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:01.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 10 monclient(hunting): authenticate will time out at 1085.946889s 2024-09-16T05:10:01.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2740a5d00 0x7fc274135b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc274137730 0x7fc274136090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2740a5d00 0x7fc274135b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59492/0 (socket says 172.21.15.114:59492) 2024-09-16T05:10:01.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 1 -- 172.21.15.114:0/1592871713 learned_addr learned my addr 172.21.15.114:0/1592871713 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:01.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 10 monclient(hunting): get_auth_request con 0x7fc2740a5d00 auth_method 0 2024-09-16T05:10:01.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26bfff640 10 monclient(hunting): get_auth_request con 0x7fc274137730 auth_method 0 2024-09-16T05:10:01.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 10 monclient(hunting): handle_auth_done global_id 27534 payload 824 2024-09-16T05:10:01.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc272d9d640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc27413ca00 0x7fc27409e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 1 -- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc274137730 msgr2=0x7fc274136090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc274137730 0x7fc274136090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 1 -- 172.21.15.114:0/1592871713 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc27413ca00 msgr2=0x7fc27409e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc27413ca00 0x7fc27409e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 10 monclient: _finish_hunting 0 2024-09-16T05:10:01.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 1 monclient: found mon.noname-a 2024-09-16T05:10:01.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:01.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:10:01.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 1 -- 172.21.15.114:0/1592871713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2740a25c0 con 0x7fc2740a5d00 2024-09-16T05:10:01.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26bfff640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc274137730 0x7fc274136090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:10:01.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc272d9d640 10 monclient: get_auth_request con 0x7fc27413ca00 auth_method 0 2024-09-16T05:10:01.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc272d9d640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc27413ca00 0x7fc27409e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:10:01.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27259c640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2740a5d00 0x7fc274135b50 secure :-1 s=READY pgs=8164 cs=0 l=1 rev1=1 crypto rx=0x7fc26400c450 tx=0x7fc26400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26affd640 10 client.?.objecter ms_handle_connect 0x7fc2740a5d00 2024-09-16T05:10:01.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26affd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:10:01.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26affd640 1 -- 172.21.15.114:0/1592871713 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc264012830 con 0x7fc2740a5d00 2024-09-16T05:10:01.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:10:01.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26affd640 10 monclient: dump: 2024-09-16T05:10:01.335 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.335 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.335 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.335 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.335 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.335 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.335 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.336 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.336 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.336 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26affd640 10 monclient: _finish_auth 0 2024-09-16T05:10:01.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26affd640 10 monclient: _check_auth_tickets 2024-09-16T05:10:01.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:01.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26affd640 1 -- 172.21.15.114:0/1592871713 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc264017070 con 0x7fc2740a5d00 2024-09-16T05:10:01.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc278ed9640 5 monclient: authenticate success, global_id 27534 2024-09-16T05:10:01.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:01.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26affd640 1 -- 172.21.15.114:0/1592871713 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc264012c30 con 0x7fc2740a5d00 2024-09-16T05:10:01.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:01.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc26affd640 10 monclient: dump: 2024-09-16T05:10:01.336 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.337 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.337 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.337 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.337 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.337 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.337 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.337 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.337 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.337 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.315+0000 7fc27159a640 4 set_mon_vals no callback set 2024-09-16T05:10:01.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc278ed9640 10 monclient: _renew_subs 2024-09-16T05:10:01.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc278ed9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:01.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc278ed9640 1 -- 172.21.15.114:0/1592871713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2740a0e90 con 0x7fc2740a5d00 2024-09-16T05:10:01.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc278ed9640 10 client.27534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:10:01.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc278ed9640 10 monclient: _renew_subs 2024-09-16T05:10:01.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc278ed9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:01.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc278ed9640 1 -- 172.21.15.114:0/1592871713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2740a8c50 con 0x7fc2740a5d00 2024-09-16T05:10:01.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc278ed9640 1 librados: init done 2024-09-16T05:10:01.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc278ed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:01.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc278ed9640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc274062ea0 0x7fc274063280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc278ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc278ed9640 1 -- 172.21.15.114:0/1592871713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc274137b10 con 0x7fc274062ea0 2024-09-16T05:10:01.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26bfff640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc274062ea0 0x7fc274063280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26bfff640 10 monclient: get_auth_request con 0x7fc274062ea0 auth_method 0 2024-09-16T05:10:01.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 1 -- 172.21.15.114:0/1592871713 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc26401c460 con 0x7fc2740a5d00 2024-09-16T05:10:01.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26bfff640 10 monclient(hunting): handle_auth_done global_id 27537 payload 824 2024-09-16T05:10:01.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26bfff640 1 -- 172.21.15.114:0/1592871713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc274073820 con 0x7fc274062ea0 2024-09-16T05:10:01.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 1 --2- 172.21.15.114:0/1592871713 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc258037810 0x7fc258039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 1 -- 172.21.15.114:0/1592871713 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7398+0+0 (secure 0 0 0) 0x7fc264049280 con 0x7fc2740a5d00 2024-09-16T05:10:01.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 10 client.27534.objecter ms_dispatch 0x7fc274000ba0 osd_map(80..80 src has 1..80) 2024-09-16T05:10:01.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 3 client.27534.objecter handle_osd_map got epochs [80,80] > 0 2024-09-16T05:10:01.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 3 client.27534.objecter handle_osd_map decoding full epoch 80 2024-09-16T05:10:01.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26bfff640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc274062ea0 0x7fc274063280 secure :-1 s=READY pgs=8165 cs=0 l=1 rev1=1 crypto rx=0x7fc2740a60e0 tx=0x7fc25c0191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 20 client.27534.objecter dump_active .. 0 homeless 2024-09-16T05:10:01.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 10 client.27534.objecter ms_handle_connect 0x7fc274062ea0 2024-09-16T05:10:01.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 10 client.27534.objecter resend_mon_ops 2024-09-16T05:10:01.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 1 -- 172.21.15.114:0/1592871713 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c0b0040 con 0x7fc274062ea0 2024-09-16T05:10:01.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:01.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 10 monclient: dump: 2024-09-16T05:10:01.340 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.340 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.340 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.340 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.340 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.340 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.340 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.340 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.341 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.341 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc272d9d640 1 --2- 172.21.15.114:0/1592871713 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc258037810 0x7fc258039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc272d9d640 10 monclient: get_auth_request con 0x7fc258037810 auth_method 0 2024-09-16T05:10:01.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 1 -- 172.21.15.114:0/1592871713 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc25c0b5640 con 0x7fc274062ea0 2024-09-16T05:10:01.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:01.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:10:01.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 1 -- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc274062ea0 msgr2=0x7fc274063280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc274062ea0 0x7fc274063280 secure :-1 s=READY pgs=8165 cs=0 l=1 rev1=1 crypto rx=0x7fc2740a60e0 tx=0x7fc25c0191e0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 1 -- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc274062ea0 msgr2=0x7fc274063280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:01.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc274062ea0 0x7fc274063280 unknown :-1 s=CLOSED pgs=8165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc26affd640 10 client.27534.objecter ms_handle_connect 0x7fc258037810 2024-09-16T05:10:01.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.319+0000 7fc272d9d640 1 --2- 172.21.15.114:0/1592871713 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc258037810 0x7fc258039cd0 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7fc26c007fd0 tx=0x7fc26c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.323+0000 7fc278ed9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:10:01.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.323+0000 7fc278ed9640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc27402c650 0x7fc27402ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.323+0000 7fc278ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.323+0000 7fc278ed9640 1 -- 172.21.15.114:0/1592871713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc27413a970 con 0x7fc27402c650 2024-09-16T05:10:01.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.323+0000 7fc26bfff640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc27402c650 0x7fc27402ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.323+0000 7fc26bfff640 10 monclient: get_auth_request con 0x7fc27402c650 auth_method 0 2024-09-16T05:10:01.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26bfff640 10 monclient(hunting): handle_auth_done global_id 27540 payload 824 2024-09-16T05:10:01.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26bfff640 1 -- 172.21.15.114:0/1592871713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc2741365d0 con 0x7fc27402c650 2024-09-16T05:10:01.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26bfff640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc27402c650 0x7fc27402ca60 secure :-1 s=READY pgs=8166 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009870 tx=0x7fc25c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26affd640 10 client.27534.objecter ms_handle_connect 0x7fc27402c650 2024-09-16T05:10:01.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26affd640 10 client.27534.objecter resend_mon_ops 2024-09-16T05:10:01.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26affd640 1 -- 172.21.15.114:0/1592871713 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c0c5070 con 0x7fc27402c650 2024-09-16T05:10:01.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:01.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26affd640 10 monclient: dump: 2024-09-16T05:10:01.345 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.345 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.345 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.345 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.345 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.345 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.345 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.345 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.346 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.346 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26affd640 1 -- 172.21.15.114:0/1592871713 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7fc25c003c60 con 0x7fc27402c650 2024-09-16T05:10:01.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:10:01.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:10:01.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26affd640 1 -- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc27402c650 msgr2=0x7fc27402ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26affd640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc27402c650 0x7fc27402ca60 secure :-1 s=READY pgs=8166 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009870 tx=0x7fc25c009240 comp rx=0 tx=0).stop 2024-09-16T05:10:01.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26affd640 1 -- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc27402c650 msgr2=0x7fc27402ca60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:01.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc26affd640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc27402c650 0x7fc27402ca60 secure :-1 s=CLOSED pgs=8166 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009870 tx=0x7fc25c009240 comp rx=0 tx=0).stop 2024-09-16T05:10:01.346 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:10:01.346 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:10:01.347 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:10:01.347 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:10:01.347 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 14, 2024-09-16T05:10:01.347 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 159316, 2024-09-16T05:10:01.347 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:10:01.347 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:10:01.347 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:10:01.347 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:10:01.347 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:01.347 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:10:01.347 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:01.347 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:10:01.347 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:10:01.348 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:01.348 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:01.348 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:01.348 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:01.348 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:01.348 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:01.348 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:01.348 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:01.348 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:01.348 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:01.348 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:01.348 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:01.348 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:10:01.348 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:10:01.349 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:01.349 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:01.349 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:01.349 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:01.349 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:01.349 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:01.349 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:01.349 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:01.349 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:01.349 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:01.349 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:01.349 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.350 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:10:01.350 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:10:01.350 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:10:01.350 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:10:01.350 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:10:01.350 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:10:01.350 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:01.350 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:01.350 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:10:01.350 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:10:01.350 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:10:01.350 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:10:01.350 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:10:01.350 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:10:01.351 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:10:01.351 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:01.351 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:10:01.351 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:01.351 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:01.351 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:01.351 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:01.351 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:01.351 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:01.351 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:01.351 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:01.351 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:01.351 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:01.351 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:01.352 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:01.352 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:10:01.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.352 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:10:01.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.352 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:10:01.352 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:10:01.352 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:01.352 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:01.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.352 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:01.352 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:10:01.352 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.353 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:01.353 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:10:01.353 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.353 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.353 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.353 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:10:01.353 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:10:01.353 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:01.353 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:01.353 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:01.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.354 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:10:01.354 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:10:01.354 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:01.354 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:01.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.354 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:01.354 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:10:01.354 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.354 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:01.354 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:10:01.354 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.354 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.355 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.355 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:10:01.355 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:10:01.355 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:01.355 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:01.355 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:01.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.355 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:10:01.355 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:10:01.355 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:01.355 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:01.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.356 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:01.356 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:10:01.356 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.356 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:01.356 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:10:01.356 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.356 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.356 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.356 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:10:01.356 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:10:01.356 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:01.357 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:01.357 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:01.357 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.357 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.357 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:10:01.357 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:10:01.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.357 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:01.357 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:01.357 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:10:01.357 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.357 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:01.357 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:10:01.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.358 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:01.358 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:01.358 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:10:01.358 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.358 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:01.358 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:10:01.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.358 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:01.358 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:01.358 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:10:01.358 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.358 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.359 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:10:01.359 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:10:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc278ed9640 10 librados: watch_flush enter 2024-09-16T05:10:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc278ed9640 10 librados: watch_flush exit 2024-09-16T05:10:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc278ed9640 20 client.27534.objecter shutdown clearing up homeless session... 2024-09-16T05:10:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc278ed9640 10 client.27534.objecter successfully canceled tick 2024-09-16T05:10:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc278ed9640 1 -- 172.21.15.114:0/1592871713 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc258037810 msgr2=0x7fc258039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc278ed9640 1 --2- 172.21.15.114:0/1592871713 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc258037810 0x7fc258039cd0 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7fc26c007fd0 tx=0x7fc26c009040 comp rx=0 tx=0).stop 2024-09-16T05:10:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc278ed9640 10 monclient: shutdown 2024-09-16T05:10:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc278ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc278ed9640 1 -- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2740a5d00 msgr2=0x7fc274135b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc278ed9640 1 --2- 172.21.15.114:0/1592871713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2740a5d00 0x7fc274135b50 secure :-1 s=READY pgs=8164 cs=0 l=1 rev1=1 crypto rx=0x7fc26400c450 tx=0x7fc26400c910 comp rx=0 tx=0).stop 2024-09-16T05:10:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc272d9d640 1 -- 172.21.15.114:0/1592871713 reap_dead start 2024-09-16T05:10:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc272d9d640 1 -- 172.21.15.114:0/1592871713 reap_dead start 2024-09-16T05:10:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc278ed9640 1 -- 172.21.15.114:0/1592871713 shutdown_connections 2024-09-16T05:10:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc278ed9640 1 -- 172.21.15.114:0/1592871713 >> 172.21.15.114:0/1592871713 conn(0x7fc2740a9c70 msgr2=0x7fc2740a84a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc278ed9640 1 -- 172.21.15.114:0/1592871713 shutdown_connections 2024-09-16T05:10:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc278ed9640 1 -- 172.21.15.114:0/1592871713 wait complete. 2024-09-16T05:10:01.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.327+0000 7fc278ed9640 1 librados: shutdown 2024-09-16T05:10:01.364 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:10:01.364 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:10:01.364 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:10:01.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 10 monclient: get_monmap_and_config 2024-09-16T05:10:01.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 10 monclient: build_initial_monmap 2024-09-16T05:10:01.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 10 monclient: monmap: 2024-09-16T05:10:01.477 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:01.478 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.478 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:10:01.476747+0000 2024-09-16T05:10:01.478 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:10:01.476747+0000 2024-09-16T05:10:01.478 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:01.478 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.478 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:10:01.478 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:10:01.478 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:10:01.478 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:10:01.478 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:10:01.478 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 1 Processor -- start 2024-09-16T05:10:01.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 1 -- start start 2024-09-16T05:10:01.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 10 monclient: init 2024-09-16T05:10:01.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:01.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-16T05:10:01.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f66a41376e0 0x7f66a4137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 10 monclient(hunting): picked mon.noname-c con 0x7f66a41376e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:10:01.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66a41370a0 addr v1:172.21.15.114:6789/0 2024-09-16T05:10:01.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66a40a5d00 addr v1:172.21.15.149:6789/0 2024-09-16T05:10:01.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66a40a5090 con 0x7f66a41370a0 2024-09-16T05:10:01.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66a409bd20 con 0x7f66a40a5d00 2024-09-16T05:10:01.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f66a4138000 con 0x7f66a41376e0 2024-09-16T05:10:01.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:01.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 10 monclient(hunting): authenticate will time out at 1086.106506s 2024-09-16T05:10:01.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a8e39640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f66a41370a0 0x7f66a40a4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36662/0 (socket says 172.21.15.114:36662) 2024-09-16T05:10:01.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a8e39640 1 -- 172.21.15.114:0/964544586 learned_addr learned my addr 172.21.15.114:0/964544586 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:10:01.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 1 --2- 172.21.15.114:0/964544586 >> v2:172.21.15.149:3300/0 conn(0x7f66a41376e0 0x7f66a4137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient(hunting): get_auth_request con 0x7f66a41376e0 auth_method 0 2024-09-16T05:10:01.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 -- 172.21.15.114:0/964544586 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2004701759 0 0) 0x7f66a40a5090 con 0x7f66a41370a0 2024-09-16T05:10:01.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:10:01.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient(hunting): my global_id is 27543 2024-09-16T05:10:01.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 -- 172.21.15.114:0/964544586 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6690003530 con 0x7f66a41370a0 2024-09-16T05:10:01.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 -- 172.21.15.114:0/964544586 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2702214357 0 0) 0x7f6690003530 con 0x7f66a41370a0 2024-09-16T05:10:01.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 -- 172.21.15.114:0/964544586 >> v2:172.21.15.149:3300/0 conn(0x7f66a41376e0 msgr2=0x7f66a4137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:01.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 --2- 172.21.15.114:0/964544586 >> v2:172.21.15.149:3300/0 conn(0x7f66a41376e0 0x7f66a4137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 -- 172.21.15.114:0/964544586 >> v1:172.21.15.149:6789/0 conn(0x7f66a40a5d00 legacy=0x7f66a40a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:10:01.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:10:01.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:01.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:10:01.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 -- 172.21.15.114:0/964544586 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a409c000 con 0x7f66a41370a0 2024-09-16T05:10:01.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient: _finish_auth 0 2024-09-16T05:10:01.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:10:01.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:01.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 -- 172.21.15.114:0/964544586 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f6694001680 con 0x7f66a41370a0 2024-09-16T05:10:01.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 5 monclient: authenticate success, global_id 27543 2024-09-16T05:10:01.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:01.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:10:01.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient: dump: 2024-09-16T05:10:01.486 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.486 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.486 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.486 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.486 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.486 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.486 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.486 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.486 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.486 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:10:01.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:01.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 -- 172.21.15.114:0/964544586 >> v1:172.21.15.114:6789/0 conn(0x7f66a41370a0 legacy=0x7f66a40a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:10:01.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 --2- 172.21.15.114:0/964544586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66900039c0 0x7f6690003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient(hunting): picked mon.a con 0x7f66900039c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:01.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 --2- 172.21.15.114:0/964544586 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6690004810 0x7f6690004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient(hunting): picked mon.b con 0x7f6690004810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:01.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 --2- 172.21.15.114:0/964544586 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66900052f0 0x7f669000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient(hunting): picked mon.c con 0x7f66900052f0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:01.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 -- 172.21.15.114:0/964544586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f669000c190 con 0x7f66900039c0 2024-09-16T05:10:01.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 -- 172.21.15.114:0/964544586 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f669000c340 con 0x7f6690004810 2024-09-16T05:10:01.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 1 -- 172.21.15.114:0/964544586 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f669000c540 con 0x7f66900052f0 2024-09-16T05:10:01.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:01.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f669bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:10:01.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66ab89e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:01.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 1 --2- 172.21.15.114:0/964544586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66900039c0 0x7f6690003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient(hunting): get_auth_request con 0x7f66900039c0 auth_method 0 2024-09-16T05:10:01.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:01.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a9e3b640 1 --2- 172.21.15.114:0/964544586 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66900052f0 0x7f669000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:10:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a9e3b640 10 monclient(hunting): get_auth_request con 0x7f66900052f0 auth_method 0 2024-09-16T05:10:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a9e3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a9e3b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a9e3b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a9e3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a9e3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a9e3b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:10:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a8e39640 1 --2- 172.21.15.114:0/964544586 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6690004810 0x7f6690004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient(hunting): handle_auth_done global_id 27543 payload 293 2024-09-16T05:10:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 1 -- 172.21.15.114:0/964544586 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66900052f0 msgr2=0x7f669000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 1 --2- 172.21.15.114:0/964544586 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66900052f0 0x7f669000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 1 -- 172.21.15.114:0/964544586 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6690004810 msgr2=0x7f6690004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 1 --2- 172.21.15.114:0/964544586 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6690004810 0x7f6690004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient: _finish_hunting 0 2024-09-16T05:10:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 1 monclient: found mon.a 2024-09-16T05:10:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 1 -- 172.21.15.114:0/964544586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6690003530 con 0x7f66900039c0 2024-09-16T05:10:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient: _finish_auth 0 2024-09-16T05:10:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 10 monclient: _check_auth_tickets 2024-09-16T05:10:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a963a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.475+0000 7f66a9e3b640 1 -- 172.21.15.114:0/964544586 reap_dead start 2024-09-16T05:10:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66a963a640 1 --2- 172.21.15.114:0/964544586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66900039c0 0x7f6690003de0 secure :-1 s=READY pgs=8168 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009ab0 tx=0x7f66a001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45eb335e1147417d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f669bfff640 1 -- 172.21.15.114:0/964544586 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a00096b0 con 0x7f66900039c0 2024-09-16T05:10:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f669bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f669bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f669bfff640 10 monclient: dump: 2024-09-16T05:10:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.499 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f669bfff640 1 -- 172.21.15.114:0/964544586 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66a00a6410 con 0x7f66900039c0 2024-09-16T05:10:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f669bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f669bfff640 1 -- 172.21.15.114:0/964544586 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a00a6800 con 0x7f66900039c0 2024-09-16T05:10:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f669bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f669bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f669bfff640 10 monclient: dump: 2024-09-16T05:10:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.500 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 10 monclient: get_monmap_and_config success 2024-09-16T05:10:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 4 set_mon_vals no callback set 2024-09-16T05:10:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 10 monclient: shutdown 2024-09-16T05:10:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 1 -- 172.21.15.114:0/964544586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66900039c0 msgr2=0x7f6690003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 1 --2- 172.21.15.114:0/964544586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66900039c0 0x7f6690003de0 secure :-1 s=READY pgs=8168 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009ab0 tx=0x7f66a001ca30 comp rx=0 tx=0).stop 2024-09-16T05:10:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 1 -- 172.21.15.114:0/964544586 shutdown_connections 2024-09-16T05:10:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 1 --2- 172.21.15.114:0/964544586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66900039c0 0x7f6690003de0 unknown :-1 s=CLOSED pgs=8168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 1 -- 172.21.15.114:0/964544586 >> 172.21.15.114:0/964544586 conn(0x7f66a40a9a40 msgr2=0x7f66a40a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 1 -- 172.21.15.114:0/964544586 shutdown_connections 2024-09-16T05:10:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 1 -- 172.21.15.114:0/964544586 wait complete. 2024-09-16T05:10:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 10 monclient: build_initial_monmap 2024-09-16T05:10:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 10 monclient: monmap: 2024-09-16T05:10:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:10:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:10:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:10:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:10:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:10:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:10:01.502 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 1 librados: starting msgr at 2024-09-16T05:10:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 1 librados: starting objecter 2024-09-16T05:10:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 1 Processor -- start 2024-09-16T05:10:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 1 -- start start 2024-09-16T05:10:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 1 librados: setting wanted keys 2024-09-16T05:10:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 1 librados: calling monclient init 2024-09-16T05:10:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.479+0000 7f66ab89e640 10 monclient: init 2024-09-16T05:10:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:10:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66900039c0 0x7f66a41cf540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 monclient(hunting): picked mon.noname-c con 0x7f66900039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66a41cfa80 0x7f66a4124a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 monclient(hunting): picked mon.noname-a con 0x7f66a41cfa80 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f66a41cfe80 0x7f66a4124f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 monclient(hunting): picked mon.noname-b con 0x7f66a41cfe80 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f66a40a4e10 con 0x7f66a41cfa80 2024-09-16T05:10:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f66a40a4c90 con 0x7f66a41cfe80 2024-09-16T05:10:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f66a409c5c0 con 0x7f66900039c0 2024-09-16T05:10:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 monclient(hunting): authenticate will time out at 1086.112278s 2024-09-16T05:10:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66a41cfa80 0x7f66a4124a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66900039c0 0x7f66a41cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66a41cfa80 0x7f66a4124a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59534/0 (socket says 172.21.15.114:59534) 2024-09-16T05:10:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 1 -- 172.21.15.114:0/782427918 learned_addr learned my addr 172.21.15.114:0/782427918 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 10 monclient(hunting): get_auth_request con 0x7f66900039c0 auth_method 0 2024-09-16T05:10:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 10 monclient(hunting): get_auth_request con 0x7f66a41cfa80 auth_method 0 2024-09-16T05:10:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a9e3b640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f66a41cfe80 0x7f66a4124f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a9e3b640 10 monclient(hunting): get_auth_request con 0x7f66a41cfe80 auth_method 0 2024-09-16T05:10:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a9e3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a9e3b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a9e3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 10 monclient(hunting): handle_auth_done global_id 27546 payload 824 2024-09-16T05:10:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 1 -- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66900039c0 msgr2=0x7f66a41cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66900039c0 0x7f66a41cf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 1 -- 172.21.15.114:0/782427918 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f66a41cfe80 msgr2=0x7f66a4124f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f66a41cfe80 0x7f66a4124f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 10 monclient: _finish_hunting 0 2024-09-16T05:10:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 1 monclient: found mon.noname-a 2024-09-16T05:10:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:10:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 1 -- 172.21.15.114:0/782427918 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a41283d0 con 0x7f66a41cfa80 2024-09-16T05:10:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66900039c0 0x7f66a41cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:10:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a9e3b640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f66a41cfe80 0x7f66a4124f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:10:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a8e39640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66a41cfa80 0x7f66a4124a00 secure :-1 s=READY pgs=8169 cs=0 l=1 rev1=1 crypto rx=0x7f6694007fd0 tx=0x7f669400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 client.?.objecter ms_handle_connect 0x7f66a41cfa80 2024-09-16T05:10:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:10:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 1 -- 172.21.15.114:0/782427918 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6694019070 con 0x7f66a41cfa80 2024-09-16T05:10:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:10:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 monclient: dump: 2024-09-16T05:10:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.508 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 monclient: _finish_auth 0 2024-09-16T05:10:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:10:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 1 -- 172.21.15.114:0/782427918 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66940072d0 con 0x7f66a41cfa80 2024-09-16T05:10:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 5 monclient: authenticate success, global_id 27546 2024-09-16T05:10:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 monclient: _renew_subs 2024-09-16T05:10:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 1 -- 172.21.15.114:0/782427918 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66a41286b0 con 0x7f66a41cfa80 2024-09-16T05:10:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 1 -- 172.21.15.114:0/782427918 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66940047f0 con 0x7f66a41cfa80 2024-09-16T05:10:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 monclient: dump: 2024-09-16T05:10:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.509 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f669b7fe640 4 set_mon_vals no callback set 2024-09-16T05:10:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 client.27546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:10:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 monclient: _renew_subs 2024-09-16T05:10:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 1 -- 172.21.15.114:0/782427918 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a40a1fe0 con 0x7f66a41cfa80 2024-09-16T05:10:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 1 librados: init done 2024-09-16T05:10:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66a402c650 0x7f66a402ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66ab89e640 1 -- 172.21.15.114:0/782427918 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f66a40a5020 con 0x7f66a402c650 2024-09-16T05:10:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66a402c650 0x7f66a402ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 1 -- 172.21.15.114:0/782427918 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f669401c070 con 0x7f66a41cfa80 2024-09-16T05:10:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 10 monclient: get_auth_request con 0x7f66a402c650 auth_method 0 2024-09-16T05:10:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 10 monclient(hunting): handle_auth_done global_id 54119 payload 824 2024-09-16T05:10:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 1 --2- 172.21.15.114:0/782427918 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6688037840 0x7f6688039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 1 -- 172.21.15.114:0/782427918 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f66a41215d0 con 0x7f66a402c650 2024-09-16T05:10:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 1 -- 172.21.15.114:0/782427918 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7398+0+0 (secure 0 0 0) 0x7f6694048e90 con 0x7f66a41cfa80 2024-09-16T05:10:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 client.27546.objecter ms_dispatch 0x7f66a4000ba0 osd_map(80..80 src has 1..80) 2024-09-16T05:10:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 3 client.27546.objecter handle_osd_map got epochs [80,80] > 0 2024-09-16T05:10:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 3 client.27546.objecter handle_osd_map decoding full epoch 80 2024-09-16T05:10:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 20 client.27546.objecter dump_active .. 0 homeless 2024-09-16T05:10:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 client.27546.objecter ms_handle_connect 0x7f66a402c650 2024-09-16T05:10:01.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 client.27546.objecter resend_mon_ops 2024-09-16T05:10:01.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a963a640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66a402c650 0x7f66a402ca30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f66900057f0 tx=0x7f66a00197d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 1 -- 172.21.15.114:0/782427918 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a00c1070 con 0x7f66a402c650 2024-09-16T05:10:01.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:01.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 monclient: dump: 2024-09-16T05:10:01.512 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.512 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.512 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.513 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a9e3b640 1 --2- 172.21.15.114:0/782427918 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6688037840 0x7f6688039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a9e3b640 10 monclient: get_auth_request con 0x7f6688037840 auth_method 0 2024-09-16T05:10:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 1 -- 172.21.15.114:0/782427918 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f66a00a6410 con 0x7f66a402c650 2024-09-16T05:10:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:10:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 1 -- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66a402c650 msgr2=0x7f66a402ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66a402c650 0x7f66a402ca30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f66900057f0 tx=0x7f66a00197d0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 1 -- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66a402c650 msgr2=0x7f66a402ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66a402c650 0x7f66a402ca30 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f66900057f0 tx=0x7f66a00197d0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f66a9e3b640 1 --2- 172.21.15.114:0/782427918 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6688037840 0x7f6688039d00 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7f66a40aa8e0 tx=0x7f669c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.483+0000 7f6699ffb640 10 client.27546.objecter ms_handle_connect 0x7f6688037840 2024-09-16T05:10:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:10:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66a402cf70 0x7f66a402d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 1 -- 172.21.15.114:0/782427918 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f66a4068820 con 0x7f66a402cf70 2024-09-16T05:10:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66a963a640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66a402cf70 0x7f66a402d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66a963a640 10 monclient: get_auth_request con 0x7f66a402cf70 auth_method 0 2024-09-16T05:10:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66a963a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66a963a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66a963a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66a963a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66a963a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66a963a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66a963a640 10 monclient(hunting): handle_auth_done global_id 54122 payload 824 2024-09-16T05:10:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66a963a640 1 -- 172.21.15.114:0/782427918 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f66a41280f0 con 0x7f66a402cf70 2024-09-16T05:10:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f6699ffb640 10 client.27546.objecter ms_handle_connect 0x7f66a402cf70 2024-09-16T05:10:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f6699ffb640 10 client.27546.objecter resend_mon_ops 2024-09-16T05:10:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66a963a640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66a402cf70 0x7f66a402d380 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009ab0 tx=0x7f66a0009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f6699ffb640 1 -- 172.21.15.114:0/782427918 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a00c1040 con 0x7f66a402cf70 2024-09-16T05:10:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f6699ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f6699ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f6699ffb640 10 monclient: dump: 2024-09-16T05:10:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.516 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f6699ffb640 1 -- 172.21.15.114:0/782427918 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f66a00c3620 con 0x7f66a402cf70 2024-09-16T05:10:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f6699ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:10:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f6699ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:10:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f6699ffb640 1 -- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66a402cf70 msgr2=0x7f66a402d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f6699ffb640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66a402cf70 0x7f66a402d380 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009ab0 tx=0x7f66a0009190 comp rx=0 tx=0).stop 2024-09-16T05:10:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f6699ffb640 1 -- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66a402cf70 msgr2=0x7f66a402d380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f6699ffb640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f66a402cf70 0x7f66a402d380 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009ab0 tx=0x7f66a0009190 comp rx=0 tx=0).stop 2024-09-16T05:10:01.517 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:10:01.517 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:10:01.517 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:10:01.517 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:10:01.517 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 14, 2024-09-16T05:10:01.517 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 15627, 2024-09-16T05:10:01.517 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:10:01.517 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:10:01.518 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:10:01.518 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:10:01.518 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:01.518 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:10:01.518 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:01.518 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:10:01.518 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:10:01.518 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:01.518 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:01.518 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:01.518 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:01.518 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:01.518 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:01.519 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:01.519 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:01.519 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:01.519 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:01.519 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:01.519 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:01.519 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:10:01.519 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:10:01.519 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:01.519 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:01.519 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:01.519 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:01.519 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:01.519 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:01.520 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:01.520 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:01.520 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:01.520 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:01.520 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:01.520 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.520 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.520 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:10:01.520 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:10:01.520 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:10:01.520 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:10:01.521 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:10:01.521 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:10:01.521 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:01.521 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:01.521 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:10:01.521 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:10:01.521 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:10:01.521 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:10:01.521 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:10:01.521 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:10:01.521 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:10:01.521 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:01.521 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:10:01.521 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:01.522 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:01.522 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:01.522 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:01.522 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:01.522 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:01.522 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:01.522 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:01.522 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:01.522 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:01.522 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:01.522 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:01.522 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:10:01.522 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.522 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:10:01.523 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.523 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:10:01.523 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:10:01.523 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:01.523 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:01.523 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.523 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:01.523 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:10:01.523 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.523 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.523 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.523 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:01.523 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:10:01.523 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.524 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.524 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.524 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.524 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:10:01.524 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:10:01.524 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:01.524 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:01.524 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:01.524 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.524 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.524 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:10:01.524 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:10:01.524 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:01.525 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:01.525 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.525 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:01.525 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:10:01.525 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.525 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.525 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.525 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:01.525 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:10:01.525 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.525 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.525 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.525 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.525 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:10:01.526 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:10:01.526 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:01.526 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:01.526 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:01.526 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.526 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.526 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:10:01.526 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:10:01.526 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:01.526 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:01.526 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.526 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:01.526 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:10:01.526 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.527 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.527 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.527 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:01.527 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:10:01.527 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.527 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.527 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.527 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.527 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:10:01.527 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:10:01.527 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:01.527 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:01.527 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:01.527 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.528 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.528 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.528 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:10:01.528 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:10:01.528 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.528 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:01.528 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:01.528 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:10:01.528 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.528 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:01.528 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:10:01.528 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.528 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:01.528 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:01.529 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:10:01.529 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.529 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.529 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.529 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:10:01.529 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:10:01.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 10 librados: watch_flush enter 2024-09-16T05:10:01.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 10 librados: watch_flush exit 2024-09-16T05:10:01.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 20 client.27546.objecter shutdown clearing up homeless session... 2024-09-16T05:10:01.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 10 client.27546.objecter successfully canceled tick 2024-09-16T05:10:01.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 1 -- 172.21.15.114:0/782427918 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6688037840 msgr2=0x7f6688039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 1 --2- 172.21.15.114:0/782427918 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6688037840 0x7f6688039d00 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7f66a40aa8e0 tx=0x7f669c009040 comp rx=0 tx=0).stop 2024-09-16T05:10:01.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 10 monclient: shutdown 2024-09-16T05:10:01.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:01.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 1 -- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66a41cfa80 msgr2=0x7f66a4124a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 1 --2- 172.21.15.114:0/782427918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66a41cfa80 0x7f66a4124a00 secure :-1 s=READY pgs=8169 cs=0 l=1 rev1=1 crypto rx=0x7f6694007fd0 tx=0x7f669400dc00 comp rx=0 tx=0).stop 2024-09-16T05:10:01.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66a9e3b640 1 -- 172.21.15.114:0/782427918 reap_dead start 2024-09-16T05:10:01.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66a9e3b640 1 -- 172.21.15.114:0/782427918 reap_dead start 2024-09-16T05:10:01.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 1 -- 172.21.15.114:0/782427918 shutdown_connections 2024-09-16T05:10:01.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 1 -- 172.21.15.114:0/782427918 >> 172.21.15.114:0/782427918 conn(0x7f66a40a9a40 msgr2=0x7f66a40a0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:01.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 1 -- 172.21.15.114:0/782427918 shutdown_connections 2024-09-16T05:10:01.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 1 -- 172.21.15.114:0/782427918 wait complete. 2024-09-16T05:10:01.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.491+0000 7f66ab89e640 1 librados: shutdown 2024-09-16T05:10:01.532 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:10:01.532 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:10:01.533 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:10:01.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.639+0000 7f759e3a6640 10 monclient: get_monmap_and_config 2024-09-16T05:10:01.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.639+0000 7f759e3a6640 10 monclient: build_initial_monmap 2024-09-16T05:10:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.639+0000 7f759e3a6640 10 monclient: monmap: 2024-09-16T05:10:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:10:01.644206+0000 2024-09-16T05:10:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:10:01.644206+0000 2024-09-16T05:10:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:10:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:10:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:10:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:10:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:10:01.646 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 1 Processor -- start 2024-09-16T05:10:01.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 1 -- start start 2024-09-16T05:10:01.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 10 monclient: init 2024-09-16T05:10:01.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:01.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T05:10:01.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75980a0ec0 0x7f759809dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 10 monclient(hunting): picked mon.noname-b con 0x7f75980a0ec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:01.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f759809e000 0x7f759809e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 10 monclient(hunting): picked mon.noname-c con 0x7f759809e000 addr v2:172.21.15.149:3300/0 2024-09-16T05:10:01.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7598138630 addr v1:172.21.15.114:6789/0 2024-09-16T05:10:01.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f759809a760 con 0x7f7598138630 2024-09-16T05:10:01.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f75980994b0 con 0x7f759809e000 2024-09-16T05:10:01.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f75980a1870 con 0x7f75980a0ec0 2024-09-16T05:10:01.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:01.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759e3a6640 10 monclient(hunting): authenticate will time out at 1086.273541s 2024-09-16T05:10:01.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75980a0ec0 0x7f759809dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75980a0ec0 0x7f759809dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:43668/0 (socket says 172.21.15.114:43668) 2024-09-16T05:10:01.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f759c943640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7598138630 0x7f759813aa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36666/0 (socket says 172.21.15.114:36666) 2024-09-16T05:10:01.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 1 -- 172.21.15.114:0/1517479520 learned_addr learned my addr 172.21.15.114:0/1517479520 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:01.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 10 monclient(hunting): get_auth_request con 0x7f75980a0ec0 auth_method 0 2024-09-16T05:10:01.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f75977fe640 1 --2- 172.21.15.114:0/1517479520 >> v2:172.21.15.149:3300/0 conn(0x7f759809e000 0x7f759809e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f75977fe640 10 monclient(hunting): get_auth_request con 0x7f759809e000 auth_method 0 2024-09-16T05:10:01.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f75977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f75977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f75977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 10 monclient(hunting): handle_auth_done global_id 54125 payload 274 2024-09-16T05:10:01.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 1 -- 172.21.15.114:0/1517479520 >> v2:172.21.15.149:3300/0 conn(0x7f759809e000 msgr2=0x7f759809e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:01.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 1 --2- 172.21.15.114:0/1517479520 >> v2:172.21.15.149:3300/0 conn(0x7f759809e000 0x7f759809e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7596ffd640 1 -- 172.21.15.114:0/1517479520 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1934200771 0 0) 0x7f759809a760 con 0x7f7598138630 2024-09-16T05:10:01.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 1 -- 172.21.15.114:0/1517479520 >> v1:172.21.15.114:6789/0 conn(0x7f7598138630 legacy=0x7f759813aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 10 monclient: _finish_hunting 0 2024-09-16T05:10:01.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 1 monclient: found mon.noname-b 2024-09-16T05:10:01.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:01.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:10:01.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7597fff640 1 -- 172.21.15.114:0/1517479520 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f759809e920 con 0x7f75980a0ec0 2024-09-16T05:10:01.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.643+0000 7f7596ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:10:01.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7597fff640 1 --2- 172.21.15.114:0/1517479520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75980a0ec0 0x7f759809dac0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7588009870 tx=0x7f758801c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=279ecc6ab6824239 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7596ffd640 1 -- 172.21.15.114:0/1517479520 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75880ab430 con 0x7f75980a0ec0 2024-09-16T05:10:01.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7596ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7596ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:10:01.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7596ffd640 10 monclient: dump: 2024-09-16T05:10:01.662 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.662 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.662 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.662 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.662 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.662 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.663 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.663 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.663 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.663 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7596ffd640 10 monclient: _finish_auth 0 2024-09-16T05:10:01.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7596ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:10:01.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7596ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:01.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 5 monclient: authenticate success, global_id 54125 2024-09-16T05:10:01.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7596ffd640 1 -- 172.21.15.114:0/1517479520 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75880b0070 con 0x7f75980a0ec0 2024-09-16T05:10:01.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:01.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7596ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:01.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient: get_monmap_and_config success 2024-09-16T05:10:01.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 4 set_mon_vals no callback set 2024-09-16T05:10:01.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7596ffd640 1 -- 172.21.15.114:0/1517479520 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75880ab7f0 con 0x7f75980a0ec0 2024-09-16T05:10:01.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7596ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7596ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:01.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7596ffd640 10 monclient: dump: 2024-09-16T05:10:01.664 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.664 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.664 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.664 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.665 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.665 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.665 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.665 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.665 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.665 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient: shutdown 2024-09-16T05:10:01.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:01.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 -- 172.21.15.114:0/1517479520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75980a0ec0 msgr2=0x7f759809dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 --2- 172.21.15.114:0/1517479520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75980a0ec0 0x7f759809dac0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7588009870 tx=0x7f758801c540 comp rx=0 tx=0).stop 2024-09-16T05:10:01.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 -- 172.21.15.114:0/1517479520 shutdown_connections 2024-09-16T05:10:01.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 --2- 172.21.15.114:0/1517479520 >> v2:172.21.15.149:3300/0 conn(0x7f759809e000 0x7f759809e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 --2- 172.21.15.114:0/1517479520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75980a0ec0 0x7f759809dac0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 -- 172.21.15.114:0/1517479520 >> 172.21.15.114:0/1517479520 conn(0x7f75980a1fb0 msgr2=0x7f75980a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:01.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 -- 172.21.15.114:0/1517479520 shutdown_connections 2024-09-16T05:10:01.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 -- 172.21.15.114:0/1517479520 wait complete. 2024-09-16T05:10:01.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient: build_initial_monmap 2024-09-16T05:10:01.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient: monmap: 2024-09-16T05:10:01.667 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:01.667 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:10:01.667 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:10:01.667 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:10:01.667 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:01.667 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.667 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:10:01.667 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:10:01.667 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:10:01.667 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 librados: starting msgr at 2024-09-16T05:10:01.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 librados: starting objecter 2024-09-16T05:10:01.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 Processor -- start 2024-09-16T05:10:01.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 -- start start 2024-09-16T05:10:01.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 librados: setting wanted keys 2024-09-16T05:10:01.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 librados: calling monclient init 2024-09-16T05:10:01.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient: init 2024-09-16T05:10:01.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:01.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:10:01.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f759809e000 0x7f75980989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient(hunting): picked mon.noname-b con 0x7f759809e000 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:01.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75980a0ec0 0x7f759808d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient(hunting): picked mon.noname-a con 0x7f75980a0ec0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:01.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7598138630 0x7f759808d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient(hunting): picked mon.noname-c con 0x7f7598138630 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:01.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f759809c3b0 con 0x7f75980a0ec0 2024-09-16T05:10:01.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f759809c230 con 0x7f759809e000 2024-09-16T05:10:01.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f759813e960 con 0x7f7598138630 2024-09-16T05:10:01.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:01.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient(hunting): authenticate will time out at 1086.278346s 2024-09-16T05:10:01.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75980a0ec0 0x7f759808d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759c943640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7598138630 0x7f759808d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75980a0ec0 0x7f759808d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59542/0 (socket says 172.21.15.114:59542) 2024-09-16T05:10:01.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 1 -- 172.21.15.114:0/4033745338 learned_addr learned my addr 172.21.15.114:0/4033745338 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:01.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 10 monclient(hunting): get_auth_request con 0x7f75980a0ec0 auth_method 0 2024-09-16T05:10:01.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759c943640 10 monclient(hunting): get_auth_request con 0x7f7598138630 auth_method 0 2024-09-16T05:10:01.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759c943640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759c943640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759c943640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759c943640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759c943640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759c943640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7597fff640 1 --2- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f759809e000 0x7f75980989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7597fff640 10 monclient(hunting): get_auth_request con 0x7f759809e000 auth_method 0 2024-09-16T05:10:01.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7597fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7597fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7597fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 10 monclient(hunting): handle_auth_done global_id 27552 payload 824 2024-09-16T05:10:01.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 1 -- 172.21.15.114:0/4033745338 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7598138630 msgr2=0x7f759808d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 1 --2- 172.21.15.114:0/4033745338 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7598138630 0x7f759808d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 1 -- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f759809e000 msgr2=0x7f75980989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 1 --2- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f759809e000 0x7f75980989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 10 monclient: _finish_hunting 0 2024-09-16T05:10:01.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 1 monclient: found mon.noname-a 2024-09-16T05:10:01.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:01.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:10:01.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 1 -- 172.21.15.114:0/4033745338 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f759808ec70 con 0x7f75980a0ec0 2024-09-16T05:10:01.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75977fe640 1 --2- 172.21.15.114:0/4033745338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75980a0ec0 0x7f759808d260 secure :-1 s=READY pgs=8171 cs=0 l=1 rev1=1 crypto rx=0x7f758c00d600 tx=0x7f758c00dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7594ff9640 10 client.?.objecter ms_handle_connect 0x7f75980a0ec0 2024-09-16T05:10:01.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7594ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:10:01.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7594ff9640 1 -- 172.21.15.114:0/4033745338 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758c0059a0 con 0x7f75980a0ec0 2024-09-16T05:10:01.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7594ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7594ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:10:01.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7594ff9640 10 monclient: dump: 2024-09-16T05:10:01.672 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.672 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.673 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.673 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.673 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.673 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.673 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.673 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.673 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.673 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7594ff9640 10 monclient: _finish_auth 0 2024-09-16T05:10:01.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7594ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:10:01.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7594ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:01.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 5 monclient: authenticate success, global_id 27552 2024-09-16T05:10:01.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient: _renew_subs 2024-09-16T05:10:01.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:01.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 -- 172.21.15.114:0/4033745338 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7598098ef0 con 0x7f75980a0ec0 2024-09-16T05:10:01.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7594ff9640 1 -- 172.21.15.114:0/4033745338 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f758c005b40 con 0x7f75980a0ec0 2024-09-16T05:10:01.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7594ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:01.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7594ff9640 1 -- 172.21.15.114:0/4033745338 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758c006030 con 0x7f75980a0ec0 2024-09-16T05:10:01.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7594ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7594ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:01.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f7594ff9640 10 monclient: dump: 2024-09-16T05:10:01.674 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.674 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.674 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.674 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.674 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.674 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.674 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.675 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f75967fc640 4 set_mon_vals no callback set 2024-09-16T05:10:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 client.27552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:10:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient: _renew_subs 2024-09-16T05:10:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 -- 172.21.15.114:0/4033745338 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f759809c6a0 con 0x7f75980a0ec0 2024-09-16T05:10:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 librados: init done 2024-09-16T05:10:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.647+0000 7f759e3a6640 1 --2- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7598137e70 0x7f759802d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f759e3a6640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f759e3a6640 1 -- 172.21.15.114:0/4033745338 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f7598069390 con 0x7f7598137e70 2024-09-16T05:10:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 1 -- 172.21.15.114:0/4033745338 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f758c0061d0 con 0x7f75980a0ec0 2024-09-16T05:10:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 1 --2- 172.21.15.114:0/4033745338 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7584037840 0x7f7584039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7597fff640 1 --2- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7598137e70 0x7f759802d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 1 -- 172.21.15.114:0/4033745338 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7398+0+0 (secure 0 0 0) 0x7f758c049830 con 0x7f75980a0ec0 2024-09-16T05:10:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 10 client.27552.objecter ms_dispatch 0x7f7598000ba0 osd_map(80..80 src has 1..80) 2024-09-16T05:10:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 3 client.27552.objecter handle_osd_map got epochs [80,80] > 0 2024-09-16T05:10:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 3 client.27552.objecter handle_osd_map decoding full epoch 80 2024-09-16T05:10:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7597fff640 10 monclient: get_auth_request con 0x7f7598137e70 auth_method 0 2024-09-16T05:10:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7597fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7597fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7597fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 20 client.27552.objecter dump_active .. 0 homeless 2024-09-16T05:10:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7597fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7597fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7597fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f759c943640 1 --2- 172.21.15.114:0/4033745338 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7584037840 0x7f7584039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f759c943640 10 monclient: get_auth_request con 0x7f7584037840 auth_method 0 2024-09-16T05:10:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7597fff640 10 monclient(hunting): handle_auth_done global_id 19552 payload 824 2024-09-16T05:10:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7597fff640 1 -- 172.21.15.114:0/4033745338 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f759809e3e0 con 0x7f7598137e70 2024-09-16T05:10:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 10 client.27552.objecter ms_handle_connect 0x7f7598137e70 2024-09-16T05:10:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 10 client.27552.objecter resend_mon_ops 2024-09-16T05:10:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 10 client.27552.objecter ms_handle_connect 0x7f7584037840 2024-09-16T05:10:01.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7597fff640 1 --2- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7598137e70 0x7f759802d7d0 secure :-1 s=READY pgs=4966 cs=0 l=1 rev1=1 crypto rx=0x7f759809ed50 tx=0x7f758801ad80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f759c943640 1 --2- 172.21.15.114:0/4033745338 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7584037840 0x7f7584039d00 secure :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 crypto rx=0x7f75900018c0 tx=0x7f7590001850 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 1 -- 172.21.15.114:0/4033745338 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75880ab640 con 0x7f7598137e70 2024-09-16T05:10:01.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:10:01.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 10 monclient: dump: 2024-09-16T05:10:01.678 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.678 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.678 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.678 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.678 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.678 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.678 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.678 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.679 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 1 -- 172.21.15.114:0/4033745338 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f75880b0040 con 0x7f7598137e70 2024-09-16T05:10:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:10:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 1 -- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7598137e70 msgr2=0x7f759802d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 1 --2- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7598137e70 0x7f759802d7d0 secure :-1 s=READY pgs=4966 cs=0 l=1 rev1=1 crypto rx=0x7f759809ed50 tx=0x7f758801ad80 comp rx=0 tx=0).stop 2024-09-16T05:10:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 1 -- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7598137e70 msgr2=0x7f759802d7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.651+0000 7f7594ff9640 1 --2- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7598137e70 0x7f759802d7d0 secure :-1 s=CLOSED pgs=4966 cs=0 l=1 rev1=1 crypto rx=0x7f759809ed50 tx=0x7f758801ad80 comp rx=0 tx=0).stop 2024-09-16T05:10:01.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:10:01.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 1 --2- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f759808dce0 0x7f7598090c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 1 -- 172.21.15.114:0/4033745338 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f759808e0f0 con 0x7f759808dce0 2024-09-16T05:10:01.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7597fff640 1 --2- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f759808dce0 0x7f7598090c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7597fff640 10 monclient: get_auth_request con 0x7f759808dce0 auth_method 0 2024-09-16T05:10:01.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7597fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7597fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7597fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7597fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7597fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7597fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7597fff640 10 monclient(hunting): handle_auth_done global_id 19555 payload 824 2024-09-16T05:10:01.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7597fff640 1 -- 172.21.15.114:0/4033745338 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f75980707c0 con 0x7f759808dce0 2024-09-16T05:10:01.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7597fff640 1 --2- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f759808dce0 0x7f7598090c50 secure :-1 s=READY pgs=4967 cs=0 l=1 rev1=1 crypto rx=0x7f758801c580 tx=0x7f75880192e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7594ff9640 10 client.27552.objecter ms_handle_connect 0x7f759808dce0 2024-09-16T05:10:01.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7594ff9640 10 client.27552.objecter resend_mon_ops 2024-09-16T05:10:01.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7594ff9640 1 -- 172.21.15.114:0/4033745338 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75880c5a80 con 0x7f759808dce0 2024-09-16T05:10:01.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7594ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7594ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:10:01.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7594ff9640 10 monclient: dump: 2024-09-16T05:10:01.682 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.683 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7594ff9640 1 -- 172.21.15.114:0/4033745338 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f75880ab640 con 0x7f759808dce0 2024-09-16T05:10:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7594ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:10:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7594ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:10:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7594ff9640 1 -- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f759808dce0 msgr2=0x7f7598090c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7594ff9640 1 --2- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f759808dce0 0x7f7598090c50 secure :-1 s=READY pgs=4967 cs=0 l=1 rev1=1 crypto rx=0x7f758801c580 tx=0x7f75880192e0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7594ff9640 1 -- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f759808dce0 msgr2=0x7f7598090c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:01.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f7594ff9640 1 --2- 172.21.15.114:0/4033745338 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f759808dce0 0x7f7598090c50 secure :-1 s=CLOSED pgs=4967 cs=0 l=1 rev1=1 crypto rx=0x7f758801c580 tx=0x7f75880192e0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.684 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:10:01.684 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:10:01.684 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:10:01.684 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:10:01.684 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 14, 2024-09-16T05:10:01.684 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 159608, 2024-09-16T05:10:01.685 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:10:01.685 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:10:01.685 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:10:01.685 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:10:01.685 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:01.685 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:10:01.685 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:01.685 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:10:01.685 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:10:01.685 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:01.685 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:01.685 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:01.686 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:01.686 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:01.686 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:01.686 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:01.686 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:01.686 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:01.686 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:01.686 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:01.686 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:01.686 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:10:01.687 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:10:01.687 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:01.687 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:01.687 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:01.687 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:01.687 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:01.687 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:01.687 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:01.687 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:01.688 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:01.688 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:01.688 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:01.688 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.688 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.688 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:10:01.688 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:10:01.688 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:10:01.688 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:10:01.688 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:10:01.688 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:10:01.688 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:01.688 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:01.689 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:10:01.689 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:10:01.689 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:10:01.689 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:10:01.689 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:10:01.689 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:10:01.689 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:10:01.689 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:01.689 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:10:01.690 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:01.690 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:01.690 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:01.690 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:01.690 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:01.690 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:01.690 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:01.690 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:01.690 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:01.690 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:01.690 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:01.690 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:01.690 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:10:01.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.691 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:10:01.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.691 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:10:01.691 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:10:01.691 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:01.691 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:01.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.691 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:01.691 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:10:01.692 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.692 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:01.692 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:10:01.692 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.692 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.692 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.692 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:10:01.692 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:10:01.692 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:01.692 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:01.693 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:01.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.693 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:10:01.693 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:10:01.693 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:01.693 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:01.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.693 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:01.693 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:10:01.694 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.694 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.694 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:01.694 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:10:01.694 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.694 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.694 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.694 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:10:01.694 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:10:01.694 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:01.694 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:01.694 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:01.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.695 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:10:01.695 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:10:01.695 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:01.695 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:01.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.695 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:01.695 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:10:01.696 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.696 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.696 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:01.696 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:10:01.696 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:01.696 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.696 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.696 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:10:01.696 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:10:01.696 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:01.696 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:01.697 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:01.697 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.697 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.697 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:10:01.697 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:10:01.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.697 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:01.697 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:01.697 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:10:01.697 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.697 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:01.698 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:10:01.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.698 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:01.698 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:01.698 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:10:01.698 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.698 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:01.698 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:10:01.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.698 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:01.699 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:01.699 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 12 2024-09-16T05:10:01.699 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.699 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:01.699 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:10:01.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:01.699 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:01.699 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:01.699 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:10:01.699 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:01.699 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:01.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:01.700 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:10:01.700 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:10:01.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 10 librados: watch_flush enter 2024-09-16T05:10:01.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 10 librados: watch_flush exit 2024-09-16T05:10:01.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 20 client.27552.objecter shutdown clearing up homeless session... 2024-09-16T05:10:01.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 10 client.27552.objecter successfully canceled tick 2024-09-16T05:10:01.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 1 -- 172.21.15.114:0/4033745338 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7584037840 msgr2=0x7f7584039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 1 --2- 172.21.15.114:0/4033745338 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7584037840 0x7f7584039d00 secure :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 crypto rx=0x7f75900018c0 tx=0x7f7590001850 comp rx=0 tx=0).stop 2024-09-16T05:10:01.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 10 monclient: shutdown 2024-09-16T05:10:01.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:01.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 1 -- 172.21.15.114:0/4033745338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75980a0ec0 msgr2=0x7f759808d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 1 --2- 172.21.15.114:0/4033745338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75980a0ec0 0x7f759808d260 secure :-1 s=READY pgs=8171 cs=0 l=1 rev1=1 crypto rx=0x7f758c00d600 tx=0x7f758c00dac0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759c943640 1 -- 172.21.15.114:0/4033745338 reap_dead start 2024-09-16T05:10:01.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759c943640 1 -- 172.21.15.114:0/4033745338 reap_dead start 2024-09-16T05:10:01.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 1 -- 172.21.15.114:0/4033745338 shutdown_connections 2024-09-16T05:10:01.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 1 -- 172.21.15.114:0/4033745338 >> 172.21.15.114:0/4033745338 conn(0x7f75980a1fb0 msgr2=0x7f759806d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:01.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 1 -- 172.21.15.114:0/4033745338 shutdown_connections 2024-09-16T05:10:01.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 1 -- 172.21.15.114:0/4033745338 wait complete. 2024-09-16T05:10:01.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.659+0000 7f759e3a6640 1 librados: shutdown 2024-09-16T05:10:01.704 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:10:01.704 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:10:01.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:10:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 10 monclient: get_monmap_and_config 2024-09-16T05:10:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 10 monclient: build_initial_monmap 2024-09-16T05:10:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 10 monclient: monmap: 2024-09-16T05:10:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:10:01.817279+0000 2024-09-16T05:10:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:10:01.817279+0000 2024-09-16T05:10:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:10:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:10:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:10:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:10:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:10:01.819 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 1 Processor -- start 2024-09-16T05:10:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 1 -- start start 2024-09-16T05:10:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 10 monclient: init 2024-09-16T05:10:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-16T05:10:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f03f809b6c0 addr v1:172.21.15.149:6789/0 2024-09-16T05:10:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f03f80a15d0 0x7f03f813a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 10 monclient(hunting): picked mon.noname-c con 0x7f03f80a15d0 addr v2:172.21.15.149:3300/0 2024-09-16T05:10:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03f80a19e0 addr v1:172.21.15.114:6789/0 2024-09-16T05:10:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f809d4e0 con 0x7f03f80a19e0 2024-09-16T05:10:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f809f430 con 0x7f03f809b6c0 2024-09-16T05:10:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f03f809e390 con 0x7f03f80a15d0 2024-09-16T05:10:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 10 monclient(hunting): authenticate will time out at 1086.446377s 2024-09-16T05:10:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fcd76640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f03f80a19e0 0x7f03f813cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36672/0 (socket says 172.21.15.114:36672) 2024-09-16T05:10:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fcd76640 1 -- 172.21.15.114:0/14317298 learned_addr learned my addr 172.21.15.114:0/14317298 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:10:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 1 --2- 172.21.15.114:0/14317298 >> v2:172.21.15.149:3300/0 conn(0x7f03f80a15d0 0x7f03f813a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 10 monclient(hunting): get_auth_request con 0x7f03f80a15d0 auth_method 0 2024-09-16T05:10:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 -- 172.21.15.114:0/14317298 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 287290810 0 0) 0x7f03f809d4e0 con 0x7f03f80a19e0 2024-09-16T05:10:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:10:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient(hunting): my global_id is 27555 2024-09-16T05:10:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 -- 172.21.15.114:0/14317298 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03e0003530 con 0x7f03f80a19e0 2024-09-16T05:10:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 -- 172.21.15.114:0/14317298 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1881949878 0 0) 0x7f03e0003530 con 0x7f03f80a19e0 2024-09-16T05:10:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 -- 172.21.15.114:0/14317298 >> v2:172.21.15.149:3300/0 conn(0x7f03f80a15d0 msgr2=0x7f03f813a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 --2- 172.21.15.114:0/14317298 >> v2:172.21.15.149:3300/0 conn(0x7f03f80a15d0 0x7f03f813a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 -- 172.21.15.114:0/14317298 >> v1:172.21.15.149:6789/0 conn(0x7f03f809b6c0 legacy=0x7f03f80a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:10:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:10:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:10:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 -- 172.21.15.114:0/14317298 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f809c2f0 con 0x7f03f80a19e0 2024-09-16T05:10:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:10:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:10:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 -- 172.21.15.114:0/14317298 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f03f0001a00 con 0x7f03f80a19e0 2024-09-16T05:10:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:10:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient: dump: 2024-09-16T05:10:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.833 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:10:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:01.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 -- 172.21.15.114:0/14317298 >> v1:172.21.15.114:6789/0 conn(0x7f03f80a19e0 legacy=0x7f03f813cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:10:01.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 --2- 172.21.15.114:0/14317298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03e00039c0 0x7f03e0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient(hunting): picked mon.c con 0x7f03e00039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:01.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 --2- 172.21.15.114:0/14317298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03e0004810 0x7f03e0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient(hunting): picked mon.a con 0x7f03e0004810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:01.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 --2- 172.21.15.114:0/14317298 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f03e00051d0 0x7f03e000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient(hunting): picked mon.b con 0x7f03e00051d0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:01.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 -- 172.21.15.114:0/14317298 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f03f809e390 con 0x7f03e0004810 2024-09-16T05:10:01.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 -- 172.21.15.114:0/14317298 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f03e000c170 con 0x7f03e00051d0 2024-09-16T05:10:01.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 1 -- 172.21.15.114:0/14317298 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f03e000c370 con 0x7f03e00039c0 2024-09-16T05:10:01.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:01.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:10:01.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 1 --2- 172.21.15.114:0/14317298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03e00039c0 0x7f03e0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 10 monclient(hunting): get_auth_request con 0x7f03e00039c0 auth_method 0 2024-09-16T05:10:01.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:01.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 1 --2- 172.21.15.114:0/14317298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03e0004810 0x7f03e0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 10 monclient(hunting): get_auth_request con 0x7f03e0004810 auth_method 0 2024-09-16T05:10:01.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:01.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:10:01.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:10:01.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fcd76640 1 --2- 172.21.15.114:0/14317298 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f03e00051d0 0x7f03e000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 10 monclient(hunting): handle_auth_done global_id 27555 payload 293 2024-09-16T05:10:01.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 1 -- 172.21.15.114:0/14317298 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f03e00051d0 msgr2=0x7f03e000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:01.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 1 --2- 172.21.15.114:0/14317298 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f03e00051d0 0x7f03e000b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 1 -- 172.21.15.114:0/14317298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03e0004810 msgr2=0x7f03e0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:01.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 1 --2- 172.21.15.114:0/14317298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03e0004810 0x7f03e0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:10:01.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 1 monclient: found mon.c 2024-09-16T05:10:01.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:01.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:01.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 1 -- 172.21.15.114:0/14317298 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e0003530 con 0x7f03e00039c0 2024-09-16T05:10:01.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 10 monclient: _finish_auth 0 2024-09-16T05:10:01.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:10:01.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:01.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 5 monclient: authenticate success, global_id 27555 2024-09-16T05:10:01.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fe7d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:01.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03f77fe640 1 --2- 172.21.15.114:0/14317298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03e0004810 0x7f03e0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:10:01.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.815+0000 7f03fcd76640 1 -- 172.21.15.114:0/14317298 reap_dead start 2024-09-16T05:10:01.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03f7fff640 1 --2- 172.21.15.114:0/14317298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03e00039c0 0x7f03e0003de0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f03ec001750 tx=0x7f03ec01c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14bec6939ec33811 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03f6ffd640 1 -- 172.21.15.114:0/14317298 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec0013a0 con 0x7f03e00039c0 2024-09-16T05:10:01.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:01.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03f6ffd640 10 monclient: dump: 2024-09-16T05:10:01.837 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.838 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.838 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.838 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.838 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.838 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.838 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.838 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.838 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.838 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:01.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03f6ffd640 1 -- 172.21.15.114:0/14317298 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03ec0052d0 con 0x7f03e00039c0 2024-09-16T05:10:01.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:01.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 10 monclient: get_monmap_and_config success 2024-09-16T05:10:01.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 4 set_mon_vals no callback set 2024-09-16T05:10:01.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03f6ffd640 1 -- 172.21.15.114:0/14317298 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec0054c0 con 0x7f03e00039c0 2024-09-16T05:10:01.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:01.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03f6ffd640 10 monclient: dump: 2024-09-16T05:10:01.839 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.839 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.839 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.839 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.839 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.839 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.839 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.839 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.839 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.839 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 10 monclient: shutdown 2024-09-16T05:10:01.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:01.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 -- 172.21.15.114:0/14317298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03e00039c0 msgr2=0x7f03e0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 --2- 172.21.15.114:0/14317298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03e00039c0 0x7f03e0003de0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f03ec001750 tx=0x7f03ec01c3a0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 -- 172.21.15.114:0/14317298 shutdown_connections 2024-09-16T05:10:01.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 --2- 172.21.15.114:0/14317298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03e00039c0 0x7f03e0003de0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 -- 172.21.15.114:0/14317298 >> 172.21.15.114:0/14317298 conn(0x7f03f80a1fb0 msgr2=0x7f03f80a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:01.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 -- 172.21.15.114:0/14317298 shutdown_connections 2024-09-16T05:10:01.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 -- 172.21.15.114:0/14317298 wait complete. 2024-09-16T05:10:01.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 10 monclient: build_initial_monmap 2024-09-16T05:10:01.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 10 monclient: monmap: 2024-09-16T05:10:01.840 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:01.840 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:10:01.840 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:10:01.841 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:10:01.841 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:01.841 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.841 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:10:01.841 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:10:01.841 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:10:01.841 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 librados: starting msgr at 2024-09-16T05:10:01.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 librados: starting objecter 2024-09-16T05:10:01.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 Processor -- start 2024-09-16T05:10:01.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 -- start start 2024-09-16T05:10:01.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 librados: setting wanted keys 2024-09-16T05:10:01.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 librados: calling monclient init 2024-09-16T05:10:01.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 10 monclient: init 2024-09-16T05:10:01.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:01.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:10:01.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f03e00039c0 0x7f03f80984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 10 monclient(hunting): picked mon.noname-b con 0x7f03e00039c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:01.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03f81cf700 0x7f03f8098a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 10 monclient(hunting): picked mon.noname-a con 0x7f03f81cf700 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:01.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03f81337e0 0x7f03f8133c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 10 monclient(hunting): picked mon.noname-c con 0x7f03f81337e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:01.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f03f80a1450 con 0x7f03f81cf700 2024-09-16T05:10:01.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f03f80a12d0 con 0x7f03e00039c0 2024-09-16T05:10:01.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:01.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f03f8137e40 con 0x7f03f81337e0 2024-09-16T05:10:01.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:01.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.819+0000 7f03fe7d9640 10 monclient(hunting): authenticate will time out at 1086.451958s 2024-09-16T05:10:01.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03f81cf700 0x7f03f8098a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fcd76640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03f81337e0 0x7f03f8133c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03f81cf700 0x7f03f8098a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59546/0 (socket says 172.21.15.114:59546) 2024-09-16T05:10:01.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 1 -- 172.21.15.114:0/4273015289 learned_addr learned my addr 172.21.15.114:0/4273015289 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:01.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 10 monclient(hunting): get_auth_request con 0x7f03f81cf700 auth_method 0 2024-09-16T05:10:01.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fcd76640 10 monclient(hunting): get_auth_request con 0x7f03f81337e0 auth_method 0 2024-09-16T05:10:01.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fcd76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fcd76640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fcd76640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fcd76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:01.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fcd76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:01.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fcd76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:01.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f7fff640 1 --2- 172.21.15.114:0/4273015289 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f03e00039c0 0x7f03f80984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f7fff640 10 monclient(hunting): get_auth_request con 0x7f03e00039c0 auth_method 0 2024-09-16T05:10:01.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:01.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:01.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:01.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 10 monclient(hunting): handle_auth_done global_id 27558 payload 824 2024-09-16T05:10:01.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 1 -- 172.21.15.114:0/4273015289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03f81337e0 msgr2=0x7f03f8133c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 1 --2- 172.21.15.114:0/4273015289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03f81337e0 0x7f03f8133c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 1 -- 172.21.15.114:0/4273015289 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f03e00039c0 msgr2=0x7f03f80984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:01.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 1 --2- 172.21.15.114:0/4273015289 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f03e00039c0 0x7f03f80984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:01.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 10 monclient: _finish_hunting 0 2024-09-16T05:10:01.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 1 monclient: found mon.noname-a 2024-09-16T05:10:01.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:01.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:10:01.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 1 -- 172.21.15.114:0/4273015289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f809aff0 con 0x7f03f81cf700 2024-09-16T05:10:01.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f77fe640 1 --2- 172.21.15.114:0/4273015289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03f81cf700 0x7f03f8098a10 secure :-1 s=READY pgs=8173 cs=0 l=1 rev1=1 crypto rx=0x7f03e8011c00 tx=0x7f03e800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 10 client.?.objecter ms_handle_connect 0x7f03f81cf700 2024-09-16T05:10:01.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:10:01.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 1 -- 172.21.15.114:0/4273015289 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e8009690 con 0x7f03f81cf700 2024-09-16T05:10:01.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:10:01.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 10 monclient: dump: 2024-09-16T05:10:01.846 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.846 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.846 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.846 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.846 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.846 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.846 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.846 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.846 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.846 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 10 monclient: _finish_auth 0 2024-09-16T05:10:01.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:10:01.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:01.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 1 -- 172.21.15.114:0/4273015289 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03e8011cf0 con 0x7f03f81cf700 2024-09-16T05:10:01.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fe7d9640 5 monclient: authenticate success, global_id 27558 2024-09-16T05:10:01.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fe7d9640 10 monclient: _renew_subs 2024-09-16T05:10:01.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fe7d9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:01.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fe7d9640 1 -- 172.21.15.114:0/4273015289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f809b2d0 con 0x7f03f81cf700 2024-09-16T05:10:01.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:01.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 1 -- 172.21.15.114:0/4273015289 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e8011ea0 con 0x7f03f81cf700 2024-09-16T05:10:01.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:01.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:01.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 10 monclient: dump: 2024-09-16T05:10:01.847 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:01.847 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:01.847 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.847 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:01.847 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:01.847 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:01.848 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fe7d9640 10 client.27558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:10:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fe7d9640 10 monclient: _renew_subs 2024-09-16T05:10:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fe7d9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fe7d9640 1 -- 172.21.15.114:0/4273015289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f809ad10 con 0x7f03f81cf700 2024-09-16T05:10:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fe7d9640 1 librados: init done 2024-09-16T05:10:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f67fc640 4 set_mon_vals no callback set 2024-09-16T05:10:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fe7d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fe7d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fe7d9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03fe7d9640 1 -- 172.21.15.114:0/4273015289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03f81380e0 con 0x7f03f81cf700 2024-09-16T05:10:01.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 1 -- 172.21.15.114:0/4273015289 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f03e8012040 con 0x7f03f81cf700 2024-09-16T05:10:01.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 1 --2- 172.21.15.114:0/4273015289 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03e4037810 0x7f03e4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:01.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 1 -- 172.21.15.114:0/4273015289 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7398+0+0 (secure 0 0 0) 0x7f03e80497e0 con 0x7f03f81cf700 2024-09-16T05:10:01.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 10 client.27558.objecter ms_dispatch 0x7f03f8000ba0 osd_map(80..80 src has 1..80) 2024-09-16T05:10:01.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 3 client.27558.objecter handle_osd_map got epochs [80,80] > 0 2024-09-16T05:10:01.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 3 client.27558.objecter handle_osd_map decoding full epoch 80 2024-09-16T05:10:01.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 20 client.27558.objecter dump_active .. 0 homeless 2024-09-16T05:10:01.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f7fff640 1 --2- 172.21.15.114:0/4273015289 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03e4037810 0x7f03e4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:01.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f7fff640 10 monclient: get_auth_request con 0x7f03e4037810 auth_method 0 2024-09-16T05:10:01.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f4ff9640 10 client.27558.objecter ms_handle_connect 0x7f03e4037810 2024-09-16T05:10:01.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.823+0000 7f03f7fff640 1 --2- 172.21.15.114:0/4273015289 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03e4037810 0x7f03e4039cd0 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7f03f809f600 tx=0x7f03ec0011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:01.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.827+0000 7f03f4ff9640 1 -- 172.21.15.114:0/4273015289 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f03e8019030 con 0x7f03f81cf700 2024-09-16T05:10:01.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.827+0000 7f03f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:01.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:01.827+0000 7f03f4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:10:02.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.047+0000 7f03fe7d9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:10:02.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.047+0000 7f03fe7d9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:10:02.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.047+0000 7f03fe7d9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:02.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.047+0000 7f03fe7d9640 1 -- 172.21.15.114:0/4273015289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f03f8099ba0 con 0x7f03f81cf700 2024-09-16T05:10:02.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03f4ff9640 1 -- 172.21.15.114:0/4273015289 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f03e8004240 con 0x7f03f81cf700 2024-09-16T05:10:02.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:10:02.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03f4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:10:02.056 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.056 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=11,logm=64} crc {auth=3282033675,config=2586702516,health=3114740572,kv=668245918,logm=3428743799}) 2024-09-16T05:10:02.057 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=12,kv=12,logm=63} crc {auth=2397288004,config=2586702516,health=1810565082,kv=2757161437,logm=252797288}) 2024-09-16T05:10:02.057 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.057 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=11,logm=64} crc {auth=3282033675,config=2586702516,health=3114740572,kv=668245918,logm=3428743799}) 2024-09-16T05:10:02.057 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=12,logm=63} crc {auth=3282033675,config=2586702516,health=3197170498,kv=2757161437,logm=557544615}) 2024-09-16T05:10:02.057 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.057 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4181991373}) 2024-09-16T05:10:02.057 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1380602019}) 2024-09-16T05:10:02.058 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.058 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4181991373}) 2024-09-16T05:10:02.058 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2710230292}) 2024-09-16T05:10:02.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03fe7d9640 10 librados: watch_flush enter 2024-09-16T05:10:02.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03fe7d9640 10 librados: watch_flush exit 2024-09-16T05:10:02.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03fe7d9640 20 client.27558.objecter shutdown clearing up homeless session... 2024-09-16T05:10:02.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03fe7d9640 10 client.27558.objecter successfully canceled tick 2024-09-16T05:10:02.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03fe7d9640 1 -- 172.21.15.114:0/4273015289 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03e4037810 msgr2=0x7f03e4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:02.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03fe7d9640 1 --2- 172.21.15.114:0/4273015289 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03e4037810 0x7f03e4039cd0 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7f03f809f600 tx=0x7f03ec0011a0 comp rx=0 tx=0).stop 2024-09-16T05:10:02.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03fe7d9640 10 monclient: shutdown 2024-09-16T05:10:02.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03fe7d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:02.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03fe7d9640 1 -- 172.21.15.114:0/4273015289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03f81cf700 msgr2=0x7f03f8098a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:02.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03fe7d9640 1 --2- 172.21.15.114:0/4273015289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03f81cf700 0x7f03f8098a10 secure :-1 s=READY pgs=8173 cs=0 l=1 rev1=1 crypto rx=0x7f03e8011c00 tx=0x7f03e800f610 comp rx=0 tx=0).stop 2024-09-16T05:10:02.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03fe7d9640 1 -- 172.21.15.114:0/4273015289 shutdown_connections 2024-09-16T05:10:02.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03fe7d9640 1 --2- 172.21.15.114:0/4273015289 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03e4037810 0x7f03e4039cd0 unknown :-1 s=CLOSED pgs=3386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:02.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.051+0000 7f03fe7d9640 1 --2- 172.21.15.114:0/4273015289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03f81cf700 0x7f03f8098a10 unknown :-1 s=CLOSED pgs=8173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:02.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f03fe7d9640 1 --2- 172.21.15.114:0/4273015289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f03f81337e0 0x7f03f8133c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:02.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f03fe7d9640 1 --2- 172.21.15.114:0/4273015289 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f03e00039c0 0x7f03f80984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:02.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f03fe7d9640 1 -- 172.21.15.114:0/4273015289 >> 172.21.15.114:0/4273015289 conn(0x7f03f80a1fb0 msgr2=0x7f03f80a4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:02.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f03fe7d9640 1 -- 172.21.15.114:0/4273015289 shutdown_connections 2024-09-16T05:10:02.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f03fe7d9640 1 -- 172.21.15.114:0/4273015289 wait complete. 2024-09-16T05:10:02.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f03fe7d9640 1 librados: shutdown 2024-09-16T05:10:02.064 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.064 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=532086002}) 2024-09-16T05:10:02.064 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=818340552}) 2024-09-16T05:10:02.064 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.064 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=532086002}) 2024-09-16T05:10:02.064 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1450067778}) 2024-09-16T05:10:02.064 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3674713653}) 2024-09-16T05:10:02.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=12304473}) 2024-09-16T05:10:02.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3674713653}) 2024-09-16T05:10:02.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1506643437}) 2024-09-16T05:10:02.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4170453709}) 2024-09-16T05:10:02.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1749311799}) 2024-09-16T05:10:02.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4170453709}) 2024-09-16T05:10:02.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.055+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1033685051}) 2024-09-16T05:10:02.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.059+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.059+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839784043}) 2024-09-16T05:10:02.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.059+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1384273259}) 2024-09-16T05:10:02.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.059+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.059+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839784043}) 2024-09-16T05:10:02.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.059+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3120378082}) 2024-09-16T05:10:02.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.059+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.059+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419010031}) 2024-09-16T05:10:02.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.059+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3709519181}) 2024-09-16T05:10:02.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.059+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.059+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419010031}) 2024-09-16T05:10:02.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.059+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1913112438}) 2024-09-16T05:10:02.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.071+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.071+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286685948}) 2024-09-16T05:10:02.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.071+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=873445346}) 2024-09-16T05:10:02.074 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.071+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.074 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.071+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286685948}) 2024-09-16T05:10:02.074 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.071+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3390495834}) 2024-09-16T05:10:02.075 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:10:02.080 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.075+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.080 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.075+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=298635265}) 2024-09-16T05:10:02.080 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.075+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1943553422}) 2024-09-16T05:10:02.081 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.075+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.081 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.075+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=298635265}) 2024-09-16T05:10:02.081 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.075+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=248729833}) 2024-09-16T05:10:02.096 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.091+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.096 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.091+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=805334727}) 2024-09-16T05:10:02.096 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.091+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1861458230}) 2024-09-16T05:10:02.096 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.091+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.097 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.091+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=805334727}) 2024-09-16T05:10:02.097 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.091+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942903279}) 2024-09-16T05:10:02.115 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.111+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.116 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.111+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1030317731}) 2024-09-16T05:10:02.116 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.111+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=279123142}) 2024-09-16T05:10:02.116 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.111+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.116 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.111+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1030317731}) 2024-09-16T05:10:02.116 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.111+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=241133825}) 2024-09-16T05:10:02.120 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.115+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.120 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.115+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1270367107}) 2024-09-16T05:10:02.120 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.115+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2327692261}) 2024-09-16T05:10:02.120 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.115+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.120 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.115+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1270367107}) 2024-09-16T05:10:02.120 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.115+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=164773827}) 2024-09-16T05:10:02.121 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.119+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.121 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.119+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3967645932}) 2024-09-16T05:10:02.121 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.119+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3416672241}) 2024-09-16T05:10:02.121 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.119+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.121 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.119+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3967645932}) 2024-09-16T05:10:02.121 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.119+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=213269187}) 2024-09-16T05:10:02.123 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.119+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.123 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.119+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=26,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=16,osdmap=8} crc {logm=3861972181,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=1497536199,mon_config_key=840575491,monmap=3045472550,osd_metadata=487805116,osd_pg_creating=2309624080,osd_snap=1453231322,osdmap=3068811669}) 2024-09-16T05:10:02.123 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.119+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15,osdmap=6} crc {logm=1672746626,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=1497536199,mon_config_key=1589051853,monmap=3045472550,osd_metadata=487805116,osd_pg_creating=2309624080,osd_snap=3091999859,osdmap=2360422902}) 2024-09-16T05:10:02.123 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.119+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.124 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.119+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=26,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=16,osdmap=8} crc {logm=3861972181,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=1497536199,mon_config_key=840575491,monmap=3045472550,osd_metadata=487805116,osd_pg_creating=2309624080,osd_snap=1453231322,osdmap=3068811669}) 2024-09-16T05:10:02.124 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.119+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=23,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=14,osdmap=16} crc {logm=3796931835,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=3842977442,mon_config_key=971488658,monmap=3045472550,osd_metadata=3499579817,osd_pg_creating=2309624080,osd_snap=1278104158,osdmap=652151409}) 2024-09-16T05:10:02.129 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.123+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.129 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.123+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3056754908}) 2024-09-16T05:10:02.129 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.123+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1662889693}) 2024-09-16T05:10:02.129 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.123+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:10:02.129 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.123+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3056754908}) 2024-09-16T05:10:02.129 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:10:02.123+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1522067758}) 2024-09-16T05:10:03.076 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:10:03.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:10:03.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 10 monclient: get_monmap_and_config 2024-09-16T05:10:03.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 10 monclient: build_initial_monmap 2024-09-16T05:10:03.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 10 monclient: monmap: 2024-09-16T05:10:03.235 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:03.235 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.235 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:10:03.233597+0000 2024-09-16T05:10:03.235 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:10:03.233597+0000 2024-09-16T05:10:03.235 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:03.235 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.235 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:10:03.235 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:10:03.236 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:10:03.236 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:10:03.236 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:10:03.236 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 1 Processor -- start 2024-09-16T05:10:03.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 1 -- start start 2024-09-16T05:10:03.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 10 monclient: init 2024-09-16T05:10:03.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:03.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-16T05:10:03.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fcd20098500 0x7fcd200988e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 10 monclient(hunting): picked mon.noname-c con 0x7fcd20098500 addr v2:172.21.15.149:3300/0 2024-09-16T05:10:03.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcd2009c9b0 addr v1:172.21.15.149:6789/0 2024-09-16T05:10:03.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcd2009d4d0 0x7fcd2013b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 10 monclient(hunting): picked mon.noname-a con 0x7fcd2009d4d0 addr v2:172.21.15.114:3300/0 2024-09-16T05:10:03.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd2009db80 con 0x7fcd2009c9b0 2024-09-16T05:10:03.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fcd2009e070 con 0x7fcd2009d4d0 2024-09-16T05:10:03.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fcd2009f550 con 0x7fcd20098500 2024-09-16T05:10:03.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:03.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd27b12640 10 monclient(hunting): authenticate will time out at 1087.863399s 2024-09-16T05:10:03.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd260af640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcd2009d4d0 0x7fcd2013b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd260af640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcd2009d4d0 0x7fcd2013b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59550/0 (socket says 172.21.15.114:59550) 2024-09-16T05:10:03.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd260af640 1 -- 172.21.15.114:0/2630209141 learned_addr learned my addr 172.21.15.114:0/2630209141 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:03.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd260af640 10 monclient(hunting): get_auth_request con 0x7fcd2009d4d0 auth_method 0 2024-09-16T05:10:03.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd260af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd260af640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd260af640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd258ae640 1 --2- 172.21.15.114:0/2630209141 >> v2:172.21.15.149:3300/0 conn(0x7fcd20098500 0x7fcd200988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd258ae640 10 monclient(hunting): get_auth_request con 0x7fcd20098500 auth_method 0 2024-09-16T05:10:03.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd258ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd258ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd258ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd260af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd260af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd260af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd258ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd258ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.231+0000 7fcd258ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd260af640 10 monclient(hunting): handle_auth_done global_id 27582 payload 274 2024-09-16T05:10:03.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd260af640 1 -- 172.21.15.114:0/2630209141 >> v2:172.21.15.149:3300/0 conn(0x7fcd20098500 msgr2=0x7fcd200988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:03.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd260af640 1 --2- 172.21.15.114:0/2630209141 >> v2:172.21.15.149:3300/0 conn(0x7fcd20098500 0x7fcd200988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd260af640 1 -- 172.21.15.114:0/2630209141 >> v1:172.21.15.149:6789/0 conn(0x7fcd2009c9b0 legacy=0x7fcd2009cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd260af640 10 monclient: _finish_hunting 0 2024-09-16T05:10:03.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd260af640 1 monclient: found mon.noname-a 2024-09-16T05:10:03.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd260af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:03.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd260af640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:10:03.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd260af640 1 -- 172.21.15.114:0/2630209141 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd2009e2d0 con 0x7fcd2009d4d0 2024-09-16T05:10:03.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd260af640 1 --2- 172.21.15.114:0/2630209141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2009d4d0 0x7fcd2013b8d0 secure :-1 s=READY pgs=8179 cs=0 l=1 rev1=1 crypto rx=0x7fcd200a2fd0 tx=0x7fcd1801c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d463682443096d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd248ac640 1 -- 172.21.15.114:0/2630209141 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd180a55d0 con 0x7fcd2009d4d0 2024-09-16T05:10:03.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd248ac640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd248ac640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:10:03.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd248ac640 10 monclient: dump: 2024-09-16T05:10:03.249 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.249 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.249 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.249 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.249 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.250 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.250 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.250 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.250 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.250 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd248ac640 10 monclient: _finish_auth 0 2024-09-16T05:10:03.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd248ac640 10 monclient: _check_auth_tickets 2024-09-16T05:10:03.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd248ac640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:03.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd248ac640 1 -- 172.21.15.114:0/2630209141 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd180a5770 con 0x7fcd2009d4d0 2024-09-16T05:10:03.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 5 monclient: authenticate success, global_id 27582 2024-09-16T05:10:03.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:03.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd248ac640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:03.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 10 monclient: get_monmap_and_config success 2024-09-16T05:10:03.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 4 set_mon_vals no callback set 2024-09-16T05:10:03.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd248ac640 1 -- 172.21.15.114:0/2630209141 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd180a5b40 con 0x7fcd2009d4d0 2024-09-16T05:10:03.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd248ac640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd248ac640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:03.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd248ac640 10 monclient: dump: 2024-09-16T05:10:03.251 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.251 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.251 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.251 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.252 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.252 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.252 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.252 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.252 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.252 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 10 monclient: shutdown 2024-09-16T05:10:03.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:03.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 1 -- 172.21.15.114:0/2630209141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2009d4d0 msgr2=0x7fcd2013b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 1 --2- 172.21.15.114:0/2630209141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2009d4d0 0x7fcd2013b8d0 secure :-1 s=READY pgs=8179 cs=0 l=1 rev1=1 crypto rx=0x7fcd200a2fd0 tx=0x7fcd1801c880 comp rx=0 tx=0).stop 2024-09-16T05:10:03.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 1 -- 172.21.15.114:0/2630209141 shutdown_connections 2024-09-16T05:10:03.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 1 --2- 172.21.15.114:0/2630209141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2009d4d0 0x7fcd2013b8d0 unknown :-1 s=CLOSED pgs=8179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 1 --2- 172.21.15.114:0/2630209141 >> v2:172.21.15.149:3300/0 conn(0x7fcd20098500 0x7fcd200988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 1 -- 172.21.15.114:0/2630209141 >> 172.21.15.114:0/2630209141 conn(0x7fcd200a1fb0 msgr2=0x7fcd200a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:03.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 1 -- 172.21.15.114:0/2630209141 shutdown_connections 2024-09-16T05:10:03.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 1 -- 172.21.15.114:0/2630209141 wait complete. 2024-09-16T05:10:03.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 10 monclient: build_initial_monmap 2024-09-16T05:10:03.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 10 monclient: monmap: 2024-09-16T05:10:03.253 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:03.253 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:10:03.253 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:10:03.253 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:10:03.254 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:03.254 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.254 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:10:03.254 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:10:03.254 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:10:03.254 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 1 librados: starting msgr at 2024-09-16T05:10:03.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 1 librados: starting objecter 2024-09-16T05:10:03.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.235+0000 7fcd27b12640 1 Processor -- start 2024-09-16T05:10:03.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 1 -- start start 2024-09-16T05:10:03.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 1 librados: setting wanted keys 2024-09-16T05:10:03.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 1 librados: calling monclient init 2024-09-16T05:10:03.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient: init 2024-09-16T05:10:03.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:03.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:10:03.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcd20098500 0x7fcd201cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient(hunting): picked mon.noname-b con 0x7fcd20098500 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:03.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd2009c9b0 0x7fcd20072e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient(hunting): picked mon.noname-c con 0x7fcd2009c9b0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:03.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2009d4d0 0x7fcd20139da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient(hunting): picked mon.noname-a con 0x7fcd2009d4d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:03.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcd20098fa0 con 0x7fcd2009d4d0 2024-09-16T05:10:03.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fcd20098e20 con 0x7fcd20098500 2024-09-16T05:10:03.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fcd20099e00 con 0x7fcd2009c9b0 2024-09-16T05:10:03.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:03.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient(hunting): authenticate will time out at 1087.868582s 2024-09-16T05:10:03.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2009d4d0 0x7fcd20139da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd250ad640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd2009c9b0 0x7fcd20072e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2009d4d0 0x7fcd20139da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59566/0 (socket says 172.21.15.114:59566) 2024-09-16T05:10:03.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 1 -- 172.21.15.114:0/4111176929 learned_addr learned my addr 172.21.15.114:0/4111176929 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:03.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 10 monclient(hunting): get_auth_request con 0x7fcd2009d4d0 auth_method 0 2024-09-16T05:10:03.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd250ad640 10 monclient(hunting): get_auth_request con 0x7fcd2009c9b0 auth_method 0 2024-09-16T05:10:03.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd250ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd250ad640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd250ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd250ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd250ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd250ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd258ae640 1 --2- 172.21.15.114:0/4111176929 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcd20098500 0x7fcd201cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd258ae640 10 monclient(hunting): get_auth_request con 0x7fcd20098500 auth_method 0 2024-09-16T05:10:03.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd258ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd258ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd258ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 10 monclient(hunting): handle_auth_done global_id 27585 payload 824 2024-09-16T05:10:03.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 1 -- 172.21.15.114:0/4111176929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd2009c9b0 msgr2=0x7fcd20072e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 1 --2- 172.21.15.114:0/4111176929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd2009c9b0 0x7fcd20072e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 1 -- 172.21.15.114:0/4111176929 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcd20098500 msgr2=0x7fcd201cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 1 --2- 172.21.15.114:0/4111176929 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcd20098500 0x7fcd201cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 10 monclient: _finish_hunting 0 2024-09-16T05:10:03.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 1 monclient: found mon.noname-a 2024-09-16T05:10:03.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:03.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:10:03.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 1 -- 172.21.15.114:0/4111176929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd2006f150 con 0x7fcd2009d4d0 2024-09-16T05:10:03.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd260af640 1 --2- 172.21.15.114:0/4111176929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2009d4d0 0x7fcd20139da0 secure :-1 s=READY pgs=8180 cs=0 l=1 rev1=1 crypto rx=0x7fcd1801a4b0 tx=0x7fcd18019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 10 client.?.objecter ms_handle_connect 0x7fcd2009d4d0 2024-09-16T05:10:03.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:10:03.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 1 -- 172.21.15.114:0/4111176929 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd180bc020 con 0x7fcd2009d4d0 2024-09-16T05:10:03.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:10:03.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 10 monclient: dump: 2024-09-16T05:10:03.260 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.260 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.260 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.260 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.260 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.260 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.261 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.261 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.261 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.261 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 10 monclient: _finish_auth 0 2024-09-16T05:10:03.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 10 monclient: _check_auth_tickets 2024-09-16T05:10:03.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:03.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 1 -- 172.21.15.114:0/4111176929 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd180b8670 con 0x7fcd2009d4d0 2024-09-16T05:10:03.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:03.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 1 -- 172.21.15.114:0/4111176929 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd18009510 con 0x7fcd2009d4d0 2024-09-16T05:10:03.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:03.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 10 monclient: dump: 2024-09-16T05:10:03.262 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.262 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.262 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.262 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.262 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.262 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.262 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.262 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.262 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.262 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd17fff640 4 set_mon_vals no callback set 2024-09-16T05:10:03.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 5 monclient: authenticate success, global_id 27585 2024-09-16T05:10:03.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient: _renew_subs 2024-09-16T05:10:03.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:03.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 1 -- 172.21.15.114:0/4111176929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd2006f430 con 0x7fcd2009d4d0 2024-09-16T05:10:03.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 client.27585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:10:03.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient: _renew_subs 2024-09-16T05:10:03.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:03.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 1 -- 172.21.15.114:0/4111176929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd2009a5c0 con 0x7fcd2009d4d0 2024-09-16T05:10:03.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 1 librados: init done 2024-09-16T05:10:03.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:03.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:03.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:03.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd27b12640 1 -- 172.21.15.114:0/4111176929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd200988e0 con 0x7fcd2009d4d0 2024-09-16T05:10:03.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 1 -- 172.21.15.114:0/4111176929 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcd180bf020 con 0x7fcd2009d4d0 2024-09-16T05:10:03.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 1 --2- 172.21.15.114:0/4111176929 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcd04037810 0x7fcd04039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 1 -- 172.21.15.114:0/4111176929 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7158+0+0 (secure 0 0 0) 0x7fcd180e43b0 con 0x7fcd2009d4d0 2024-09-16T05:10:03.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 10 client.27585.objecter ms_dispatch 0x7fcd20000ba0 osd_map(82..82 src has 1..82) 2024-09-16T05:10:03.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 3 client.27585.objecter handle_osd_map got epochs [82,82] > 0 2024-09-16T05:10:03.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 3 client.27585.objecter handle_osd_map decoding full epoch 82 2024-09-16T05:10:03.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd167fc640 20 client.27585.objecter dump_active .. 0 homeless 2024-09-16T05:10:03.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd258ae640 1 --2- 172.21.15.114:0/4111176929 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcd04037810 0x7fcd04039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.239+0000 7fcd258ae640 10 monclient: get_auth_request con 0x7fcd04037810 auth_method 0 2024-09-16T05:10:03.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.243+0000 7fcd167fc640 10 client.27585.objecter ms_handle_connect 0x7fcd04037810 2024-09-16T05:10:03.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.243+0000 7fcd258ae640 1 --2- 172.21.15.114:0/4111176929 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcd04037810 0x7fcd04039cd0 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009af0 tx=0x7fcd1c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.247+0000 7fcd167fc640 1 -- 172.21.15.114:0/4111176929 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fcd180c6030 con 0x7fcd2009d4d0 2024-09-16T05:10:03.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.247+0000 7fcd167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:03.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.247+0000 7fcd167fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:10:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.467+0000 7fcd27b12640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:10:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.467+0000 7fcd27b12640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:10:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.467+0000 7fcd27b12640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.467+0000 7fcd27b12640 1 -- 172.21.15.114:0/4111176929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcd2013b830 con 0x7fcd2009d4d0 2024-09-16T05:10:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd167fc640 1 -- 172.21.15.114:0/4111176929 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fcd180165c0 con 0x7fcd2009d4d0 2024-09-16T05:10:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:10:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd167fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:10:03.473 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:10:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 10 librados: watch_flush enter 2024-09-16T05:10:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 10 librados: watch_flush exit 2024-09-16T05:10:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 20 client.27585.objecter shutdown clearing up homeless session... 2024-09-16T05:10:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 10 client.27585.objecter successfully canceled tick 2024-09-16T05:10:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 1 -- 172.21.15.114:0/4111176929 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcd04037810 msgr2=0x7fcd04039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 1 --2- 172.21.15.114:0/4111176929 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcd04037810 0x7fcd04039cd0 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009af0 tx=0x7fcd1c008040 comp rx=0 tx=0).stop 2024-09-16T05:10:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 10 monclient: shutdown 2024-09-16T05:10:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 1 -- 172.21.15.114:0/4111176929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2009d4d0 msgr2=0x7fcd20139da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 1 --2- 172.21.15.114:0/4111176929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2009d4d0 0x7fcd20139da0 secure :-1 s=READY pgs=8180 cs=0 l=1 rev1=1 crypto rx=0x7fcd1801a4b0 tx=0x7fcd18019620 comp rx=0 tx=0).stop 2024-09-16T05:10:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 1 -- 172.21.15.114:0/4111176929 shutdown_connections 2024-09-16T05:10:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 1 --2- 172.21.15.114:0/4111176929 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcd04037810 0x7fcd04039cd0 unknown :-1 s=CLOSED pgs=3389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 1 --2- 172.21.15.114:0/4111176929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd2009d4d0 0x7fcd20139da0 unknown :-1 s=CLOSED pgs=8180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 1 --2- 172.21.15.114:0/4111176929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd2009c9b0 0x7fcd20072e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 1 --2- 172.21.15.114:0/4111176929 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcd20098500 0x7fcd201cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 1 -- 172.21.15.114:0/4111176929 >> 172.21.15.114:0/4111176929 conn(0x7fcd200a1fb0 msgr2=0x7fcd200a4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 1 -- 172.21.15.114:0/4111176929 shutdown_connections 2024-09-16T05:10:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 1 -- 172.21.15.114:0/4111176929 wait complete. 2024-09-16T05:10:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.471+0000 7fcd27b12640 1 librados: shutdown 2024-09-16T05:10:03.492 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:10:03.493 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:10:03.493 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:10:03.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:10:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 10 monclient: get_monmap_and_config 2024-09-16T05:10:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 10 monclient: build_initial_monmap 2024-09-16T05:10:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 10 monclient: monmap: 2024-09-16T05:10:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:10:03.609062+0000 2024-09-16T05:10:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:10:03.609062+0000 2024-09-16T05:10:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:10:03.611 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:10:03.611 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:10:03.611 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:10:03.611 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:10:03.611 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 1 Processor -- start 2024-09-16T05:10:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 1 -- start start 2024-09-16T05:10:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 10 monclient: init 2024-09-16T05:10:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-16T05:10:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53900a7d60 0x7f53900a8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 10 monclient(hunting): picked mon.noname-b con 0x7f53900a7d60 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53900a9190 addr v1:172.21.15.149:6789/0 2024-09-16T05:10:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53900a7600 addr v1:172.21.15.114:6789/0 2024-09-16T05:10:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5390137f50 con 0x7f53900a7600 2024-09-16T05:10:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53900a8a70 con 0x7f53900a9190 2024-09-16T05:10:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f53900a88a0 con 0x7f53900a7d60 2024-09-16T05:10:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f5395f9b640 10 monclient(hunting): authenticate will time out at 1088.238304s 2024-09-16T05:10:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53900a7d60 0x7f53900a8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538ffff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f53900a7600 0x7f5390137840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36680/0 (socket says 172.21.15.114:36680) 2024-09-16T05:10:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538ffff640 1 -- 172.21.15.114:0/2405494828 learned_addr learned my addr 172.21.15.114:0/2405494828 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:10:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 10 monclient(hunting): get_auth_request con 0x7f53900a7d60 auth_method 0 2024-09-16T05:10:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538e7fc640 1 -- 172.21.15.114:0/2405494828 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1022823778 0 0) 0x7f5390137f50 con 0x7f53900a7600 2024-09-16T05:10:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:10:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538e7fc640 10 monclient(hunting): my global_id is 27588 2024-09-16T05:10:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538e7fc640 1 -- 172.21.15.114:0/2405494828 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f537c003530 con 0x7f53900a7600 2024-09-16T05:10:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 10 monclient(hunting): handle_auth_done global_id 54152 payload 274 2024-09-16T05:10:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 1 -- 172.21.15.114:0/2405494828 >> v1:172.21.15.149:6789/0 conn(0x7f53900a9190 legacy=0x7f5390137130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 1 -- 172.21.15.114:0/2405494828 >> v1:172.21.15.114:6789/0 conn(0x7f53900a7600 legacy=0x7f5390137840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:10:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 1 monclient: found mon.noname-b 2024-09-16T05:10:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:10:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.607+0000 7f538f7fe640 1 -- 172.21.15.114:0/2405494828 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53900a3710 con 0x7f53900a7d60 2024-09-16T05:10:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538f7fe640 1 --2- 172.21.15.114:0/2405494828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53900a7d60 0x7f53900a8140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f53880097f0 tx=0x7f538801c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16c20d8861051bf4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538e7fc640 1 -- 172.21.15.114:0/2405494828 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53880b0070 con 0x7f53900a7d60 2024-09-16T05:10:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:10:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538e7fc640 10 monclient: dump: 2024-09-16T05:10:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.628 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:10:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:10:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 5 monclient: authenticate success, global_id 54152 2024-09-16T05:10:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538e7fc640 1 -- 172.21.15.114:0/2405494828 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53880ab600 con 0x7f53900a7d60 2024-09-16T05:10:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 10 monclient: get_monmap_and_config success 2024-09-16T05:10:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538e7fc640 1 -- 172.21.15.114:0/2405494828 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53880ab9b0 con 0x7f53900a7d60 2024-09-16T05:10:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 4 set_mon_vals no callback set 2024-09-16T05:10:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538e7fc640 10 monclient: dump: 2024-09-16T05:10:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.630 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 10 monclient: shutdown 2024-09-16T05:10:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 -- 172.21.15.114:0/2405494828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53900a7d60 msgr2=0x7f53900a8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 --2- 172.21.15.114:0/2405494828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53900a7d60 0x7f53900a8140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f53880097f0 tx=0x7f538801c430 comp rx=0 tx=0).stop 2024-09-16T05:10:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 -- 172.21.15.114:0/2405494828 shutdown_connections 2024-09-16T05:10:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 --2- 172.21.15.114:0/2405494828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53900a7d60 0x7f53900a8140 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 -- 172.21.15.114:0/2405494828 >> 172.21.15.114:0/2405494828 conn(0x7f53900a9e00 msgr2=0x7f53900a6300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 -- 172.21.15.114:0/2405494828 shutdown_connections 2024-09-16T05:10:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 -- 172.21.15.114:0/2405494828 wait complete. 2024-09-16T05:10:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 10 monclient: build_initial_monmap 2024-09-16T05:10:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 10 monclient: monmap: 2024-09-16T05:10:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:10:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:10:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:10:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:10:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:10:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:10:03.631 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 librados: starting msgr at 2024-09-16T05:10:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 librados: starting objecter 2024-09-16T05:10:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 Processor -- start 2024-09-16T05:10:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 -- start start 2024-09-16T05:10:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 librados: setting wanted keys 2024-09-16T05:10:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 librados: calling monclient init 2024-09-16T05:10:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 10 monclient: init 2024-09-16T05:10:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:10:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f53900a7600 0x7f5390136880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 10 monclient(hunting): picked mon.noname-b con 0x7f53900a7600 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53900a7d60 0x7f5390136dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 10 monclient(hunting): picked mon.noname-a con 0x7f53900a7d60 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53900a9190 0x7f539013cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 10 monclient(hunting): picked mon.noname-c con 0x7f53900a9190 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5390142e20 con 0x7f53900a7d60 2024-09-16T05:10:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f5390142ca0 con 0x7f53900a7600 2024-09-16T05:10:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f5390142fa0 con 0x7f53900a9190 2024-09-16T05:10:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f5395f9b640 10 monclient(hunting): authenticate will time out at 1088.243259s 2024-09-16T05:10:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53900a9190 0x7f539013cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53900a7d60 0x7f5390136dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53900a9190 0x7f539013cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:43720/0 (socket says 172.21.15.114:43720) 2024-09-16T05:10:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53900a7d60 0x7f5390136dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59574/0 (socket says 172.21.15.114:59574) 2024-09-16T05:10:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 1 -- 172.21.15.114:0/4280735531 learned_addr learned my addr 172.21.15.114:0/4280735531 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 10 monclient(hunting): get_auth_request con 0x7f53900a9190 auth_method 0 2024-09-16T05:10:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538effd640 10 monclient(hunting): get_auth_request con 0x7f53900a7d60 auth_method 0 2024-09-16T05:10:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538f7fe640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f53900a7600 0x7f5390136880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538f7fe640 10 monclient(hunting): get_auth_request con 0x7f53900a7600 auth_method 0 2024-09-16T05:10:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 10 monclient(hunting): handle_auth_done global_id 54155 payload 824 2024-09-16T05:10:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 1 -- 172.21.15.114:0/4280735531 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f53900a7600 msgr2=0x7f5390136880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f53900a7600 0x7f5390136880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 1 -- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53900a7d60 msgr2=0x7f5390136dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53900a7d60 0x7f5390136dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:10:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 1 monclient: found mon.noname-c 2024-09-16T05:10:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:10:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 1 -- 172.21.15.114:0/4280735531 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53901202a0 con 0x7f53900a9190 2024-09-16T05:10:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538f7fe640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f53900a7600 0x7f5390136880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:10:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538effd640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53900a7d60 0x7f5390136dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:10:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.611+0000 7f538ffff640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53900a9190 0x7f539013cc40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f538000c480 tx=0x7f538000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 client.?.objecter ms_handle_connect 0x7f53900a9190 2024-09-16T05:10:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:10:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 1 -- 172.21.15.114:0/4280735531 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5380017070 con 0x7f53900a9190 2024-09-16T05:10:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:10:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 monclient: dump: 2024-09-16T05:10:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.637 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 monclient: _finish_auth 0 2024-09-16T05:10:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 monclient: _check_auth_tickets 2024-09-16T05:10:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 1 -- 172.21.15.114:0/4280735531 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5380005cf0 con 0x7f53900a9190 2024-09-16T05:10:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5395f9b640 5 monclient: authenticate success, global_id 54155 2024-09-16T05:10:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5395f9b640 10 monclient: _renew_subs 2024-09-16T05:10:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5395f9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5395f9b640 1 -- 172.21.15.114:0/4280735531 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f539013f440 con 0x7f53900a9190 2024-09-16T05:10:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 1 -- 172.21.15.114:0/4280735531 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53800072b0 con 0x7f53900a9190 2024-09-16T05:10:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 monclient: dump: 2024-09-16T05:10:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.639 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f538dffb640 4 set_mon_vals no callback set 2024-09-16T05:10:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5395f9b640 10 client.54155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:10:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5395f9b640 10 monclient: _renew_subs 2024-09-16T05:10:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5395f9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5395f9b640 1 -- 172.21.15.114:0/4280735531 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f539009d390 con 0x7f53900a9190 2024-09-16T05:10:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5395f9b640 1 librados: init done 2024-09-16T05:10:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5395f9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5395f9b640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539013d940 0x7f539002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5395f9b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5395f9b640 1 -- 172.21.15.114:0/4280735531 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5390068f10 con 0x7f539013d940 2024-09-16T05:10:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f538f7fe640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539013d940 0x7f539002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 1 -- 172.21.15.114:0/4280735531 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f538001a070 con 0x7f53900a9190 2024-09-16T05:10:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f538f7fe640 10 monclient: get_auth_request con 0x7f539013d940 auth_method 0 2024-09-16T05:10:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f538f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f538f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f538f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f538f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f538f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f538f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 1 --2- 172.21.15.114:0/4280735531 >> v2:172.21.15.149:6812/1185531135 conn(0x7f5370037810 0x7f5370039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f538f7fe640 10 monclient(hunting): handle_auth_done global_id 27594 payload 824 2024-09-16T05:10:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 1 -- 172.21.15.114:0/4280735531 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(82..82 src has 1..82) ==== 7158+0+0 (secure 0 0 0) 0x7f53800487a0 con 0x7f53900a9190 2024-09-16T05:10:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 client.54155.objecter ms_dispatch 0x7f5390000ba0 osd_map(82..82 src has 1..82) 2024-09-16T05:10:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f538f7fe640 1 -- 172.21.15.114:0/4280735531 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53900a27b0 con 0x7f539013d940 2024-09-16T05:10:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 3 client.54155.objecter handle_osd_map got epochs [82,82] > 0 2024-09-16T05:10:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 3 client.54155.objecter handle_osd_map decoding full epoch 82 2024-09-16T05:10:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 20 client.54155.objecter dump_active .. 0 homeless 2024-09-16T05:10:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 client.54155.objecter ms_handle_connect 0x7f539013d940 2024-09-16T05:10:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 client.54155.objecter resend_mon_ops 2024-09-16T05:10:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f538f7fe640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539013d940 0x7f539002c650 secure :-1 s=READY pgs=8182 cs=0 l=1 rev1=1 crypto rx=0x7f53900a3450 tx=0x7f53880191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f538effd640 1 --2- 172.21.15.114:0/4280735531 >> v2:172.21.15.149:6812/1185531135 conn(0x7f5370037810 0x7f5370039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f538effd640 10 monclient: get_auth_request con 0x7f5370037810 auth_method 0 2024-09-16T05:10:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 1 -- 172.21.15.114:0/4280735531 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53880bb030 con 0x7f539013d940 2024-09-16T05:10:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 monclient: dump: 2024-09-16T05:10:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.642 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 1 -- 172.21.15.114:0/4280735531 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f53880b1040 con 0x7f539013d940 2024-09-16T05:10:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:10:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 1 -- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539013d940 msgr2=0x7f539002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539013d940 0x7f539002c650 secure :-1 s=READY pgs=8182 cs=0 l=1 rev1=1 crypto rx=0x7f53900a3450 tx=0x7f53880191d0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 1 -- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539013d940 msgr2=0x7f539002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539013d940 0x7f539002c650 secure :-1 s=CLOSED pgs=8182 cs=0 l=1 rev1=1 crypto rx=0x7f53900a3450 tx=0x7f53880191d0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f5387fff640 10 client.54155.objecter ms_handle_connect 0x7f5370037810 2024-09-16T05:10:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.615+0000 7f538effd640 1 --2- 172.21.15.114:0/4280735531 >> v2:172.21.15.149:6812/1185531135 conn(0x7f5370037810 0x7f5370039cd0 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7f53901204b0 tx=0x7f5378009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:10:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539013dd20 0x7f539002cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 1 -- 172.21.15.114:0/4280735531 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f53900a8140 con 0x7f539013dd20 2024-09-16T05:10:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f538f7fe640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539013dd20 0x7f539002cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f538f7fe640 10 monclient: get_auth_request con 0x7f539013dd20 auth_method 0 2024-09-16T05:10:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f538f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f538f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f538f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f538f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f538f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f538f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f538f7fe640 10 monclient(hunting): handle_auth_done global_id 27597 payload 824 2024-09-16T05:10:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f538f7fe640 1 -- 172.21.15.114:0/4280735531 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f539013f1c0 con 0x7f539013dd20 2024-09-16T05:10:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f538f7fe640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539013dd20 0x7f539002cb90 secure :-1 s=READY pgs=8183 cs=0 l=1 rev1=1 crypto rx=0x7f5388009740 tx=0x7f5388009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5387fff640 10 client.54155.objecter ms_handle_connect 0x7f539013dd20 2024-09-16T05:10:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5387fff640 10 client.54155.objecter resend_mon_ops 2024-09-16T05:10:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5387fff640 1 -- 172.21.15.114:0/4280735531 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53880bb040 con 0x7f539013dd20 2024-09-16T05:10:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5387fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5387fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5387fff640 10 monclient: dump: 2024-09-16T05:10:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.645 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5387fff640 1 -- 172.21.15.114:0/4280735531 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f5388003bc0 con 0x7f539013dd20 2024-09-16T05:10:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5387fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:10:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5387fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:10:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5387fff640 1 -- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539013dd20 msgr2=0x7f539002cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5387fff640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539013dd20 0x7f539002cb90 secure :-1 s=READY pgs=8183 cs=0 l=1 rev1=1 crypto rx=0x7f5388009740 tx=0x7f5388009190 comp rx=0 tx=0).stop 2024-09-16T05:10:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5387fff640 1 -- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539013dd20 msgr2=0x7f539002cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5387fff640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539013dd20 0x7f539002cb90 secure :-1 s=CLOSED pgs=8183 cs=0 l=1 rev1=1 crypto rx=0x7f5388009740 tx=0x7f5388009190 comp rx=0 tx=0).stop 2024-09-16T05:10:03.646 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:10:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:10:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:10:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:10:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 14, 2024-09-16T05:10:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 161612, 2024-09-16T05:10:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:10:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:10:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:10:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:10:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:10:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:10:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:10:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:10:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:10:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:10:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:10:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:10:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:10:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:10:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:10:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:10:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:10:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:10:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:10:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:10:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:10:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:10:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:10:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:10:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:10:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:10:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:10:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:10:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:10:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:10:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:10:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:10:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:10:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:10:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:10:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:10:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:10:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:10:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:10:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:10:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:10:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:10:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:10:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:10:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:10:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:10:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:10:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:10:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:10:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:10:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:10:03.659 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:10:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 10 librados: watch_flush enter 2024-09-16T05:10:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 10 librados: watch_flush exit 2024-09-16T05:10:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 20 client.54155.objecter shutdown clearing up homeless session... 2024-09-16T05:10:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 10 client.54155.objecter successfully canceled tick 2024-09-16T05:10:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 1 -- 172.21.15.114:0/4280735531 >> v2:172.21.15.149:6812/1185531135 conn(0x7f5370037810 msgr2=0x7f5370039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 1 --2- 172.21.15.114:0/4280735531 >> v2:172.21.15.149:6812/1185531135 conn(0x7f5370037810 0x7f5370039cd0 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7f53901204b0 tx=0x7f5378009040 comp rx=0 tx=0).stop 2024-09-16T05:10:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 10 monclient: shutdown 2024-09-16T05:10:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:03.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 1 -- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53900a9190 msgr2=0x7f539013cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 1 --2- 172.21.15.114:0/4280735531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53900a9190 0x7f539013cc40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f538000c480 tx=0x7f538000c940 comp rx=0 tx=0).stop 2024-09-16T05:10:03.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f538ffff640 1 -- 172.21.15.114:0/4280735531 reap_dead start 2024-09-16T05:10:03.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f538ffff640 1 -- 172.21.15.114:0/4280735531 reap_dead start 2024-09-16T05:10:03.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 1 -- 172.21.15.114:0/4280735531 shutdown_connections 2024-09-16T05:10:03.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 1 -- 172.21.15.114:0/4280735531 >> 172.21.15.114:0/4280735531 conn(0x7f53900a9e00 msgr2=0x7f53900a1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:03.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 1 -- 172.21.15.114:0/4280735531 shutdown_connections 2024-09-16T05:10:03.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 1 -- 172.21.15.114:0/4280735531 wait complete. 2024-09-16T05:10:03.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.623+0000 7f5395f9b640 1 librados: shutdown 2024-09-16T05:10:03.663 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:10:03.663 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:10:03.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:10:03.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 10 monclient: get_monmap_and_config 2024-09-16T05:10:03.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 10 monclient: build_initial_monmap 2024-09-16T05:10:03.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 10 monclient: monmap: 2024-09-16T05:10:03.777 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:10:03.776434+0000 2024-09-16T05:10:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:10:03.776434+0000 2024-09-16T05:10:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:10:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:10:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:10:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:10:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:10:03.778 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 1 Processor -- start 2024-09-16T05:10:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 1 -- start start 2024-09-16T05:10:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 10 monclient: init 2024-09-16T05:10:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-16T05:10:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c101376e0 0x7f6c10137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 10 monclient(hunting): picked mon.noname-b con 0x7f6c101376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6c101370a0 0x7f6c100a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 10 monclient(hunting): picked mon.noname-a con 0x7f6c101370a0 addr v2:172.21.15.114:3300/0 2024-09-16T05:10:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f6c100a5d00 0x7f6c100a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 10 monclient(hunting): picked mon.noname-c con 0x7f6c100a5d00 addr v2:172.21.15.149:3300/0 2024-09-16T05:10:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f6c10138000 con 0x7f6c101370a0 2024-09-16T05:10:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f6c1009c5c0 con 0x7f6c100a5d00 2024-09-16T05:10:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6c1009bd20 con 0x7f6c101376e0 2024-09-16T05:10:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1748e640 10 monclient(hunting): authenticate will time out at 1088.405590s 2024-09-16T05:10:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c14a29640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6c101370a0 0x7f6c100a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c101376e0 0x7f6c10137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c14a29640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6c101370a0 0x7f6c100a4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59610/0 (socket says 172.21.15.114:59610) 2024-09-16T05:10:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c14a29640 1 -- 172.21.15.114:0/2045368367 learned_addr learned my addr 172.21.15.114:0/2045368367 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c14a29640 10 monclient(hunting): get_auth_request con 0x7f6c101370a0 auth_method 0 2024-09-16T05:10:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c14a29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c14a29640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c14a29640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 10 monclient(hunting): get_auth_request con 0x7f6c101376e0 auth_method 0 2024-09-16T05:10:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c14a29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c14a29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c14a29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c15a2b640 1 --2- 172.21.15.114:0/2045368367 >> v2:172.21.15.149:3300/0 conn(0x7f6c100a5d00 0x7f6c100a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c15a2b640 10 monclient(hunting): get_auth_request con 0x7f6c100a5d00 auth_method 0 2024-09-16T05:10:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c15a2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c15a2b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c15a2b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 10 monclient(hunting): handle_auth_done global_id 54158 payload 274 2024-09-16T05:10:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 1 -- 172.21.15.114:0/2045368367 >> v2:172.21.15.149:3300/0 conn(0x7f6c100a5d00 msgr2=0x7f6c100a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 1 --2- 172.21.15.114:0/2045368367 >> v2:172.21.15.149:3300/0 conn(0x7f6c100a5d00 0x7f6c100a47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 1 -- 172.21.15.114:0/2045368367 >> v2:172.21.15.114:3300/0 conn(0x7f6c101370a0 msgr2=0x7f6c100a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 1 --2- 172.21.15.114:0/2045368367 >> v2:172.21.15.114:3300/0 conn(0x7f6c101370a0 0x7f6c100a4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 10 monclient: _finish_hunting 0 2024-09-16T05:10:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 1 monclient: found mon.noname-b 2024-09-16T05:10:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:10:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c1522a640 1 -- 172.21.15.114:0/2045368367 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1009c000 con 0x7f6c101376e0 2024-09-16T05:10:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.775+0000 7f6c14a29640 1 --2- 172.21.15.114:0/2045368367 >> v2:172.21.15.114:3300/0 conn(0x7f6c101370a0 0x7f6c100a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:10:03.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 1 --2- 172.21.15.114:0/2045368367 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c101376e0 0x7f6c10137ac0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6c10073760 tx=0x7f6c0801c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1edff6a887f6292d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c0ffff640 1 -- 172.21.15.114:0/2045368367 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c080ac070 con 0x7f6c101376e0 2024-09-16T05:10:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c0ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:10:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c0ffff640 10 monclient: dump: 2024-09-16T05:10:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.796 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c0ffff640 10 monclient: _finish_auth 0 2024-09-16T05:10:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c0ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:10:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c0ffff640 1 -- 172.21.15.114:0/2045368367 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c080a6680 con 0x7f6c101376e0 2024-09-16T05:10:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 5 monclient: authenticate success, global_id 54158 2024-09-16T05:10:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c0ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 10 monclient: get_monmap_and_config success 2024-09-16T05:10:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c0ffff640 1 -- 172.21.15.114:0/2045368367 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c080a6a50 con 0x7f6c101376e0 2024-09-16T05:10:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 4 set_mon_vals no callback set 2024-09-16T05:10:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c0ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c0ffff640 10 monclient: dump: 2024-09-16T05:10:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.798 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 10 monclient: shutdown 2024-09-16T05:10:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 -- 172.21.15.114:0/2045368367 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c101376e0 msgr2=0x7f6c10137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 --2- 172.21.15.114:0/2045368367 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c101376e0 0x7f6c10137ac0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6c10073760 tx=0x7f6c0801c7f0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 -- 172.21.15.114:0/2045368367 shutdown_connections 2024-09-16T05:10:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 --2- 172.21.15.114:0/2045368367 >> v2:172.21.15.149:3300/0 conn(0x7f6c100a5d00 0x7f6c100a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 --2- 172.21.15.114:0/2045368367 >> v2:172.21.15.114:3300/0 conn(0x7f6c101370a0 0x7f6c100a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 --2- 172.21.15.114:0/2045368367 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c101376e0 0x7f6c10137ac0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 -- 172.21.15.114:0/2045368367 >> 172.21.15.114:0/2045368367 conn(0x7f6c100a9a40 msgr2=0x7f6c100a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 -- 172.21.15.114:0/2045368367 shutdown_connections 2024-09-16T05:10:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 -- 172.21.15.114:0/2045368367 wait complete. 2024-09-16T05:10:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 10 monclient: build_initial_monmap 2024-09-16T05:10:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 10 monclient: monmap: 2024-09-16T05:10:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:10:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:10:03.800 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:10:03.800 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:03.800 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.800 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:10:03.800 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:10:03.800 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:10:03.801 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 librados: starting msgr at 2024-09-16T05:10:03.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 librados: starting objecter 2024-09-16T05:10:03.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 Processor -- start 2024-09-16T05:10:03.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 -- start start 2024-09-16T05:10:03.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 librados: setting wanted keys 2024-09-16T05:10:03.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 librados: calling monclient init 2024-09-16T05:10:03.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 10 monclient: init 2024-09-16T05:10:03.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:03.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:10:03.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c100a5d00 0x7f6c101cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 10 monclient(hunting): picked mon.noname-c con 0x7f6c100a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:03.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c101370a0 0x7f6c10136250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 10 monclient(hunting): picked mon.noname-a con 0x7f6c101370a0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:03.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6c101376e0 0x7f6c1013c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 10 monclient(hunting): picked mon.noname-b con 0x7f6c101376e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:03.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6c10140f80 con 0x7f6c101370a0 2024-09-16T05:10:03.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f6c10140e00 con 0x7f6c101376e0 2024-09-16T05:10:03.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6c10141100 con 0x7f6c100a5d00 2024-09-16T05:10:03.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:03.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1748e640 10 monclient(hunting): authenticate will time out at 1088.411376s 2024-09-16T05:10:03.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c100a5d00 0x7f6c101cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c100a5d00 0x7f6c101cf2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:43734/0 (socket says 172.21.15.114:43734) 2024-09-16T05:10:03.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c14a29640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c101370a0 0x7f6c10136250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 1 -- 172.21.15.114:0/2955129607 learned_addr learned my addr 172.21.15.114:0/2955129607 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:03.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 10 monclient(hunting): get_auth_request con 0x7f6c100a5d00 auth_method 0 2024-09-16T05:10:03.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c14a29640 10 monclient(hunting): get_auth_request con 0x7f6c101370a0 auth_method 0 2024-09-16T05:10:03.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c14a29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c14a29640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c14a29640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c14a29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c14a29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c14a29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 10 monclient(hunting): handle_auth_done global_id 54161 payload 824 2024-09-16T05:10:03.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c15a2b640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6c101376e0 0x7f6c1013c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 1 -- 172.21.15.114:0/2955129607 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6c101376e0 msgr2=0x7f6c1013c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6c101376e0 0x7f6c1013c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 1 -- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c101370a0 msgr2=0x7f6c10136250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c101370a0 0x7f6c10136250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 10 monclient: _finish_hunting 0 2024-09-16T05:10:03.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 1 monclient: found mon.noname-c 2024-09-16T05:10:03.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:03.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:10:03.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c1522a640 1 -- 172.21.15.114:0/2955129607 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c100a9100 con 0x7f6c100a5d00 2024-09-16T05:10:03.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c14a29640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c101370a0 0x7f6c10136250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:10:03.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c15a2b640 10 monclient: get_auth_request con 0x7f6c101376e0 auth_method 0 2024-09-16T05:10:03.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.779+0000 7f6c15a2b640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6c101376e0 0x7f6c1013c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:10:03.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c1522a640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c100a5d00 0x7f6c101cf2b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6c0801d090 tx=0x7f6c08019590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 client.?.objecter ms_handle_connect 0x7f6c100a5d00 2024-09-16T05:10:03.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:10:03.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 1 -- 172.21.15.114:0/2955129607 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c080c0070 con 0x7f6c100a5d00 2024-09-16T05:10:03.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:10:03.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 monclient: dump: 2024-09-16T05:10:03.805 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.805 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.806 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 monclient: _finish_auth 0 2024-09-16T05:10:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:10:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 1 -- 172.21.15.114:0/2955129607 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c080a6480 con 0x7f6c100a5d00 2024-09-16T05:10:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 1 -- 172.21.15.114:0/2955129607 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c080ac050 con 0x7f6c100a5d00 2024-09-16T05:10:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c1748e640 5 monclient: authenticate success, global_id 54161 2024-09-16T05:10:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c1748e640 10 monclient: _renew_subs 2024-09-16T05:10:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c1748e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c1748e640 1 -- 172.21.15.114:0/2955129607 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c10073d20 con 0x7f6c100a5d00 2024-09-16T05:10:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0f7fe640 4 set_mon_vals no callback set 2024-09-16T05:10:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 monclient: dump: 2024-09-16T05:10:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.808 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c1748e640 10 client.54161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:10:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c1748e640 10 monclient: _renew_subs 2024-09-16T05:10:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c1748e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c1748e640 1 -- 172.21.15.114:0/2955129607 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c10121350 con 0x7f6c100a5d00 2024-09-16T05:10:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c1748e640 1 librados: init done 2024-09-16T05:10:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c1748e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c1748e640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c10120460 0x7f6c1013c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c1748e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c1748e640 1 -- 172.21.15.114:0/2955129607 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6c10137ac0 con 0x7f6c10120460 2024-09-16T05:10:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c14a29640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c10120460 0x7f6c1013c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c14a29640 10 monclient: get_auth_request con 0x7f6c10120460 auth_method 0 2024-09-16T05:10:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c14a29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c14a29640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c14a29640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 1 -- 172.21.15.114:0/2955129607 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6c080c3070 con 0x7f6c100a5d00 2024-09-16T05:10:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c14a29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c14a29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c14a29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c14a29640 10 monclient(hunting): handle_auth_done global_id 54164 payload 824 2024-09-16T05:10:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c14a29640 1 -- 172.21.15.114:0/2955129607 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6c10121cb0 con 0x7f6c10120460 2024-09-16T05:10:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 1 --2- 172.21.15.114:0/2955129607 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6bfc037810 0x7f6bfc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 1 -- 172.21.15.114:0/2955129607 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(82..82 src has 1..82) ==== 7158+0+0 (secure 0 0 0) 0x7f6c080e3f40 con 0x7f6c100a5d00 2024-09-16T05:10:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 client.54161.objecter ms_dispatch 0x7f6c10000ba0 osd_map(82..82 src has 1..82) 2024-09-16T05:10:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 3 client.54161.objecter handle_osd_map got epochs [82,82] > 0 2024-09-16T05:10:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 3 client.54161.objecter handle_osd_map decoding full epoch 82 2024-09-16T05:10:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c14a29640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c10120460 0x7f6c1013c650 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6c1013d890 tx=0x7f6c0400f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 20 client.54161.objecter dump_active .. 0 homeless 2024-09-16T05:10:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 client.54161.objecter ms_handle_connect 0x7f6c10120460 2024-09-16T05:10:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 client.54161.objecter resend_mon_ops 2024-09-16T05:10:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 1 -- 172.21.15.114:0/2955129607 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04014440 con 0x7f6c10120460 2024-09-16T05:10:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 monclient: dump: 2024-09-16T05:10:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.811 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c15a2b640 1 --2- 172.21.15.114:0/2955129607 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6bfc037810 0x7f6bfc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c15a2b640 10 monclient: get_auth_request con 0x7f6bfc037810 auth_method 0 2024-09-16T05:10:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 1 -- 172.21.15.114:0/2955129607 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6c04014440 con 0x7f6c10120460 2024-09-16T05:10:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:10:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 1 -- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c10120460 msgr2=0x7f6c1013c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c10120460 0x7f6c1013c650 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6c1013d890 tx=0x7f6c0400f610 comp rx=0 tx=0).stop 2024-09-16T05:10:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 1 -- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c10120460 msgr2=0x7f6c1013c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c10120460 0x7f6c1013c650 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6c1013d890 tx=0x7f6c0400f610 comp rx=0 tx=0).stop 2024-09-16T05:10:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c0dffb640 10 client.54161.objecter ms_handle_connect 0x7f6bfc037810 2024-09-16T05:10:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.783+0000 7f6c15a2b640 1 --2- 172.21.15.114:0/2955129607 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6bfc037810 0x7f6bfc039cd0 secure :-1 s=READY pgs=3391 cs=0 l=1 rev1=1 crypto rx=0x7f6c000097c0 tx=0x7f6c00008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:10:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c10136790 0x7f6c1013cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 1 -- 172.21.15.114:0/2955129607 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6c101216c0 con 0x7f6c10136790 2024-09-16T05:10:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c14a29640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c10136790 0x7f6c1013cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c14a29640 10 monclient: get_auth_request con 0x7f6c10136790 auth_method 0 2024-09-16T05:10:03.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c14a29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c14a29640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c14a29640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c14a29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c14a29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c14a29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c14a29640 10 monclient(hunting): handle_auth_done global_id 54167 payload 824 2024-09-16T05:10:03.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c14a29640 1 -- 172.21.15.114:0/2955129607 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6c10068820 con 0x7f6c10136790 2024-09-16T05:10:03.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c14a29640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c10136790 0x7f6c1013cb90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6c1013e3a0 tx=0x7f6c040046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c0dffb640 10 client.54161.objecter ms_handle_connect 0x7f6c10136790 2024-09-16T05:10:03.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c0dffb640 10 client.54161.objecter resend_mon_ops 2024-09-16T05:10:03.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c0dffb640 1 -- 172.21.15.114:0/2955129607 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04004220 con 0x7f6c10136790 2024-09-16T05:10:03.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c0dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:03.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c0dffb640 10 monclient: dump: 2024-09-16T05:10:03.814 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.814 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.814 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.814 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.814 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.814 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.814 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.814 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.814 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.814 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c0dffb640 1 -- 172.21.15.114:0/2955129607 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f6c04004220 con 0x7f6c10136790 2024-09-16T05:10:03.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c0dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:10:03.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c0dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:10:03.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c0dffb640 1 -- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c10136790 msgr2=0x7f6c1013cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c0dffb640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c10136790 0x7f6c1013cb90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6c1013e3a0 tx=0x7f6c040046d0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c0dffb640 1 -- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c10136790 msgr2=0x7f6c1013cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:03.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c0dffb640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c10136790 0x7f6c1013cb90 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6c1013e3a0 tx=0x7f6c040046d0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.815 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:10:03.815 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:10:03.815 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:10:03.815 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:10:03.816 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 14, 2024-09-16T05:10:03.816 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 17927, 2024-09-16T05:10:03.816 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:10:03.816 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:10:03.816 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:10:03.816 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:10:03.816 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:03.816 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:10:03.816 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:03.816 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:10:03.816 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:10:03.816 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:03.816 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:03.817 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:03.817 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:03.817 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:03.817 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:03.817 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:03.817 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:03.817 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:03.817 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:03.817 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:03.820 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:03.820 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:10:03.820 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:10:03.820 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:03.820 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:03.821 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:03.821 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:03.821 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:03.821 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:03.821 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:03.821 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:03.822 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:03.822 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:03.822 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:03.822 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:03.822 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.822 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:10:03.822 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:10:03.822 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:10:03.822 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:10:03.822 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:10:03.822 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:10:03.822 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:03.822 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:03.823 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:10:03.823 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:10:03.823 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:10:03.823 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:10:03.823 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:10:03.823 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:10:03.823 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:10:03.823 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:03.823 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:10:03.823 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:03.823 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:03.823 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:03.823 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:03.824 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:03.824 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:03.824 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:03.824 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:03.824 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:03.824 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:03.824 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:03.824 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:03.824 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:10:03.824 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.825 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:10:03.825 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.825 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:10:03.825 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:10:03.825 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:03.825 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:03.825 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.825 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:03.825 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:10:03.825 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:03.825 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.825 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.825 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:03.826 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:10:03.826 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:03.826 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:03.826 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:03.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.826 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:10:03.826 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:10:03.826 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:03.826 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:03.826 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:03.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.826 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.826 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:10:03.826 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:10:03.827 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:03.827 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:03.827 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.827 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:03.827 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:10:03.827 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:03.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.827 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.827 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:03.827 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:10:03.827 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:03.827 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:03.828 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:03.828 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.828 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:10:03.828 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:10:03.828 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:03.828 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:03.828 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:03.828 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.828 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.828 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:10:03.828 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:10:03.828 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:03.828 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:03.829 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.829 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:03.829 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:10:03.829 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:03.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.829 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.829 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:03.829 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:10:03.829 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:03.829 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:03.829 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:03.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.829 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:10:03.829 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:10:03.830 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:03.830 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:03.830 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:03.830 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:03.830 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:03.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.830 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:10:03.830 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:10:03.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.830 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:03.830 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:03.830 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:10:03.830 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:03.830 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:03.831 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:10:03.831 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:03.831 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:03.831 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:03.831 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:10:03.831 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:03.831 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:03.831 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:03.831 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:10:03.831 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:10:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 10 librados: watch_flush enter 2024-09-16T05:10:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 10 librados: watch_flush exit 2024-09-16T05:10:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 20 client.54161.objecter shutdown clearing up homeless session... 2024-09-16T05:10:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 10 client.54161.objecter successfully canceled tick 2024-09-16T05:10:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 1 -- 172.21.15.114:0/2955129607 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6bfc037810 msgr2=0x7f6bfc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 1 --2- 172.21.15.114:0/2955129607 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6bfc037810 0x7f6bfc039cd0 secure :-1 s=READY pgs=3391 cs=0 l=1 rev1=1 crypto rx=0x7f6c000097c0 tx=0x7f6c00008040 comp rx=0 tx=0).stop 2024-09-16T05:10:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 10 monclient: shutdown 2024-09-16T05:10:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 1 -- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c100a5d00 msgr2=0x7f6c101cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 1 --2- 172.21.15.114:0/2955129607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c100a5d00 0x7f6c101cf2b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6c0801d090 tx=0x7f6c08019590 comp rx=0 tx=0).stop 2024-09-16T05:10:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c15a2b640 1 -- 172.21.15.114:0/2955129607 reap_dead start 2024-09-16T05:10:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c15a2b640 1 -- 172.21.15.114:0/2955129607 reap_dead start 2024-09-16T05:10:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 1 -- 172.21.15.114:0/2955129607 shutdown_connections 2024-09-16T05:10:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 1 -- 172.21.15.114:0/2955129607 >> 172.21.15.114:0/2955129607 conn(0x7f6c100a9a40 msgr2=0x7f6c100a82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 1 -- 172.21.15.114:0/2955129607 shutdown_connections 2024-09-16T05:10:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 1 -- 172.21.15.114:0/2955129607 wait complete. 2024-09-16T05:10:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.791+0000 7f6c1748e640 1 librados: shutdown 2024-09-16T05:10:03.835 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:10:03.835 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:10:03.835 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:10:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.947+0000 7f80ddea1640 10 monclient: get_monmap_and_config 2024-09-16T05:10:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.947+0000 7f80ddea1640 10 monclient: build_initial_monmap 2024-09-16T05:10:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 10 monclient: monmap: 2024-09-16T05:10:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:10:03.952305+0000 2024-09-16T05:10:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:10:03.952305+0000 2024-09-16T05:10:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:10:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:10:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:10:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:10:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:10:03.954 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 1 Processor -- start 2024-09-16T05:10:03.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 1 -- start start 2024-09-16T05:10:03.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 10 monclient: init 2024-09-16T05:10:03.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:03.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T05:10:03.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f80d81376e0 addr v1:172.21.15.114:6789/0 2024-09-16T05:10:03.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f80d81370a0 0x7f80d8137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 10 monclient(hunting): picked mon.noname-c con 0x7f80d81370a0 addr v2:172.21.15.149:3300/0 2024-09-16T05:10:03.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f80d80a5d00 addr v1:172.21.15.149:6789/0 2024-09-16T05:10:03.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80d80a5090 con 0x7f80d81376e0 2024-09-16T05:10:03.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80d809bd20 con 0x7f80d80a5d00 2024-09-16T05:10:03.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f80d8138000 con 0x7f80d81370a0 2024-09-16T05:10:03.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 10 monclient(hunting): authenticate will time out at 1088.581725s 2024-09-16T05:10:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d77fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f80d81376e0 0x7f80d80a4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36686/0 (socket says 172.21.15.114:36686) 2024-09-16T05:10:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d77fe640 1 -- 172.21.15.114:0/3923521377 learned_addr learned my addr 172.21.15.114:0/3923521377 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:10:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 1 --2- 172.21.15.114:0/3923521377 >> v2:172.21.15.149:3300/0 conn(0x7f80d81370a0 0x7f80d8137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 10 monclient(hunting): get_auth_request con 0x7f80d81370a0 auth_method 0 2024-09-16T05:10:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 -- 172.21.15.114:0/3923521377 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030431198 0 0) 0x7f80d80a5090 con 0x7f80d81376e0 2024-09-16T05:10:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:10:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient(hunting): my global_id is 27606 2024-09-16T05:10:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 -- 172.21.15.114:0/3923521377 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80c0003530 con 0x7f80d81376e0 2024-09-16T05:10:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 -- 172.21.15.114:0/3923521377 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3220692021 0 0) 0x7f80c0003530 con 0x7f80d81376e0 2024-09-16T05:10:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 -- 172.21.15.114:0/3923521377 >> v2:172.21.15.149:3300/0 conn(0x7f80d81370a0 msgr2=0x7f80d8137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 --2- 172.21.15.114:0/3923521377 >> v2:172.21.15.149:3300/0 conn(0x7f80d81370a0 0x7f80d8137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 -- 172.21.15.114:0/3923521377 >> v1:172.21.15.149:6789/0 conn(0x7f80d80a5d00 legacy=0x7f80d80a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient: _finish_hunting 0 2024-09-16T05:10:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:10:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:10:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 -- 172.21.15.114:0/3923521377 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d809c000 con 0x7f80d81376e0 2024-09-16T05:10:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient: _finish_auth 0 2024-09-16T05:10:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient: _check_auth_tickets 2024-09-16T05:10:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 5 monclient: authenticate success, global_id 27606 2024-09-16T05:10:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 -- 172.21.15.114:0/3923521377 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f80cc001680 con 0x7f80d81376e0 2024-09-16T05:10:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:10:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient: dump: 2024-09-16T05:10:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.974 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.974 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.974 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.974 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.974 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.974 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.974 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.974 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.974 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:10:03.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:03.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 -- 172.21.15.114:0/3923521377 >> v1:172.21.15.114:6789/0 conn(0x7f80d81376e0 legacy=0x7f80d80a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:10:03.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 --2- 172.21.15.114:0/3923521377 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80c00039c0 0x7f80c0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient(hunting): picked mon.b con 0x7f80c00039c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:03.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 --2- 172.21.15.114:0/3923521377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c0004810 0x7f80c0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient(hunting): picked mon.a con 0x7f80c0004810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:03.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 --2- 172.21.15.114:0/3923521377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80c00052f0 0x7f80c000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient(hunting): picked mon.c con 0x7f80c00052f0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:03.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 -- 172.21.15.114:0/3923521377 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f80c000c190 con 0x7f80c0004810 2024-09-16T05:10:03.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 -- 172.21.15.114:0/3923521377 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f80c000c340 con 0x7f80c00039c0 2024-09-16T05:10:03.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 1 -- 172.21.15.114:0/3923521377 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f80c000c540 con 0x7f80c00052f0 2024-09-16T05:10:03.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:03.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:10:03.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80ddea1640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:03.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 1 --2- 172.21.15.114:0/3923521377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80c00052f0 0x7f80c000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 10 monclient(hunting): get_auth_request con 0x7f80c00052f0 auth_method 0 2024-09-16T05:10:03.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:03.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d77fe640 1 --2- 172.21.15.114:0/3923521377 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80c00039c0 0x7f80c0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 1 --2- 172.21.15.114:0/3923521377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c0004810 0x7f80c0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 10 monclient(hunting): get_auth_request con 0x7f80c0004810 auth_method 0 2024-09-16T05:10:03.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:03.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:10:03.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d77fe640 10 monclient(hunting): get_auth_request con 0x7f80c00039c0 auth_method 0 2024-09-16T05:10:03.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:03.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:10:03.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:10:03.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 10 monclient(hunting): handle_auth_done global_id 27606 payload 293 2024-09-16T05:10:03.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 1 -- 172.21.15.114:0/3923521377 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80c00039c0 msgr2=0x7f80c0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:03.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 1 --2- 172.21.15.114:0/3923521377 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80c00039c0 0x7f80c0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 1 -- 172.21.15.114:0/3923521377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c0004810 msgr2=0x7f80c0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:10:03.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 1 --2- 172.21.15.114:0/3923521377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c0004810 0x7f80c0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:10:03.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 1 monclient: found mon.c 2024-09-16T05:10:03.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:03.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:03.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 1 -- 172.21.15.114:0/3923521377 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c0003530 con 0x7f80c00052f0 2024-09-16T05:10:03.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 10 monclient: _finish_auth 0 2024-09-16T05:10:03.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:10:03.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.951+0000 7f80d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:03.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d7fff640 1 -- 172.21.15.114:0/3923521377 reap_dead start 2024-09-16T05:10:03.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d7fff640 1 --2- 172.21.15.114:0/3923521377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80c00052f0 0x7f80c000b9d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f80d0001e40 tx=0x7f80d001c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=654855bc4ac9ac7a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d67fc640 1 -- 172.21.15.114:0/3923521377 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d0001a90 con 0x7f80c00052f0 2024-09-16T05:10:03.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:03.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d67fc640 10 monclient: dump: 2024-09-16T05:10:03.979 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.979 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.979 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.979 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.979 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.979 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.979 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.980 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.980 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.980 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d67fc640 1 -- 172.21.15.114:0/3923521377 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80d0003b80 con 0x7f80c00052f0 2024-09-16T05:10:03.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:03.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:03.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 10 monclient: get_monmap_and_config success 2024-09-16T05:10:03.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d67fc640 1 -- 172.21.15.114:0/3923521377 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d00050d0 con 0x7f80c00052f0 2024-09-16T05:10:03.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 4 set_mon_vals no callback set 2024-09-16T05:10:03.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:03.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d67fc640 10 monclient: dump: 2024-09-16T05:10:03.980 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.981 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 10 monclient: shutdown 2024-09-16T05:10:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 -- 172.21.15.114:0/3923521377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80c00052f0 msgr2=0x7f80c000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 --2- 172.21.15.114:0/3923521377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80c00052f0 0x7f80c000b9d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f80d0001e40 tx=0x7f80d001c740 comp rx=0 tx=0).stop 2024-09-16T05:10:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 -- 172.21.15.114:0/3923521377 shutdown_connections 2024-09-16T05:10:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 --2- 172.21.15.114:0/3923521377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80c00052f0 0x7f80c000b9d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 -- 172.21.15.114:0/3923521377 >> 172.21.15.114:0/3923521377 conn(0x7f80d80a9a40 msgr2=0x7f80d80a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 -- 172.21.15.114:0/3923521377 shutdown_connections 2024-09-16T05:10:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 -- 172.21.15.114:0/3923521377 wait complete. 2024-09-16T05:10:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 10 monclient: build_initial_monmap 2024-09-16T05:10:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 10 monclient: monmap: 2024-09-16T05:10:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:10:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:10:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:10:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:10:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:10:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:10:03.983 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 librados: starting msgr at 2024-09-16T05:10:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 librados: starting objecter 2024-09-16T05:10:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 Processor -- start 2024-09-16T05:10:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 -- start start 2024-09-16T05:10:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 librados: setting wanted keys 2024-09-16T05:10:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 librados: calling monclient init 2024-09-16T05:10:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 10 monclient: init 2024-09-16T05:10:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:10:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c00052f0 0x7f80d81cec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 10 monclient(hunting): picked mon.noname-a con 0x7f80c00052f0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d8136400 0x7f80d8136860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 10 monclient(hunting): picked mon.noname-c con 0x7f80d8136400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d813bf20 0x7f80d813c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 10 monclient(hunting): picked mon.noname-b con 0x7f80d813bf20 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f80d8137e50 con 0x7f80c00052f0 2024-09-16T05:10:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f80d8137cd0 con 0x7f80d813bf20 2024-09-16T05:10:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f80d80a4270 con 0x7f80d8136400 2024-09-16T05:10:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80ddea1640 10 monclient(hunting): authenticate will time out at 1088.587785s 2024-09-16T05:10:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d6ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d8136400 0x7f80d8136860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c00052f0 0x7f80d81cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d6ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d8136400 0x7f80d8136860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:43766/0 (socket says 172.21.15.114:43766) 2024-09-16T05:10:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c00052f0 0x7f80d81cec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59624/0 (socket says 172.21.15.114:59624) 2024-09-16T05:10:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d6ffd640 1 -- 172.21.15.114:0/4118762345 learned_addr learned my addr 172.21.15.114:0/4118762345 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d6ffd640 10 monclient(hunting): get_auth_request con 0x7f80d8136400 auth_method 0 2024-09-16T05:10:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d77fe640 10 monclient(hunting): get_auth_request con 0x7f80c00052f0 auth_method 0 2024-09-16T05:10:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.955+0000 7f80d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d7fff640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d813bf20 0x7f80d813c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d7fff640 10 monclient(hunting): get_auth_request con 0x7f80d813bf20 auth_method 0 2024-09-16T05:10:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d77fe640 10 monclient(hunting): handle_auth_done global_id 27609 payload 824 2024-09-16T05:10:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d77fe640 1 -- 172.21.15.114:0/4118762345 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d8136400 msgr2=0x7f80d8136860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d77fe640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d8136400 0x7f80d8136860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d77fe640 1 -- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d813bf20 msgr2=0x7f80d813c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d77fe640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d813bf20 0x7f80d813c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d77fe640 10 monclient: _finish_hunting 0 2024-09-16T05:10:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d77fe640 1 monclient: found mon.noname-a 2024-09-16T05:10:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:10:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d77fe640 1 -- 172.21.15.114:0/4118762345 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d813fbd0 con 0x7f80c00052f0 2024-09-16T05:10:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d6ffd640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80d8136400 0x7f80d8136860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:10:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d77fe640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c00052f0 0x7f80d81cec80 secure :-1 s=READY pgs=8185 cs=0 l=1 rev1=1 crypto rx=0x7f80cc004f80 tx=0x7f80cc007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 client.?.objecter ms_handle_connect 0x7f80c00052f0 2024-09-16T05:10:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:10:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 1 -- 172.21.15.114:0/4118762345 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc00f750 con 0x7f80c00052f0 2024-09-16T05:10:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:10:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 monclient: dump: 2024-09-16T05:10:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.988 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.988 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.988 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.988 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.988 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.988 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.988 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.988 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.988 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 monclient: _finish_auth 0 2024-09-16T05:10:03.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 monclient: _check_auth_tickets 2024-09-16T05:10:03.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:03.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 1 -- 172.21.15.114:0/4118762345 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80cc018070 con 0x7f80c00052f0 2024-09-16T05:10:03.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:03.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 1 -- 172.21.15.114:0/4118762345 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc00fb30 con 0x7f80c00052f0 2024-09-16T05:10:03.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80ddea1640 5 monclient: authenticate success, global_id 27609 2024-09-16T05:10:03.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80ddea1640 10 monclient: _renew_subs 2024-09-16T05:10:03.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80ddea1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:03.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80ddea1640 1 -- 172.21.15.114:0/4118762345 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80d80a8a00 con 0x7f80c00052f0 2024-09-16T05:10:03.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d5ffb640 4 set_mon_vals no callback set 2024-09-16T05:10:03.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:03.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 monclient: dump: 2024-09-16T05:10:03.989 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.989 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.989 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.989 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.989 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.990 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80ddea1640 10 client.27609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:10:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80ddea1640 10 monclient: _renew_subs 2024-09-16T05:10:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80ddea1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:10:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80ddea1640 1 -- 172.21.15.114:0/4118762345 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80d8073d20 con 0x7f80c00052f0 2024-09-16T05:10:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80ddea1640 1 librados: init done 2024-09-16T05:10:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80ddea1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80ddea1640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d813c8e0 0x7f80d8137fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80ddea1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80ddea1640 1 -- 172.21.15.114:0/4118762345 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f80d813eac0 con 0x7f80d813c8e0 2024-09-16T05:10:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 1 -- 172.21.15.114:0/4118762345 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f80cc00fcd0 con 0x7f80c00052f0 2024-09-16T05:10:03.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d6ffd640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d813c8e0 0x7f80d8137fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d6ffd640 10 monclient: get_auth_request con 0x7f80d813c8e0 auth_method 0 2024-09-16T05:10:03.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 1 --2- 172.21.15.114:0/4118762345 >> v2:172.21.15.149:6812/1185531135 conn(0x7f80c4037810 0x7f80c4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 1 -- 172.21.15.114:0/4118762345 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7158+0+0 (secure 0 0 0) 0x7f80cc0494b0 con 0x7f80c00052f0 2024-09-16T05:10:03.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 client.27609.objecter ms_dispatch 0x7f80d8000ba0 osd_map(82..82 src has 1..82) 2024-09-16T05:10:03.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 3 client.27609.objecter handle_osd_map got epochs [82,82] > 0 2024-09-16T05:10:03.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 3 client.27609.objecter handle_osd_map decoding full epoch 82 2024-09-16T05:10:03.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 20 client.27609.objecter dump_active .. 0 homeless 2024-09-16T05:10:03.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d7fff640 1 --2- 172.21.15.114:0/4118762345 >> v2:172.21.15.149:6812/1185531135 conn(0x7f80c4037810 0x7f80c4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d7fff640 10 monclient: get_auth_request con 0x7f80c4037810 auth_method 0 2024-09-16T05:10:03.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_done global_id 19621 payload 824 2024-09-16T05:10:03.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d6ffd640 1 -- 172.21.15.114:0/4118762345 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80d80a4600 con 0x7f80d813c8e0 2024-09-16T05:10:03.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 client.27609.objecter ms_handle_connect 0x7f80d813c8e0 2024-09-16T05:10:03.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d6ffd640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d813c8e0 0x7f80d8137fd0 secure :-1 s=READY pgs=4983 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009ee0 tx=0x7f80c8012700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 client.27609.objecter resend_mon_ops 2024-09-16T05:10:03.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 1 -- 172.21.15.114:0/4118762345 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c8022070 con 0x7f80d813c8e0 2024-09-16T05:10:03.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:10:03.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 monclient: dump: 2024-09-16T05:10:03.992 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.992 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.992 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.993 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80d7fff640 1 --2- 172.21.15.114:0/4118762345 >> v2:172.21.15.149:6812/1185531135 conn(0x7f80c4037810 0x7f80c4039cd0 secure :-1 s=READY pgs=3392 cs=0 l=1 rev1=1 crypto rx=0x7f80c000c710 tx=0x7f80d0001520 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 client.27609.objecter ms_handle_connect 0x7f80c4037810 2024-09-16T05:10:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 1 -- 172.21.15.114:0/4118762345 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f80c8015630 con 0x7f80d813c8e0 2024-09-16T05:10:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:10:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 1 -- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d813c8e0 msgr2=0x7f80d8137fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d813c8e0 0x7f80d8137fd0 secure :-1 s=READY pgs=4983 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009ee0 tx=0x7f80c8012700 comp rx=0 tx=0).stop 2024-09-16T05:10:03.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 1 -- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d813c8e0 msgr2=0x7f80d8137fd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:03.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.959+0000 7f80bffff640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d813c8e0 0x7f80d8137fd0 secure :-1 s=CLOSED pgs=4983 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009ee0 tx=0x7f80c8012700 comp rx=0 tx=0).stop 2024-09-16T05:10:03.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80ddea1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:10:03.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80ddea1640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d8138510 0x7f80d8138920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:03.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80ddea1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:03.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80ddea1640 1 -- 172.21.15.114:0/4118762345 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f80d813e4a0 con 0x7f80d8138510 2024-09-16T05:10:03.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80d6ffd640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d8138510 0x7f80d8138920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:03.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80d6ffd640 10 monclient: get_auth_request con 0x7f80d8138510 auth_method 0 2024-09-16T05:10:03.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:03.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:03.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:03.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:03.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:03.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:03.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80d6ffd640 10 monclient(hunting): handle_auth_done global_id 19624 payload 824 2024-09-16T05:10:03.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80d6ffd640 1 -- 172.21.15.114:0/4118762345 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f80d813ccc0 con 0x7f80d8138510 2024-09-16T05:10:03.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80bffff640 10 client.27609.objecter ms_handle_connect 0x7f80d8138510 2024-09-16T05:10:03.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80bffff640 10 client.27609.objecter resend_mon_ops 2024-09-16T05:10:03.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80d6ffd640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d8138510 0x7f80d8138920 secure :-1 s=READY pgs=4984 cs=0 l=1 rev1=1 crypto rx=0x7f80c801f000 tx=0x7f80c8004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:03.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80bffff640 1 -- 172.21.15.114:0/4118762345 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c8022040 con 0x7f80d8138510 2024-09-16T05:10:03.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:03.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:10:03.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.967+0000 7f80bffff640 10 monclient: dump: 2024-09-16T05:10:03.998 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:03.998 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:03.998 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.998 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:03.998 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:03.998 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:03.998 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:03.998 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:03.999 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:03.999 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:03.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80bffff640 1 -- 172.21.15.114:0/4118762345 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f80c801d440 con 0x7f80d8138510 2024-09-16T05:10:03.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80bffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:10:03.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80bffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:10:03.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80bffff640 1 -- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d8138510 msgr2=0x7f80d8138920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:03.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80bffff640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d8138510 0x7f80d8138920 secure :-1 s=READY pgs=4984 cs=0 l=1 rev1=1 crypto rx=0x7f80c801f000 tx=0x7f80c8004740 comp rx=0 tx=0).stop 2024-09-16T05:10:03.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80bffff640 1 -- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d8138510 msgr2=0x7f80d8138920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:03.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80bffff640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80d8138510 0x7f80d8138920 unknown :-1 s=CLOSED pgs=4984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:03.999 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:10:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:10:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:10:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:10:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 14, 2024-09-16T05:10:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 161916, 2024-09-16T05:10:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:10:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:10:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:10:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:10:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 3, 2024-09-16T05:10:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:10:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:10:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:10:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:10:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:10:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:10:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:10:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:10:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:10:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:10:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:10:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:10:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:10:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:10:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:10:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:10:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:10:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:10:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:10:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:10:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:10:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:10:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:10:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:10:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:10:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:10:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:10:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:10:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:10:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:10:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:10:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:10:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:10:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:10:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:10:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:10:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:10:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:10:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:10:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:10:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:10:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:10:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:10:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:10:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 11 2024-09-16T05:10:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:10:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:10:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:10:04.012 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:10:04.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80ddea1640 10 librados: watch_flush enter 2024-09-16T05:10:04.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80ddea1640 10 librados: watch_flush exit 2024-09-16T05:10:04.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80ddea1640 20 client.27609.objecter shutdown clearing up homeless session... 2024-09-16T05:10:04.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80ddea1640 10 client.27609.objecter successfully canceled tick 2024-09-16T05:10:04.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80ddea1640 1 -- 172.21.15.114:0/4118762345 >> v2:172.21.15.149:6812/1185531135 conn(0x7f80c4037810 msgr2=0x7f80c4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:04.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80ddea1640 1 --2- 172.21.15.114:0/4118762345 >> v2:172.21.15.149:6812/1185531135 conn(0x7f80c4037810 0x7f80c4039cd0 secure :-1 s=READY pgs=3392 cs=0 l=1 rev1=1 crypto rx=0x7f80c000c710 tx=0x7f80d0001520 comp rx=0 tx=0).stop 2024-09-16T05:10:04.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80ddea1640 10 monclient: shutdown 2024-09-16T05:10:04.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80ddea1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:04.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80ddea1640 1 -- 172.21.15.114:0/4118762345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c00052f0 msgr2=0x7f80d81cec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:04.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80ddea1640 1 --2- 172.21.15.114:0/4118762345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80c00052f0 0x7f80d81cec80 secure :-1 s=READY pgs=8185 cs=0 l=1 rev1=1 crypto rx=0x7f80cc004f80 tx=0x7f80cc007430 comp rx=0 tx=0).stop 2024-09-16T05:10:04.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80d7fff640 1 -- 172.21.15.114:0/4118762345 reap_dead start 2024-09-16T05:10:04.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80d7fff640 1 -- 172.21.15.114:0/4118762345 reap_dead start 2024-09-16T05:10:04.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80ddea1640 1 -- 172.21.15.114:0/4118762345 shutdown_connections 2024-09-16T05:10:04.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80ddea1640 1 -- 172.21.15.114:0/4118762345 >> 172.21.15.114:0/4118762345 conn(0x7f80d80a9a40 msgr2=0x7f80d80a7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:04.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80ddea1640 1 -- 172.21.15.114:0/4118762345 shutdown_connections 2024-09-16T05:10:04.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80ddea1640 1 -- 172.21.15.114:0/4118762345 wait complete. 2024-09-16T05:10:04.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:03.971+0000 7f80ddea1640 1 librados: shutdown 2024-09-16T05:10:04.015 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:10:04.016 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T05:10:04.016 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-16T05:10:04.016 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T05:10:04.016 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:10:04.016 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T05:10:04.016 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:10:04.043 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T05:10:04.043 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T05:10:04.043 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T05:10:04.043 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T05:10:19.044 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T05:10:19.045 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T05:10:19.046 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:10:19.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:10:19.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 10 monclient: get_monmap_and_config 2024-09-16T05:10:19.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 10 monclient: build_initial_monmap 2024-09-16T05:10:19.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 10 monclient: monmap: 2024-09-16T05:10:19.174 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:19.175 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:19.175 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:10:19.173263+0000 2024-09-16T05:10:19.175 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:10:19.173263+0000 2024-09-16T05:10:19.175 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:19.175 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:19.175 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:10:19.175 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:10:19.175 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:10:19.175 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:10:19.175 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:10:19.175 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:19.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 1 Processor -- start 2024-09-16T05:10:19.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 1 -- start start 2024-09-16T05:10:19.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 10 monclient: init 2024-09-16T05:10:19.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:19.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-16T05:10:19.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fcea009dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:19.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 10 monclient(hunting): picked mon.noname-b con 0x7fcea00a0ec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:19.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcea009e000 addr v1:172.21.15.114:6789/0 2024-09-16T05:10:19.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 0x7fcea013ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:19.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 10 monclient(hunting): picked mon.noname-a con 0x7fcea0139640 addr v2:172.21.15.114:3300/0 2024-09-16T05:10:19.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcea009a760 con 0x7fcea009e000 2024-09-16T05:10:19.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:19.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fcea00a12a0 con 0x7fcea0139640 2024-09-16T05:10:19.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:19.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fcea00994b0 con 0x7fcea00a0ec0 2024-09-16T05:10:19.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:19.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea842b640 10 monclient(hunting): authenticate will time out at 1103.802958s 2024-09-16T05:10:19.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea61c7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fcea009dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:19.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea61c7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fcea009dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:60980/0 (socket says 172.21.15.114:60980) 2024-09-16T05:10:19.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea69c8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 0x7fcea013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:19.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea59c6640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fcea009e000 0x7fcea009e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38160/0 (socket says 172.21.15.114:38160) 2024-09-16T05:10:19.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea61c7640 1 -- 172.21.15.114:0/1398196621 learned_addr learned my addr 172.21.15.114:0/1398196621 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:19.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea61c7640 10 monclient(hunting): get_auth_request con 0x7fcea00a0ec0 auth_method 0 2024-09-16T05:10:19.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea61c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:19.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea61c7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:19.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea61c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:19.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea69c8640 10 monclient(hunting): get_auth_request con 0x7fcea0139640 auth_method 0 2024-09-16T05:10:19.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea69c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:19.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea69c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:19.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea69c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:19.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea61c7640 1 -- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 msgr2=0x7fcea009dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:10:19.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea61c7640 1 -- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 msgr2=0x7fcea009dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:10:19.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea69c8640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 msgr2=0x7fcea013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:10:19.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea69c8640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 msgr2=0x7fcea013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:10:19.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea61c7640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fcea009dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:10:19.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea69c8640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 0x7fcea013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:10:19.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea69c8640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 0x7fcea013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:10:19.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.170+0000 7fcea61c7640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fcea009dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:10:19.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea61c7640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fcea009dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:19.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea69c8640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 0x7fcea013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:19.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea61c7640 10 monclient(hunting): get_auth_request con 0x7fcea00a0ec0 auth_method 0 2024-09-16T05:10:19.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea61c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:19.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea61c7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:19.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea61c7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:19.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea69c8640 10 monclient(hunting): get_auth_request con 0x7fcea0139640 auth_method 0 2024-09-16T05:10:19.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea69c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:19.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea69c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:19.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea69c8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:19.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea61c7640 1 -- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 msgr2=0x7fcea009dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:10:19.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea61c7640 1 -- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 msgr2=0x7fcea009dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:10:19.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea61c7640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fcea009dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:10:19.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea69c8640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 msgr2=0x7fcea013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:10:19.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea69c8640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 msgr2=0x7fcea013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:10:19.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea69c8640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 0x7fcea013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:10:19.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea61c7640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fcea009dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:10:19.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.374+0000 7fcea69c8640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 0x7fcea013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:10:19.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea61c7640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fcea009dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:19.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea69c8640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 0x7fcea013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:19.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea61c7640 10 monclient(hunting): get_auth_request con 0x7fcea00a0ec0 auth_method 0 2024-09-16T05:10:19.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea61c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:19.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea61c7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:19.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea61c7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:19.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea69c8640 10 monclient(hunting): get_auth_request con 0x7fcea0139640 auth_method 0 2024-09-16T05:10:19.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea69c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:19.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea69c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:19.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea69c8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:19.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea61c7640 1 -- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 msgr2=0x7fcea009dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:10:19.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea61c7640 1 -- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 msgr2=0x7fcea009dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:10:19.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea69c8640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 msgr2=0x7fcea013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:10:19.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea69c8640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 msgr2=0x7fcea013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:10:19.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea69c8640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 0x7fcea013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:10:19.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea61c7640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fcea009dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:10:19.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea61c7640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fcea009dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:10:19.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:19.778+0000 7fcea69c8640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 0x7fcea013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:10:20.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea61c7640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fcea009dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:20.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea61c7640 10 monclient(hunting): get_auth_request con 0x7fcea00a0ec0 auth_method 0 2024-09-16T05:10:20.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea61c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:20.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea61c7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:20.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea61c7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:20.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea69c8640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 0x7fcea013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:20.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea69c8640 10 monclient(hunting): get_auth_request con 0x7fcea0139640 auth_method 0 2024-09-16T05:10:20.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea69c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:20.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea69c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:20.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea69c8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:10:20.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea61c7640 1 -- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 msgr2=0x7fcea009dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:10:20.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea61c7640 1 -- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 msgr2=0x7fcea009dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:10:20.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea61c7640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fcea009dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:10:20.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea69c8640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 msgr2=0x7fcea013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:10:20.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea69c8640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 msgr2=0x7fcea013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:10:20.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea61c7640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fcea009dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:10:20.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea69c8640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 0x7fcea013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:10:20.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:20.582+0000 7fcea69c8640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 0x7fcea013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:10:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 10 monclient(hunting): tick 2024-09-16T05:10:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:10:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 1 monclient(hunting): continuing hunt 2024-09-16T05:10:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:10:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 1 -- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 msgr2=0x7fcea009dac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:10:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fcea009dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 msgr2=0x7fcea013ba30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:10:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.114:3300/0 conn(0x7fcea0139640 0x7fcea013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 1 -- 172.21.15.114:0/1398196621 >> v1:172.21.15.114:6789/0 conn(0x7fcea009e000 legacy=0x7fcea009e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 10 monclient: _add_conns ranks=[1,4] 2024-09-16T05:10:22.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce840036f0 addr v1:172.21.15.149:6789/0 2024-09-16T05:10:22.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 0x7fce84009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:22.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 10 monclient(hunting): picked mon.noname-c con 0x7fce84007380 addr v2:172.21.15.149:3300/0 2024-09-16T05:10:22.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 1 -- 172.21.15.114:0/1398196621 --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcea00a1870 con 0x7fce840036f0 2024-09-16T05:10:22.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:22.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 1 -- 172.21.15.114:0/1398196621 --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fcea00a12a0 con 0x7fce84007380 2024-09-16T05:10:22.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fce97fff640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:22.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fcea59c6640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 msgr2=0x7fce84009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:10:22.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fcea61c7640 1 -- 172.21.15.114:0/1398196621 >> v1:172.21.15.149:6789/0 conn(0x7fce840036f0 legacy=0x7fce84005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:10:22.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.170+0000 7fcea59c6640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 0x7fce84009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:10:22.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.374+0000 7fcea61c7640 1 -- 172.21.15.114:0/1398196621 >> v1:172.21.15.149:6789/0 conn(0x7fce840036f0 legacy=0x7fce84005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:10:22.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.374+0000 7fcea59c6640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 msgr2=0x7fce84009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:10:22.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.374+0000 7fcea59c6640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 0x7fce84009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:10:22.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.778+0000 7fcea61c7640 1 -- 172.21.15.114:0/1398196621 >> v1:172.21.15.149:6789/0 conn(0x7fce840036f0 legacy=0x7fce84005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:10:22.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.778+0000 7fcea59c6640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 msgr2=0x7fce84009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:10:22.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:22.778+0000 7fcea59c6640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 0x7fce84009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:10:23.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:23.582+0000 7fcea61c7640 1 -- 172.21.15.114:0/1398196621 >> v1:172.21.15.149:6789/0 conn(0x7fce840036f0 legacy=0x7fce84005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:10:23.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:23.582+0000 7fcea59c6640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 msgr2=0x7fce84009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:10:23.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:23.582+0000 7fcea59c6640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 0x7fce84009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:10:25.117 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: Running main() from gmock_main.cc 2024-09-16T05:10:25.117 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-09-16T05:10:25.117 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [----------] Global test environment set-up. 2024-09-16T05:10:25.117 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-09-16T05:10:25.117 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-09-16T05:10:25.117 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (4999 ms) 2024-09-16T05:10:25.117 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-09-16T05:10:25.117 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (4970 ms) 2024-09-16T05:10:25.118 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-09-16T05:10:25.118 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (3476 ms) 2024-09-16T05:10:25.118 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-09-16T05:10:25.118 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5051 ms) 2024-09-16T05:10:25.118 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-09-16T05:10:25.118 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7011 ms) 2024-09-16T05:10:25.118 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (25507 ms total) 2024-09-16T05:10:25.118 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: 2024-09-16T05:10:25.118 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-09-16T05:10:25.118 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-09-16T05:10:25.118 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (4960 ms) 2024-09-16T05:10:25.118 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-09-16T05:10:25.118 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6035 ms) 2024-09-16T05:10:25.118 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-09-16T05:10:25.119 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8003 ms) 2024-09-16T05:10:25.119 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-09-16T05:10:25.119 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (5999 ms) 2024-09-16T05:10:25.119 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-09-16T05:10:25.119 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4026 ms) 2024-09-16T05:10:25.119 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-09-16T05:10:25.119 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: Deleting snap 4 in pool ReusePurgedSnapsmithi149-30274-11. 2024-09-16T05:10:25.119 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: Waiting for snaps to purge. 2024-09-16T05:10:25.119 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (40950 ms) 2024-09-16T05:10:25.119 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (69973 ms total) 2024-09-16T05:10:25.119 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: 2024-09-16T05:10:25.119 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [----------] Global test environment tear-down 2024-09-16T05:10:25.119 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (95480 ms total) 2024-09-16T05:10:25.119 INFO:tasks.workunit.client.0.smithi149.stdout: snapshots: [ PASSED ] 11 tests. 2024-09-16T05:10:25.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 10 monclient(hunting): tick 2024-09-16T05:10:25.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:10:25.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 1 monclient(hunting): continuing hunt 2024-09-16T05:10:25.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:10:25.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 msgr2=0x7fce84009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:10:25.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 0x7fce84009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 1 -- 172.21.15.114:0/1398196621 >> v1:172.21.15.149:6789/0 conn(0x7fce840036f0 legacy=0x7fce84005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:10:25.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-16T05:10:25.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcea009e000 addr v1:172.21.15.114:6789/0 2024-09-16T05:10:25.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 0x7fce840097a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 10 monclient(hunting): picked mon.noname-c con 0x7fce84007380 addr v2:172.21.15.149:3300/0 2024-09-16T05:10:25.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fce8400a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 10 monclient(hunting): picked mon.noname-b con 0x7fcea00a0ec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:25.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 1 -- 172.21.15.114:0/1398196621 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce8400af30 con 0x7fcea009e000 2024-09-16T05:10:25.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 1 -- 172.21.15.114:0/1398196621 --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fcea00a12a0 con 0x7fce84007380 2024-09-16T05:10:25.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 1 -- 172.21.15.114:0/1398196621 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fcea00994b0 con 0x7fcea00a0ec0 2024-09-16T05:10:25.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fce97fff640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:25.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fcea69c8640 1 -- 172.21.15.114:0/1398196621 reap_dead start 2024-09-16T05:10:25.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fcea59c6640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 msgr2=0x7fce840097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:10:25.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.170+0000 7fcea59c6640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 0x7fce840097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:10:25.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fce8400a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:25.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 10 monclient(hunting): get_auth_request con 0x7fcea00a0ec0 auth_method 0 2024-09-16T05:10:25.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:25.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:25.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:25.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:25.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:25.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:25.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 10 monclient(hunting): handle_auth_done global_id 54182 payload 274 2024-09-16T05:10:25.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 1 -- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 msgr2=0x7fce840097a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:10:25.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 0x7fce840097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 1 -- 172.21.15.114:0/1398196621 >> v1:172.21.15.114:6789/0 conn(0x7fcea009e000 legacy=0x7fce84009ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 10 monclient: _finish_hunting 0 2024-09-16T05:10:25.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 1 monclient: found mon.noname-b 2024-09-16T05:10:25.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:25.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:10:25.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.174+0000 7fcea69c8640 1 -- 172.21.15.114:0/1398196621 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea00a1870 con 0x7fcea00a0ec0 2024-09-16T05:10:25.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea69c8640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fce8400a660 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcea0074130 tx=0x7fce9801daf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f32b35ac66f894ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:25.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea51c5640 1 -- 172.21.15.114:0/1398196621 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce980ac070 con 0x7fcea00a0ec0 2024-09-16T05:10:25.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea51c5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea51c5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:10:25.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea51c5640 10 monclient: dump: 2024-09-16T05:10:25.193 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.193 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.194 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.194 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.194 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.194 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.194 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.194 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.194 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.194 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea51c5640 10 monclient: _finish_auth 0 2024-09-16T05:10:25.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea51c5640 10 monclient: _check_auth_tickets 2024-09-16T05:10:25.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea51c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:25.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea51c5640 1 -- 172.21.15.114:0/1398196621 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce980a69f0 con 0x7fcea00a0ec0 2024-09-16T05:10:25.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea51c5640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:25.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea51c5640 1 -- 172.21.15.114:0/1398196621 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce980a6de0 con 0x7fcea00a0ec0 2024-09-16T05:10:25.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea51c5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea51c5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:25.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea51c5640 10 monclient: dump: 2024-09-16T05:10:25.195 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.196 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.196 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.196 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.196 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.196 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.196 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.196 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.196 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.196 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea842b640 5 monclient: authenticate success, global_id 54182 2024-09-16T05:10:25.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea842b640 10 monclient: get_monmap_and_config success 2024-09-16T05:10:25.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea842b640 4 set_mon_vals no callback set 2024-09-16T05:10:25.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea842b640 10 monclient: shutdown 2024-09-16T05:10:25.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea842b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:25.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea842b640 1 -- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 msgr2=0x7fce8400a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea842b640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fce8400a660 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcea0074130 tx=0x7fce9801daf0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea842b640 1 -- 172.21.15.114:0/1398196621 shutdown_connections 2024-09-16T05:10:25.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea842b640 1 --2- 172.21.15.114:0/1398196621 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00a0ec0 0x7fce8400a660 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea842b640 1 --2- 172.21.15.114:0/1398196621 >> v2:172.21.15.149:3300/0 conn(0x7fce84007380 0x7fce840097a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea842b640 1 -- 172.21.15.114:0/1398196621 >> 172.21.15.114:0/1398196621 conn(0x7fcea00a1fb0 msgr2=0x7fcea00a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:25.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea842b640 1 -- 172.21.15.114:0/1398196621 shutdown_connections 2024-09-16T05:10:25.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.178+0000 7fcea842b640 1 -- 172.21.15.114:0/1398196621 wait complete. 2024-09-16T05:10:25.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 10 monclient: build_initial_monmap 2024-09-16T05:10:25.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 10 monclient: monmap: 2024-09-16T05:10:25.199 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:25.199 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:10:25.199 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:10:25.199 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:10:25.199 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:25.200 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.200 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:10:25.200 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:10:25.200 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:10:25.200 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 1 librados: starting msgr at 2024-09-16T05:10:25.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 1 librados: starting objecter 2024-09-16T05:10:25.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 1 Processor -- start 2024-09-16T05:10:25.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 1 -- start start 2024-09-16T05:10:25.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 1 librados: setting wanted keys 2024-09-16T05:10:25.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 1 librados: calling monclient init 2024-09-16T05:10:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 10 monclient: init 2024-09-16T05:10:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:10:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce84007380 0x7fcea0071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 10 monclient(hunting): picked mon.noname-b con 0x7fce84007380 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea009e000 0x7fcea0071560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 10 monclient(hunting): picked mon.noname-c con 0x7fcea009e000 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcea00a0ec0 0x7fcea0071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 10 monclient(hunting): picked mon.noname-a con 0x7fcea00a0ec0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcea009c3b0 con 0x7fcea00a0ec0 2024-09-16T05:10:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fcea009c230 con 0x7fce84007380 2024-09-16T05:10:25.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fcea009d1c0 con 0x7fcea009e000 2024-09-16T05:10:25.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:25.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea842b640 10 monclient(hunting): authenticate will time out at 1109.814532s 2024-09-16T05:10:25.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea61c7640 1 -- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce84007380 msgr2=0x7fcea0071020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:10:25.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea61c7640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce84007380 0x7fcea0071020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:10:25.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea009e000 0x7fcea0071560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:25.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea009e000 0x7fcea0071560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:32790/0 (socket says 172.21.15.114:32790) 2024-09-16T05:10:25.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 1 -- 172.21.15.114:0/2357988592 learned_addr learned my addr 172.21.15.114:0/2357988592 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:25.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 10 monclient(hunting): get_auth_request con 0x7fcea009e000 auth_method 0 2024-09-16T05:10:25.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 10 monclient(hunting): handle_auth_done global_id 54185 payload 824 2024-09-16T05:10:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 1 -- 172.21.15.114:0/2357988592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce84007380 msgr2=0x7fcea0071020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:10:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 1 --2- 172.21.15.114:0/2357988592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce84007380 0x7fcea0071020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 1 -- 172.21.15.114:0/2357988592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcea00a0ec0 msgr2=0x7fcea0071aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 1 --2- 172.21.15.114:0/2357988592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcea00a0ec0 0x7fcea0071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 10 monclient: _finish_hunting 0 2024-09-16T05:10:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 1 monclient: found mon.noname-c 2024-09-16T05:10:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:10:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 1 -- 172.21.15.114:0/2357988592 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea012a880 con 0x7fcea009e000 2024-09-16T05:10:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fcea59c6640 1 --2- 172.21.15.114:0/2357988592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea009e000 0x7fcea0071560 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fce900012c0 tx=0x7fce9000cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fce96ffd640 10 client.?.objecter ms_handle_connect 0x7fcea009e000 2024-09-16T05:10:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.182+0000 7fce96ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:10:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 1 -- 172.21.15.114:0/2357988592 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce90018070 con 0x7fcea009e000 2024-09-16T05:10:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:10:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 10 monclient: dump: 2024-09-16T05:10:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.207 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 10 monclient: _finish_auth 0 2024-09-16T05:10:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:10:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 1 -- 172.21.15.114:0/2357988592 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce90007270 con 0x7fcea009e000 2024-09-16T05:10:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea842b640 5 monclient: authenticate success, global_id 54185 2024-09-16T05:10:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea842b640 10 monclient: _renew_subs 2024-09-16T05:10:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea842b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea842b640 1 -- 172.21.15.114:0/2357988592 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcea012b700 con 0x7fcea009e000 2024-09-16T05:10:25.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:25.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 1 -- 172.21.15.114:0/2357988592 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce90007660 con 0x7fcea009e000 2024-09-16T05:10:25.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:25.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 10 monclient: dump: 2024-09-16T05:10:25.208 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.208 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.208 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.208 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.208 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.208 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.208 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.208 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.208 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.208 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce97fff640 4 set_mon_vals no callback set 2024-09-16T05:10:25.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea842b640 10 client.54185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:10:25.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea842b640 10 monclient: _renew_subs 2024-09-16T05:10:25.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea842b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:25.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea842b640 1 -- 172.21.15.114:0/2357988592 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcea009c6a0 con 0x7fcea009e000 2024-09-16T05:10:25.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea842b640 1 librados: init done 2024-09-16T05:10:25.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 1 -- 172.21.15.114:0/2357988592 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fce900047f0 con 0x7fcea009e000 2024-09-16T05:10:25.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea842b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:25.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 1 --2- 172.21.15.114:0/2357988592 >> v2:172.21.15.149:6812/1185531135 conn(0x7fce84038440 0x7fce8403a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 1 -- 172.21.15.114:0/2357988592 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(84..84 src has 1..84) ==== 7208+0+0 (secure 0 0 0) 0x7fce90048700 con 0x7fcea009e000 2024-09-16T05:10:25.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 10 client.54185.objecter ms_dispatch 0x7fcea0000ba0 osd_map(84..84 src has 1..84) 2024-09-16T05:10:25.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 3 client.54185.objecter handle_osd_map got epochs [84,84] > 0 2024-09-16T05:10:25.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 3 client.54185.objecter handle_osd_map decoding full epoch 84 2024-09-16T05:10:25.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 20 client.54185.objecter dump_active .. 0 homeless 2024-09-16T05:10:25.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea61c7640 1 --2- 172.21.15.114:0/2357988592 >> v2:172.21.15.149:6812/1185531135 conn(0x7fce84038440 0x7fce8403a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:25.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea842b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:25.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea842b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:25.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea842b640 1 -- 172.21.15.114:0/2357988592 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcea009d580 con 0x7fcea009e000 2024-09-16T05:10:25.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea61c7640 10 monclient: get_auth_request con 0x7fce84038440 auth_method 0 2024-09-16T05:10:25.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fce96ffd640 10 client.54185.objecter ms_handle_connect 0x7fce84038440 2024-09-16T05:10:25.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.186+0000 7fcea61c7640 1 --2- 172.21.15.114:0/2357988592 >> v2:172.21.15.149:6812/1185531135 conn(0x7fce84038440 0x7fce8403a900 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 crypto rx=0x7fce9c007fd0 tx=0x7fce9c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:25.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.194+0000 7fce96ffd640 1 -- 172.21.15.114:0/2357988592 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fce9001a310 con 0x7fcea009e000 2024-09-16T05:10:25.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.194+0000 7fce96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:25.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.194+0000 7fce96ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:10:25.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:10:25.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:10:25.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:25.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 1 -- 172.21.15.114:0/2357988592 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcea00686f0 con 0x7fcea009e000 2024-09-16T05:10:25.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fce96ffd640 1 -- 172.21.15.114:0/2357988592 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fce90004ae0 con 0x7fcea009e000 2024-09-16T05:10:25.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fce96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:10:25.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fce96ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:10:25.430 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":16,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:10:25.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 10 librados: watch_flush enter 2024-09-16T05:10:25.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 10 librados: watch_flush exit 2024-09-16T05:10:25.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 20 client.54185.objecter shutdown clearing up homeless session... 2024-09-16T05:10:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 10 client.54185.objecter successfully canceled tick 2024-09-16T05:10:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 1 -- 172.21.15.114:0/2357988592 >> v2:172.21.15.149:6812/1185531135 conn(0x7fce84038440 msgr2=0x7fce8403a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 1 --2- 172.21.15.114:0/2357988592 >> v2:172.21.15.149:6812/1185531135 conn(0x7fce84038440 0x7fce8403a900 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 crypto rx=0x7fce9c007fd0 tx=0x7fce9c008040 comp rx=0 tx=0).stop 2024-09-16T05:10:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 10 monclient: shutdown 2024-09-16T05:10:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 1 -- 172.21.15.114:0/2357988592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea009e000 msgr2=0x7fcea0071560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 1 --2- 172.21.15.114:0/2357988592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea009e000 0x7fcea0071560 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fce900012c0 tx=0x7fce9000cb90 comp rx=0 tx=0).stop 2024-09-16T05:10:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 1 -- 172.21.15.114:0/2357988592 shutdown_connections 2024-09-16T05:10:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 1 --2- 172.21.15.114:0/2357988592 >> v2:172.21.15.149:6812/1185531135 conn(0x7fce84038440 0x7fce8403a900 unknown :-1 s=CLOSED pgs=3394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 1 --2- 172.21.15.114:0/2357988592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcea00a0ec0 0x7fcea0071aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 1 --2- 172.21.15.114:0/2357988592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea009e000 0x7fcea0071560 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 1 --2- 172.21.15.114:0/2357988592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce84007380 0x7fcea0071020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 1 -- 172.21.15.114:0/2357988592 >> 172.21.15.114:0/2357988592 conn(0x7fcea00a1fb0 msgr2=0x7fcea0133c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 1 -- 172.21.15.114:0/2357988592 shutdown_connections 2024-09-16T05:10:25.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 1 -- 172.21.15.114:0/2357988592 wait complete. 2024-09-16T05:10:25.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.426+0000 7fcea842b640 1 librados: shutdown 2024-09-16T05:10:25.449 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:10:25.449 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T05:10:25.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:10:25.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 10 monclient: get_monmap_and_config 2024-09-16T05:10:25.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 10 monclient: build_initial_monmap 2024-09-16T05:10:25.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 10 monclient: monmap: 2024-09-16T05:10:25.565 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:25.565 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:10:25.564396+0000 2024-09-16T05:10:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:10:25.564396+0000 2024-09-16T05:10:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:10:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:10:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:10:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:10:25.566 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:10:25.566 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 1 Processor -- start 2024-09-16T05:10:25.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 1 -- start start 2024-09-16T05:10:25.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 10 monclient: init 2024-09-16T05:10:25.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:25.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T05:10:25.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f902c137480 addr v1:172.21.15.149:6789/0 2024-09-16T05:10:25.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f902c0a57b0 addr v1:172.21.15.114:6789/0 2024-09-16T05:10:25.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f902c13da70 0x7f902c13fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 10 monclient(hunting): picked mon.noname-b con 0x7f902c13da70 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:25.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f902c09cfa0 con 0x7f902c0a57b0 2024-09-16T05:10:25.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f902c0a52f0 con 0x7f902c137480 2024-09-16T05:10:25.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f902c137f70 con 0x7f902c13da70 2024-09-16T05:10:25.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:25.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f90336ed640 10 monclient(hunting): authenticate will time out at 1110.193564s 2024-09-16T05:10:25.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031489640 1 -- >> v1:172.21.15.149:6789/0 conn(0x7f902c137480 legacy=0x7f902c137860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:10:25.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f902c13da70 0x7f902c13fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:25.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9030c88640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f902c0a57b0 0x7f902c13c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38184/0 (socket says 172.21.15.114:38184) 2024-09-16T05:10:25.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f902c13da70 0x7f902c13fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:32794/0 (socket says 172.21.15.114:32794) 2024-09-16T05:10:25.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9030c88640 1 -- 172.21.15.114:0/1168061767 learned_addr learned my addr 172.21.15.114:0/1168061767 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:10:25.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 10 monclient(hunting): get_auth_request con 0x7f902c13da70 auth_method 0 2024-09-16T05:10:25.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:25.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:25.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:25.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:25.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:25.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:25.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 10 monclient(hunting): handle_auth_done global_id 54188 payload 274 2024-09-16T05:10:25.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 1 -- 172.21.15.114:0/1168061767 >> v1:172.21.15.149:6789/0 conn(0x7f902c137480 legacy=0x7f902c137860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:10:25.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 1 -- 172.21.15.114:0/1168061767 >> v1:172.21.15.114:6789/0 conn(0x7f902c0a57b0 legacy=0x7f902c13c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 10 monclient: _finish_hunting 0 2024-09-16T05:10:25.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f902bfff640 1 -- 172.21.15.114:0/1168061767 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 494803270 0 0) 0x7f9020001120 con 0x7f902c0a57b0 2024-09-16T05:10:25.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 1 monclient: found mon.noname-b 2024-09-16T05:10:25.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:25.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:10:25.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f9031c8a640 1 -- 172.21.15.114:0/1168061767 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f902c0a5570 con 0x7f902c13da70 2024-09-16T05:10:25.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.562+0000 7f902bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:10:25.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 1 --2- 172.21.15.114:0/1168061767 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f902c13da70 0x7f902c13fe60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f90240097f0 tx=0x7f902401c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be77fc72f4e0a2ce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:25.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f902bfff640 1 -- 172.21.15.114:0/1168061767 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90240b0070 con 0x7f902c13da70 2024-09-16T05:10:25.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f902bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f902bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:10:25.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f902bfff640 10 monclient: dump: 2024-09-16T05:10:25.578 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.579 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.579 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.579 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.579 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.579 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.579 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.579 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.579 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.579 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f902bfff640 10 monclient: _finish_auth 0 2024-09-16T05:10:25.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f902bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:10:25.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f902bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:25.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 5 monclient: authenticate success, global_id 54188 2024-09-16T05:10:25.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f902bfff640 1 -- 172.21.15.114:0/1168061767 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90240ab600 con 0x7f902c13da70 2024-09-16T05:10:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f902bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient: get_monmap_and_config success 2024-09-16T05:10:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f902bfff640 1 -- 172.21.15.114:0/1168061767 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90240ab9d0 con 0x7f902c13da70 2024-09-16T05:10:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 4 set_mon_vals no callback set 2024-09-16T05:10:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f902bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f902bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f902bfff640 10 monclient: dump: 2024-09-16T05:10:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.581 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient: shutdown 2024-09-16T05:10:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 -- 172.21.15.114:0/1168061767 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f902c13da70 msgr2=0x7f902c13fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 --2- 172.21.15.114:0/1168061767 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f902c13da70 0x7f902c13fe60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f90240097f0 tx=0x7f902401c330 comp rx=0 tx=0).stop 2024-09-16T05:10:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 -- 172.21.15.114:0/1168061767 shutdown_connections 2024-09-16T05:10:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 --2- 172.21.15.114:0/1168061767 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f902c13da70 0x7f902c13fe60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 -- 172.21.15.114:0/1168061767 >> 172.21.15.114:0/1168061767 conn(0x7f902c0a9c40 msgr2=0x7f902c0aa050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 -- 172.21.15.114:0/1168061767 shutdown_connections 2024-09-16T05:10:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 -- 172.21.15.114:0/1168061767 wait complete. 2024-09-16T05:10:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient: build_initial_monmap 2024-09-16T05:10:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient: monmap: 2024-09-16T05:10:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:10:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:10:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:10:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:10:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:10:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:10:25.583 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 librados: starting msgr at 2024-09-16T05:10:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 librados: starting objecter 2024-09-16T05:10:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 Processor -- start 2024-09-16T05:10:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 -- start start 2024-09-16T05:10:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 librados: setting wanted keys 2024-09-16T05:10:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 librados: calling monclient init 2024-09-16T05:10:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient: init 2024-09-16T05:10:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:10:25.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c0a57b0 0x7f902c136020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient(hunting): picked mon.noname-a con 0x7f902c0a57b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:25.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f902c137480 0x7f902c09f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient(hunting): picked mon.noname-b con 0x7f902c137480 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:25.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f902c13da70 0x7f902c09f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient(hunting): picked mon.noname-c con 0x7f902c13da70 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:25.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f902c072a90 con 0x7f902c0a57b0 2024-09-16T05:10:25.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f902c072910 con 0x7f902c137480 2024-09-16T05:10:25.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f902c142bc0 con 0x7f902c13da70 2024-09-16T05:10:25.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:25.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient(hunting): authenticate will time out at 1110.198475s 2024-09-16T05:10:25.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031489640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c0a57b0 0x7f902c136020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:25.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f902c13da70 0x7f902c09f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:25.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031489640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c0a57b0 0x7f902c136020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35968/0 (socket says 172.21.15.114:35968) 2024-09-16T05:10:25.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031489640 1 -- 172.21.15.114:0/2714640302 learned_addr learned my addr 172.21.15.114:0/2714640302 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:25.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f902c13da70 0x7f902c09f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:32804/0 (socket says 172.21.15.114:32804) 2024-09-16T05:10:25.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031489640 10 monclient(hunting): get_auth_request con 0x7f902c0a57b0 auth_method 0 2024-09-16T05:10:25.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031489640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:25.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031489640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:25.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031489640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:25.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 10 monclient(hunting): get_auth_request con 0x7f902c13da70 auth_method 0 2024-09-16T05:10:25.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:25.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:25.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:25.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9030c88640 1 -- 172.21.15.114:0/2714640302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f902c137480 msgr2=0x7f902c09f0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:10:25.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9030c88640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f902c137480 0x7f902c09f0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:10:25.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031489640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:25.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031489640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:25.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031489640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:25.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:25.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:25.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:25.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 10 monclient(hunting): handle_auth_done global_id 54191 payload 824 2024-09-16T05:10:25.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 1 -- 172.21.15.114:0/2714640302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f902c137480 msgr2=0x7f902c09f0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:10:25.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f902c137480 0x7f902c09f0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 1 -- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c0a57b0 msgr2=0x7f902c136020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c0a57b0 0x7f902c136020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 10 monclient: _finish_hunting 0 2024-09-16T05:10:25.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 1 monclient: found mon.noname-c 2024-09-16T05:10:25.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:25.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:10:25.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 1 -- 172.21.15.114:0/2714640302 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f902c0a8d30 con 0x7f902c13da70 2024-09-16T05:10:25.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031489640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c0a57b0 0x7f902c136020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:10:25.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9031c8a640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f902c13da70 0x7f902c09f5e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f90240ab000 tx=0x7f902401ab70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:25.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9029ffb640 10 client.?.objecter ms_handle_connect 0x7f902c13da70 2024-09-16T05:10:25.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9029ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:10:25.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9029ffb640 1 -- 172.21.15.114:0/2714640302 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90240b0050 con 0x7f902c13da70 2024-09-16T05:10:25.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9029ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9029ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:10:25.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9029ffb640 10 monclient: dump: 2024-09-16T05:10:25.588 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.588 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.588 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.588 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.588 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.588 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.588 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.588 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.588 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.588 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9029ffb640 10 monclient: _finish_auth 0 2024-09-16T05:10:25.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9029ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:10:25.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9029ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:25.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9029ffb640 1 -- 172.21.15.114:0/2714640302 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90240b1040 con 0x7f902c13da70 2024-09-16T05:10:25.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 5 monclient: authenticate success, global_id 54191 2024-09-16T05:10:25.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9029ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:25.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9029ffb640 1 -- 172.21.15.114:0/2714640302 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90240b5610 con 0x7f902c13da70 2024-09-16T05:10:25.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient: _renew_subs 2024-09-16T05:10:25.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:25.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 -- 172.21.15.114:0/2714640302 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f902c13d600 con 0x7f902c13da70 2024-09-16T05:10:25.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9029ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9029ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:25.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f902affd640 4 set_mon_vals no callback set 2024-09-16T05:10:25.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f9029ffb640 10 monclient: dump: 2024-09-16T05:10:25.590 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.590 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.590 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.590 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.590 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.590 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.590 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.590 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.590 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.590 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 client.54191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:10:25.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient: _renew_subs 2024-09-16T05:10:25.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:25.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 -- 172.21.15.114:0/2714640302 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f902c0a0bf0 con 0x7f902c13da70 2024-09-16T05:10:25.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.566+0000 7f90336ed640 1 librados: init done 2024-09-16T05:10:25.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f90336ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:25.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f90336ed640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c136560 0x7f902c02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f90336ed640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f90336ed640 1 -- 172.21.15.114:0/2714640302 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f902c069390 con 0x7f902c136560 2024-09-16T05:10:25.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 1 -- 172.21.15.114:0/2714640302 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f90240b57d0 con 0x7f902c13da70 2024-09-16T05:10:25.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9031489640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c136560 0x7f902c02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:25.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9031489640 10 monclient: get_auth_request con 0x7f902c136560 auth_method 0 2024-09-16T05:10:25.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9031489640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:25.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9031489640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:25.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9031489640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:25.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9031489640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:25.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9031489640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9031489640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 1 --2- 172.21.15.114:0/2714640302 >> v2:172.21.15.149:6812/1185531135 conn(0x7f90180377c0 0x7f9018039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9031489640 10 monclient(hunting): handle_auth_done global_id 27633 payload 824 2024-09-16T05:10:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 1 -- 172.21.15.114:0/2714640302 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(84..84 src has 1..84) ==== 7208+0+0 (secure 0 0 0) 0x7f90240e3330 con 0x7f902c13da70 2024-09-16T05:10:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 10 client.54191.objecter ms_dispatch 0x7f902c000ba0 osd_map(84..84 src has 1..84) 2024-09-16T05:10:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 3 client.54191.objecter handle_osd_map got epochs [84,84] > 0 2024-09-16T05:10:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 3 client.54191.objecter handle_osd_map decoding full epoch 84 2024-09-16T05:10:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9031489640 1 -- 172.21.15.114:0/2714640302 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f902c137860 con 0x7f902c136560 2024-09-16T05:10:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 20 client.54191.objecter dump_active .. 0 homeless 2024-09-16T05:10:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 10 client.54191.objecter ms_handle_connect 0x7f902c136560 2024-09-16T05:10:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9031489640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c136560 0x7f902c02c650 secure :-1 s=READY pgs=8205 cs=0 l=1 rev1=1 crypto rx=0x7f901c00e3a0 tx=0x7f901c00e860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:25.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 10 client.54191.objecter resend_mon_ops 2024-09-16T05:10:25.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9030c88640 1 --2- 172.21.15.114:0/2714640302 >> v2:172.21.15.149:6812/1185531135 conn(0x7f90180377c0 0x7f9018039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:25.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 1 -- 172.21.15.114:0/2714640302 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c0054e0 con 0x7f902c136560 2024-09-16T05:10:25.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:25.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 10 monclient: dump: 2024-09-16T05:10:25.593 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.593 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.593 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.593 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.593 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.593 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.594 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9030c88640 10 monclient: get_auth_request con 0x7f90180377c0 auth_method 0 2024-09-16T05:10:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 1 -- 172.21.15.114:0/2714640302 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f901c018070 con 0x7f902c136560 2024-09-16T05:10:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:10:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 1 -- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c136560 msgr2=0x7f902c02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c136560 0x7f902c02c650 secure :-1 s=READY pgs=8205 cs=0 l=1 rev1=1 crypto rx=0x7f901c00e3a0 tx=0x7f901c00e860 comp rx=0 tx=0).stop 2024-09-16T05:10:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 1 -- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c136560 msgr2=0x7f902c02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c136560 0x7f902c02c650 secure :-1 s=CLOSED pgs=8205 cs=0 l=1 rev1=1 crypto rx=0x7f901c00e3a0 tx=0x7f901c00e860 comp rx=0 tx=0).stop 2024-09-16T05:10:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9029ffb640 10 client.54191.objecter ms_handle_connect 0x7f90180377c0 2024-09-16T05:10:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.570+0000 7f9030c88640 1 --2- 172.21.15.114:0/2714640302 >> v2:172.21.15.149:6812/1185531135 conn(0x7f90180377c0 0x7f9018039c80 secure :-1 s=READY pgs=3395 cs=0 l=1 rev1=1 crypto rx=0x7f902c0a18a0 tx=0x7f9020007960 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:25.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.574+0000 7f90336ed640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:10:25.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.574+0000 7f90336ed640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c02cb90 0x7f902c02cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.574+0000 7f90336ed640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.574+0000 7f90336ed640 1 -- 172.21.15.114:0/2714640302 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f902c13b840 con 0x7f902c02cb90 2024-09-16T05:10:25.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.574+0000 7f9031489640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c02cb90 0x7f902c02cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:25.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.574+0000 7f9031489640 10 monclient: get_auth_request con 0x7f902c02cb90 auth_method 0 2024-09-16T05:10:25.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.574+0000 7f9031489640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:25.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.574+0000 7f9031489640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:25.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.574+0000 7f9031489640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:25.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.574+0000 7f9031489640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:25.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.574+0000 7f9031489640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:25.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.574+0000 7f9031489640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.574+0000 7f9031489640 10 monclient(hunting): handle_auth_done global_id 27636 payload 824 2024-09-16T05:10:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.574+0000 7f9031489640 1 -- 172.21.15.114:0/2714640302 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f902c0a2ec0 con 0x7f902c02cb90 2024-09-16T05:10:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9029ffb640 10 client.54191.objecter ms_handle_connect 0x7f902c02cb90 2024-09-16T05:10:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9029ffb640 10 client.54191.objecter resend_mon_ops 2024-09-16T05:10:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9031489640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c02cb90 0x7f902c02cf70 secure :-1 s=READY pgs=8206 cs=0 l=1 rev1=1 crypto rx=0x7f901c00e3a0 tx=0x7f901c005d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9029ffb640 1 -- 172.21.15.114:0/2714640302 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c026410 con 0x7f902c02cb90 2024-09-16T05:10:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9029ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9029ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:10:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9029ffb640 10 monclient: dump: 2024-09-16T05:10:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.599 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9029ffb640 1 -- 172.21.15.114:0/2714640302 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f901c01c410 con 0x7f902c02cb90 2024-09-16T05:10:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9029ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:10:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9029ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:10:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9029ffb640 1 -- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c02cb90 msgr2=0x7f902c02cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9029ffb640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c02cb90 0x7f902c02cf70 secure :-1 s=READY pgs=8206 cs=0 l=1 rev1=1 crypto rx=0x7f901c00e3a0 tx=0x7f901c005d10 comp rx=0 tx=0).stop 2024-09-16T05:10:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9029ffb640 1 -- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c02cb90 msgr2=0x7f902c02cf70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9029ffb640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f902c02cb90 0x7f902c02cf70 unknown :-1 s=CLOSED pgs=8206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.600 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:10:25.600 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:10:25.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:10:25.600 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:10:25.600 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 16, 2024-09-16T05:10:25.600 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 183567, 2024-09-16T05:10:25.600 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:10:25.600 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:10:25.600 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:10:25.601 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:25.601 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 1, 2024-09-16T05:10:25.601 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:25.601 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:10:25.601 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:10:25.601 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:25.601 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:25.601 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:25.601 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:25.601 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:25.601 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:25.602 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:25.602 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:25.602 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:25.602 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:25.602 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:25.602 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:25.602 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:10:25.602 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:10:25.602 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:25.602 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:25.602 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:25.602 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:25.602 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:25.603 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:25.603 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:25.603 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:25.603 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:25.603 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:25.603 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:25.603 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:10:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:10:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:10:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:10:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:10:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:10:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:10:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:10:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:10:25.604 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:10:25.605 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:10:25.605 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:10:25.605 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:10:25.605 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:25.605 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:10:25.605 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:25.605 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:25.605 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:25.605 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:25.605 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:25.605 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:25.605 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:25.606 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:25.606 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:25.606 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:25.606 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:25.606 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:25.606 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:10:25.606 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.606 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:10:25.606 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.606 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:10:25.606 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:10:25.606 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:25.606 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:25.606 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.607 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:25.607 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:10:25.607 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:25.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.607 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.607 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:25.607 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:10:25.607 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:25.607 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:25.607 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:25.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.607 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:10:25.607 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:10:25.608 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:25.608 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:25.608 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:25.608 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.608 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.608 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:10:25.608 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:10:25.608 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:25.608 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:25.608 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.608 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:25.608 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:10:25.608 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:25.609 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.609 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.609 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:25.609 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:10:25.609 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:25.609 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:25.609 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:25.609 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.609 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:10:25.609 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:10:25.609 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:25.609 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:25.609 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:25.610 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.610 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.610 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:10:25.610 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:10:25.610 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:25.610 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:25.610 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.610 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:25.610 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:10:25.610 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:25.610 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.610 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.610 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:25.611 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:10:25.611 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:25.611 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:25.611 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:25.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.611 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:10:25.611 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:10:25.611 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:25.611 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:25.611 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:25.611 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:25.611 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:25.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.611 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:10:25.612 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:10:25.612 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.612 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:25.612 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:25.612 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:10:25.612 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:25.612 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:25.612 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:10:25.612 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.612 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:25.612 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:25.613 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:10:25.613 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:25.613 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:25.613 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:10:25.613 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.613 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:25.613 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:25.613 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 7 2024-09-16T05:10:25.613 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:25.613 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:25.613 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.613 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:10:25.614 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:10:25.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f90336ed640 10 librados: watch_flush enter 2024-09-16T05:10:25.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f90336ed640 10 librados: watch_flush exit 2024-09-16T05:10:25.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f90336ed640 20 client.54191.objecter shutdown clearing up homeless session... 2024-09-16T05:10:25.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f90336ed640 10 client.54191.objecter successfully canceled tick 2024-09-16T05:10:25.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f90336ed640 1 -- 172.21.15.114:0/2714640302 >> v2:172.21.15.149:6812/1185531135 conn(0x7f90180377c0 msgr2=0x7f9018039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f90336ed640 1 --2- 172.21.15.114:0/2714640302 >> v2:172.21.15.149:6812/1185531135 conn(0x7f90180377c0 0x7f9018039c80 secure :-1 s=READY pgs=3395 cs=0 l=1 rev1=1 crypto rx=0x7f902c0a18a0 tx=0x7f9020007960 comp rx=0 tx=0).stop 2024-09-16T05:10:25.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f90336ed640 10 monclient: shutdown 2024-09-16T05:10:25.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f90336ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:25.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f90336ed640 1 -- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f902c13da70 msgr2=0x7f902c09f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f90336ed640 1 --2- 172.21.15.114:0/2714640302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f902c13da70 0x7f902c09f5e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f90240ab000 tx=0x7f902401ab70 comp rx=0 tx=0).stop 2024-09-16T05:10:25.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9031c8a640 1 -- 172.21.15.114:0/2714640302 reap_dead start 2024-09-16T05:10:25.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f9031c8a640 1 -- 172.21.15.114:0/2714640302 reap_dead start 2024-09-16T05:10:25.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f90336ed640 1 -- 172.21.15.114:0/2714640302 shutdown_connections 2024-09-16T05:10:25.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f90336ed640 1 -- 172.21.15.114:0/2714640302 >> 172.21.15.114:0/2714640302 conn(0x7f902c0a9c40 msgr2=0x7f902c13fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:25.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f90336ed640 1 -- 172.21.15.114:0/2714640302 shutdown_connections 2024-09-16T05:10:25.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f90336ed640 1 -- 172.21.15.114:0/2714640302 wait complete. 2024-09-16T05:10:25.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.578+0000 7f90336ed640 1 librados: shutdown 2024-09-16T05:10:25.617 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:10:25.617 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T05:10:25.617 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:10:25.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 10 monclient: get_monmap_and_config 2024-09-16T05:10:25.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 10 monclient: build_initial_monmap 2024-09-16T05:10:25.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 10 monclient: monmap: 2024-09-16T05:10:25.731 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:25.731 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.731 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:10:25.729661+0000 2024-09-16T05:10:25.731 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:10:25.729661+0000 2024-09-16T05:10:25.731 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:25.731 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.731 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:10:25.731 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:10:25.731 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:10:25.731 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:10:25.732 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:10:25.732 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 1 Processor -- start 2024-09-16T05:10:25.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 1 -- start start 2024-09-16T05:10:25.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 10 monclient: init 2024-09-16T05:10:25.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:25.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-16T05:10:25.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84001376e0 addr v1:172.21.15.114:6789/0 2024-09-16T05:10:25.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f84001370a0 0x7f8400137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 10 monclient(hunting): picked mon.noname-c con 0x7f84001370a0 addr v2:172.21.15.149:3300/0 2024-09-16T05:10:25.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84000a5d00 0x7f84000a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 10 monclient(hunting): picked mon.noname-b con 0x7f84000a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:25.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f840009bd20 con 0x7f84001376e0 2024-09-16T05:10:25.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f8400138000 con 0x7f84001370a0 2024-09-16T05:10:25.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f840009c5c0 con 0x7f84000a5d00 2024-09-16T05:10:25.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:25.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8407817640 10 monclient(hunting): authenticate will time out at 1110.358907s 2024-09-16T05:10:25.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8404db2640 1 -- >> v2:172.21.15.149:3300/0 conn(0x7f84001370a0 msgr2=0x7f8400137ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:10:25.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f84055b3640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f84001376e0 0x7f84000a4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38200/0 (socket says 172.21.15.114:38200) 2024-09-16T05:10:25.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8404db2640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f84001370a0 0x7f8400137ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:10:25.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f84055b3640 1 -- 172.21.15.114:0/3597645320 learned_addr learned my addr 172.21.15.114:0/3597645320 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:10:25.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 1 --2- 172.21.15.114:0/3597645320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84000a5d00 0x7f84000a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:25.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 10 monclient(hunting): get_auth_request con 0x7f84000a5d00 auth_method 0 2024-09-16T05:10:25.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:25.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:25.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:25.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:25.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:25.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:25.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f83fffff640 1 -- 172.21.15.114:0/3597645320 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2652965561 0 0) 0x7f840009bd20 con 0x7f84001376e0 2024-09-16T05:10:25.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f83fffff640 10 monclient(hunting): _negotiate 2024-09-16T05:10:25.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f83fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:25.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f83fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:25.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f83fffff640 10 monclient(hunting): my global_id is 27639 2024-09-16T05:10:25.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f83fffff640 1 -- 172.21.15.114:0/3597645320 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83e8003530 con 0x7f84001376e0 2024-09-16T05:10:25.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 10 monclient(hunting): handle_auth_done global_id 54194 payload 274 2024-09-16T05:10:25.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 1 -- 172.21.15.114:0/3597645320 >> v2:172.21.15.149:3300/0 conn(0x7f84001370a0 msgr2=0x7f8400137ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:10:25.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 1 --2- 172.21.15.114:0/3597645320 >> v2:172.21.15.149:3300/0 conn(0x7f84001370a0 0x7f8400137ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 1 -- 172.21.15.114:0/3597645320 >> v1:172.21.15.114:6789/0 conn(0x7f84001376e0 legacy=0x7f84000a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 10 monclient: _finish_hunting 0 2024-09-16T05:10:25.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 1 monclient: found mon.noname-b 2024-09-16T05:10:25.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:25.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:10:25.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.726+0000 7f8405db4640 1 -- 172.21.15.114:0/3597645320 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840009c000 con 0x7f84000a5d00 2024-09-16T05:10:25.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8405db4640 1 --2- 172.21.15.114:0/3597645320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84000a5d00 0x7f84000a4980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f83f80097c0 tx=0x7f83f801c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28b5cc93cc3d4a31 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:25.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f83fffff640 1 -- 172.21.15.114:0/3597645320 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f80b0070 con 0x7f84000a5d00 2024-09-16T05:10:25.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f83fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f83fffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:10:25.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f83fffff640 10 monclient: dump: 2024-09-16T05:10:25.739 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.739 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.739 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.740 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.740 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.740 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.740 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.740 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.740 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.740 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f83fffff640 10 monclient: _finish_auth 0 2024-09-16T05:10:25.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f83fffff640 10 monclient: _check_auth_tickets 2024-09-16T05:10:25.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f83fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:25.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f83fffff640 1 -- 172.21.15.114:0/3597645320 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f80ab600 con 0x7f84000a5d00 2024-09-16T05:10:25.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 5 monclient: authenticate success, global_id 54194 2024-09-16T05:10:25.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:10:25.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f83fffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:25.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f83fffff640 1 -- 172.21.15.114:0/3597645320 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f80ab9d0 con 0x7f84000a5d00 2024-09-16T05:10:25.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f83fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f83fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:25.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f83fffff640 10 monclient: dump: 2024-09-16T05:10:25.741 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.741 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.741 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.741 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.741 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.741 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.742 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.742 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.742 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.742 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 10 monclient: get_monmap_and_config success 2024-09-16T05:10:25.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 4 set_mon_vals no callback set 2024-09-16T05:10:25.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 10 monclient: shutdown 2024-09-16T05:10:25.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:25.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 -- 172.21.15.114:0/3597645320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84000a5d00 msgr2=0x7f84000a4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 --2- 172.21.15.114:0/3597645320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84000a5d00 0x7f84000a4980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f83f80097c0 tx=0x7f83f801c380 comp rx=0 tx=0).stop 2024-09-16T05:10:25.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 -- 172.21.15.114:0/3597645320 shutdown_connections 2024-09-16T05:10:25.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 --2- 172.21.15.114:0/3597645320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84000a5d00 0x7f84000a4980 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 --2- 172.21.15.114:0/3597645320 >> v2:172.21.15.149:3300/0 conn(0x7f84001370a0 0x7f8400137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 -- 172.21.15.114:0/3597645320 >> 172.21.15.114:0/3597645320 conn(0x7f84000a9a40 msgr2=0x7f84000a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:25.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 -- 172.21.15.114:0/3597645320 shutdown_connections 2024-09-16T05:10:25.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 -- 172.21.15.114:0/3597645320 wait complete. 2024-09-16T05:10:25.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 10 monclient: build_initial_monmap 2024-09-16T05:10:25.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 10 monclient: monmap: 2024-09-16T05:10:25.750 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:10:25.750 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:10:25.750 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:10:25.750 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:10:25.750 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:10:25.750 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.751 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:10:25.751 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:10:25.751 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:10:25.751 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 librados: starting msgr at 2024-09-16T05:10:25.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 librados: starting objecter 2024-09-16T05:10:25.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 Processor -- start 2024-09-16T05:10:25.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 -- start start 2024-09-16T05:10:25.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 librados: setting wanted keys 2024-09-16T05:10:25.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 librados: calling monclient init 2024-09-16T05:10:25.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 10 monclient: init 2024-09-16T05:10:25.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 10 monclient: _reopen_session rank -1 2024-09-16T05:10:25.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:10:25.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f84000a5d00 0x7f8400136040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 10 monclient(hunting): picked mon.noname-b con 0x7f84000a5d00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:10:25.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.730+0000 7f8407817640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84001370a0 0x7f840013c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 10 monclient(hunting): picked mon.noname-a con 0x7f84001370a0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:10:25.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84001376e0 0x7f840013c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 10 monclient(hunting): picked mon.noname-c con 0x7f84001376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:10:25.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8400141d80 con 0x7f84001370a0 2024-09-16T05:10:25.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f8400141c00 con 0x7f84000a5d00 2024-09-16T05:10:25.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8400141f00 con 0x7f84001376e0 2024-09-16T05:10:25.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 10 monclient(hunting): _renew_subs 2024-09-16T05:10:25.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 10 monclient(hunting): authenticate will time out at 1110.364061s 2024-09-16T05:10:25.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84001376e0 0x7f840013c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:25.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8404db2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84001370a0 0x7f840013c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:25.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f84055b3640 1 -- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f84000a5d00 msgr2=0x7f8400136040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:10:25.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84001376e0 0x7f840013c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:32822/0 (socket says 172.21.15.114:32822) 2024-09-16T05:10:25.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 1 -- 172.21.15.114:0/4069991436 learned_addr learned my addr 172.21.15.114:0/4069991436 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:10:25.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 10 monclient(hunting): get_auth_request con 0x7f84001376e0 auth_method 0 2024-09-16T05:10:25.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:25.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:25.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:25.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8404db2640 10 monclient(hunting): get_auth_request con 0x7f84001370a0 auth_method 0 2024-09-16T05:10:25.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8404db2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:25.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8404db2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:25.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8404db2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:25.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f84055b3640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f84000a5d00 0x7f8400136040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:10:25.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:25.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:25.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:25.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8404db2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:25.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8404db2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:25.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8404db2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:25.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 10 monclient(hunting): handle_auth_done global_id 54197 payload 824 2024-09-16T05:10:25.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 1 -- 172.21.15.114:0/4069991436 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f84000a5d00 msgr2=0x7f8400136040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:10:25.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f84000a5d00 0x7f8400136040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 1 -- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84001370a0 msgr2=0x7f840013c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84001370a0 0x7f840013c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 10 monclient: _finish_hunting 0 2024-09-16T05:10:25.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 1 monclient: found mon.noname-c 2024-09-16T05:10:25.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:10:25.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:10:25.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 1 -- 172.21.15.114:0/4069991436 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84000a24e0 con 0x7f84001376e0 2024-09-16T05:10:25.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8404db2640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84001370a0 0x7f840013c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:10:25.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8405db4640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84001376e0 0x7f840013c670 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f83f80ae040 tx=0x7f83f801abc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:25.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 client.?.objecter ms_handle_connect 0x7f84001376e0 2024-09-16T05:10:25.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:10:25.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 1 -- 172.21.15.114:0/4069991436 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f80b0050 con 0x7f84001376e0 2024-09-16T05:10:25.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:10:25.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 monclient: dump: 2024-09-16T05:10:25.756 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.756 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.756 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.756 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.756 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.756 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.756 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.757 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.757 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.757 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 monclient: _finish_auth 0 2024-09-16T05:10:25.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 monclient: _check_auth_tickets 2024-09-16T05:10:25.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:10:25.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 1 -- 172.21.15.114:0/4069991436 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f80bc070 con 0x7f84001376e0 2024-09-16T05:10:25.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 5 monclient: authenticate success, global_id 54197 2024-09-16T05:10:25.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 10 monclient: _renew_subs 2024-09-16T05:10:25.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:25.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 1 -- 172.21.15.114:0/4069991436 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84000a8d00 con 0x7f84001376e0 2024-09-16T05:10:25.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:10:25.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 1 -- 172.21.15.114:0/4069991436 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f8006e20 con 0x7f84001376e0 2024-09-16T05:10:25.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:25.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 monclient: dump: 2024-09-16T05:10:25.758 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.758 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.758 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.758 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.758 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.758 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.758 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.758 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.759 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.759 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 10 client.54197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:10:25.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 10 monclient: _renew_subs 2024-09-16T05:10:25.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83feffd640 4 set_mon_vals no callback set 2024-09-16T05:10:25.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:10:25.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 1 -- 172.21.15.114:0/4069991436 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f84001207f0 con 0x7f84001376e0 2024-09-16T05:10:25.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 1 librados: init done 2024-09-16T05:10:25.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:25.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84001209c0 0x7f840013cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8407817640 1 -- 172.21.15.114:0/4069991436 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f840011fec0 con 0x7f84001209c0 2024-09-16T05:10:25.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f84055b3640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84001209c0 0x7f840013cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:25.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 1 -- 172.21.15.114:0/4069991436 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f83f80b5810 con 0x7f84001376e0 2024-09-16T05:10:25.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f84055b3640 10 monclient: get_auth_request con 0x7f84001209c0 auth_method 0 2024-09-16T05:10:25.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f84055b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:25.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f84055b3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:25.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f84055b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:25.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f84055b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:25.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f84055b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:25.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f84055b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:25.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 1 --2- 172.21.15.114:0/4069991436 >> v2:172.21.15.149:6812/1185531135 conn(0x7f83ec0377c0 0x7f83ec039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f84055b3640 10 monclient(hunting): handle_auth_done global_id 54200 payload 824 2024-09-16T05:10:25.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 1 -- 172.21.15.114:0/4069991436 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(84..84 src has 1..84) ==== 7208+0+0 (secure 0 0 0) 0x7f83f80e3bf0 con 0x7f84001376e0 2024-09-16T05:10:25.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f84055b3640 1 -- 172.21.15.114:0/4069991436 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8400137ac0 con 0x7f84001209c0 2024-09-16T05:10:25.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 client.54197.objecter ms_dispatch 0x7f8400000ba0 osd_map(84..84 src has 1..84) 2024-09-16T05:10:25.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 3 client.54197.objecter handle_osd_map got epochs [84,84] > 0 2024-09-16T05:10:25.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 3 client.54197.objecter handle_osd_map decoding full epoch 84 2024-09-16T05:10:25.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 20 client.54197.objecter dump_active .. 0 homeless 2024-09-16T05:10:25.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f84055b3640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84001209c0 0x7f840013cbb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f83f0000c10 tx=0x7f83f000c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:25.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 client.54197.objecter ms_handle_connect 0x7f84001209c0 2024-09-16T05:10:25.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 client.54197.objecter resend_mon_ops 2024-09-16T05:10:25.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 1 -- 172.21.15.114:0/4069991436 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f0017070 con 0x7f84001209c0 2024-09-16T05:10:25.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8404db2640 1 --2- 172.21.15.114:0/4069991436 >> v2:172.21.15.149:6812/1185531135 conn(0x7f83ec0377c0 0x7f83ec039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:25.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:25.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 monclient: dump: 2024-09-16T05:10:25.761 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.762 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.762 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.762 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.762 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.762 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.762 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.762 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.762 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.762 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8404db2640 10 monclient: get_auth_request con 0x7f83ec0377c0 auth_method 0 2024-09-16T05:10:25.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 1 -- 172.21.15.114:0/4069991436 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f83f0005cf0 con 0x7f84001209c0 2024-09-16T05:10:25.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:10:25.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:10:25.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 1 -- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84001209c0 msgr2=0x7f840013cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84001209c0 0x7f840013cbb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f83f0000c10 tx=0x7f83f000c900 comp rx=0 tx=0).stop 2024-09-16T05:10:25.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 1 -- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84001209c0 msgr2=0x7f840013cbb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:25.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84001209c0 0x7f840013cbb0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f83f0000c10 tx=0x7f83f000c900 comp rx=0 tx=0).stop 2024-09-16T05:10:25.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f83fdffb640 10 client.54197.objecter ms_handle_connect 0x7f83ec0377c0 2024-09-16T05:10:25.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.734+0000 7f8404db2640 1 --2- 172.21.15.114:0/4069991436 >> v2:172.21.15.149:6812/1185531135 conn(0x7f83ec0377c0 0x7f83ec039c80 secure :-1 s=READY pgs=3396 cs=0 l=1 rev1=1 crypto rx=0x7f83f4009420 tx=0x7f83f400d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:25.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:10:25.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8400136580 0x7f840013d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:10:25.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 10 monclient(hunting): start opening mon connection 2024-09-16T05:10:25.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 1 -- 172.21.15.114:0/4069991436 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8400073d00 con 0x7f8400136580 2024-09-16T05:10:25.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f84055b3640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8400136580 0x7f840013d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:10:25.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f84055b3640 10 monclient: get_auth_request con 0x7f8400136580 auth_method 0 2024-09-16T05:10:25.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f84055b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:10:25.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f84055b3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:10:25.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f84055b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:10:25.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f84055b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:10:25.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f84055b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:10:25.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f84055b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:10:25.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f84055b3640 10 monclient(hunting): handle_auth_done global_id 54203 payload 824 2024-09-16T05:10:25.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f84055b3640 1 -- 172.21.15.114:0/4069991436 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f840013e520 con 0x7f8400136580 2024-09-16T05:10:25.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f83fdffb640 10 client.54197.objecter ms_handle_connect 0x7f8400136580 2024-09-16T05:10:25.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f84055b3640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8400136580 0x7f840013d0f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f83f00011f0 tx=0x7f83f0000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:10:25.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f83fdffb640 10 client.54197.objecter resend_mon_ops 2024-09-16T05:10:25.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f83fdffb640 1 -- 172.21.15.114:0/4069991436 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f0017040 con 0x7f8400136580 2024-09-16T05:10:25.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f83fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:10:25.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f83fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:10:25.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f83fdffb640 10 monclient: dump: 2024-09-16T05:10:25.765 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:10:25.765 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:10:25.765 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.765 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:10:25.765 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:10:25.765 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:10:25.765 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:10:25.765 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:10:25.765 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:10:25.765 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:10:25.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f83fdffb640 1 -- 172.21.15.114:0/4069991436 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f83f00213b0 con 0x7f8400136580 2024-09-16T05:10:25.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f83fdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:10:25.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f83fdffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:10:25.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f83fdffb640 1 -- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8400136580 msgr2=0x7f840013d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f83fdffb640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8400136580 0x7f840013d0f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f83f00011f0 tx=0x7f83f0000ee0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f83fdffb640 1 -- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8400136580 msgr2=0x7f840013d0f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:10:25.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f83fdffb640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8400136580 0x7f840013d0f0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f83f00011f0 tx=0x7f83f0000ee0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.766 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:10:25.766 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:10:25.766 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:10:25.766 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:10:25.766 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 16, 2024-09-16T05:10:25.767 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 39878, 2024-09-16T05:10:25.767 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:10:25.767 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:10:25.767 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:10:25.767 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:25.767 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 1, 2024-09-16T05:10:25.767 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:25.767 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:10:25.767 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:10:25.767 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:25.767 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:25.767 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:25.768 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:25.768 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:25.768 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:25.768 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:25.768 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:25.768 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:25.768 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:25.768 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:25.768 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:25.768 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:10:25.769 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:10:25.769 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:25.769 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:25.769 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:25.769 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:25.769 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:25.769 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:25.769 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:25.769 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:25.769 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:25.769 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:25.769 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:25.769 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:25.769 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.770 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:10:25.770 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:10:25.770 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:10:25.770 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:10:25.770 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:10:25.770 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:10:25.770 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:25.770 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:10:25.770 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:10:25.770 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:10:25.770 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:10:25.770 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:10:25.770 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:10:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:10:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:10:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:10:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:10:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:10:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:10:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:10:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:10:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:10:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:10:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:10:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:10:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:10:25.771 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:10:25.772 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:10:25.772 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:25.772 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:10:25.772 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.772 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:10:25.772 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.772 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:10:25.772 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:10:25.772 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:25.772 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:25.772 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.772 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:25.773 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:10:25.773 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:25.773 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.773 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.773 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:25.773 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:10:25.773 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:25.773 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:25.773 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:25.773 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.773 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:10:25.773 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:10:25.773 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:25.773 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:25.774 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:25.774 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.774 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.774 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:10:25.774 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:10:25.774 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:25.774 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:25.774 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.774 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:25.774 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:10:25.774 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:25.774 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.774 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.774 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:25.775 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:10:25.775 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:25.775 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:25.775 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:25.775 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.775 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:10:25.775 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:10:25.775 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:25.775 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:25.775 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:25.775 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.775 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.775 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:10:25.775 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:10:25.776 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:10:25.776 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:10:25.776 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.776 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:10:25.776 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:10:25.776 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:25.776 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.776 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.776 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:10:25.776 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:10:25.776 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:10:25.776 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:25.777 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:25.777 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.777 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:10:25.777 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:10:25.777 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:10:25.777 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:10:25.777 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:10:25.777 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:25.777 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:25.777 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.777 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:10:25.777 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:10:25.777 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.778 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:25.778 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:25.778 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:10:25.778 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:25.778 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:25.778 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:10:25.778 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.778 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:25.778 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:25.778 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:10:25.778 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:25.778 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:25.778 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:10:25.778 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.779 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:25.779 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:25.779 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 6 2024-09-16T05:10:25.779 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:25.779 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:10:25.779 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:10:25.779 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:10:25.779 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:10:25.779 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:10:25.779 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:10:25.779 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:10:25.779 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:10:25.779 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:10:25.779 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:10:25.780 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:10:25.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 10 librados: watch_flush enter 2024-09-16T05:10:25.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 10 librados: watch_flush exit 2024-09-16T05:10:25.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 20 client.54197.objecter shutdown clearing up homeless session... 2024-09-16T05:10:25.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 10 client.54197.objecter successfully canceled tick 2024-09-16T05:10:25.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 1 -- 172.21.15.114:0/4069991436 >> v2:172.21.15.149:6812/1185531135 conn(0x7f83ec0377c0 msgr2=0x7f83ec039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 1 --2- 172.21.15.114:0/4069991436 >> v2:172.21.15.149:6812/1185531135 conn(0x7f83ec0377c0 0x7f83ec039c80 secure :-1 s=READY pgs=3396 cs=0 l=1 rev1=1 crypto rx=0x7f83f4009420 tx=0x7f83f400d040 comp rx=0 tx=0).stop 2024-09-16T05:10:25.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 10 monclient: shutdown 2024-09-16T05:10:25.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:10:25.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 1 -- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84001376e0 msgr2=0x7f840013c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:10:25.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 1 --2- 172.21.15.114:0/4069991436 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84001376e0 0x7f840013c670 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f83f80ae040 tx=0x7f83f801abc0 comp rx=0 tx=0).stop 2024-09-16T05:10:25.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8405db4640 1 -- 172.21.15.114:0/4069991436 reap_dead start 2024-09-16T05:10:25.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8405db4640 1 -- 172.21.15.114:0/4069991436 reap_dead start 2024-09-16T05:10:25.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 1 -- 172.21.15.114:0/4069991436 shutdown_connections 2024-09-16T05:10:25.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 1 -- 172.21.15.114:0/4069991436 >> 172.21.15.114:0/4069991436 conn(0x7f84000a9a40 msgr2=0x7f84000a84a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:10:25.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 1 -- 172.21.15.114:0/4069991436 shutdown_connections 2024-09-16T05:10:25.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 1 -- 172.21.15.114:0/4069991436 wait complete. 2024-09-16T05:10:25.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:10:25.742+0000 7f8407817640 1 librados: shutdown 2024-09-16T05:10:25.783 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:10:25.783 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:10:27.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-16T05:10:27.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-16T05:10:27.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-16T05:10:27.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-16T05:10:27.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-16T05:10:27.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-16T05:10:27.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: 2024-09-16T05:10:27.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-16T05:10:27.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: seed 29587 2024-09-16T05:10:27.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-16T05:10:27.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-09-16T05:10:27.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-16T05:10:27.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (341 ms) 2024-09-16T05:10:27.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-16T05:10:27.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (15 ms) 2024-09-16T05:10:27.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-16T05:10:27.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (17 ms) 2024-09-16T05:10:27.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-16T05:10:27.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (24 ms) 2024-09-16T05:10:27.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-16T05:10:27.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (4 ms) 2024-09-16T05:10:27.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-16T05:10:27.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (5 ms) 2024-09-16T05:10:27.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-16T05:10:27.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (9 ms) 2024-09-16T05:10:27.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-16T05:10:27.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (5 ms) 2024-09-16T05:10:27.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-16T05:10:27.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (21 ms) 2024-09-16T05:10:27.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-16T05:10:27.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2 ms) 2024-09-16T05:10:27.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-16T05:10:27.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (20 ms) 2024-09-16T05:10:27.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-16T05:10:27.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (25 ms) 2024-09-16T05:10:27.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-16T05:10:27.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-16T05:10:27.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-16T05:10:27.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (3995 ms) 2024-09-16T05:10:27.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-16T05:10:27.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (393 ms) 2024-09-16T05:10:27.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-16T05:10:27.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-16T05:10:27.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: done waiting, doing copies 2024-09-16T05:10:27.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-16T05:10:27.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: done waiting 2024-09-16T05:10:27.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (61068 ms) 2024-09-16T05:10:27.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-16T05:10:27.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (13 ms) 2024-09-16T05:10:27.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-16T05:10:27.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (8 ms) 2024-09-16T05:10:27.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-16T05:10:27.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (28790 ms) 2024-09-16T05:10:27.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-16T05:10:27.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-16T05:10:27.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-16T05:10:27.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-09-16T05:10:27.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-16T05:10:27.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-09-16T05:10:34.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] 22 tests from LibRadosMi api_snapshots_pp: Running main() from gmock_main.cc 2024-09-16T05:10:34.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-09-16T05:10:34.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-16T05:10:34.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-16T05:10:34.103 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: seed 29688 2024-09-16T05:10:34.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-16T05:10:34.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (1978 ms) 2024-09-16T05:10:34.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-16T05:10:34.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2021 ms) 2024-09-16T05:10:34.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-16T05:10:34.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2107 ms) 2024-09-16T05:10:34.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-16T05:10:34.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2025 ms) 2024-09-16T05:10:34.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-16T05:10:34.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3477 ms) 2024-09-16T05:10:34.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11608 ms total) 2024-09-16T05:10:34.104 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: 2024-09-16T05:10:34.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-09-16T05:10:34.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-16T05:10:34.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4124 ms) 2024-09-16T05:10:34.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-16T05:10:34.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3893 ms) 2024-09-16T05:10:34.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-16T05:10:34.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5979 ms) 2024-09-16T05:10:34.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-16T05:10:34.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4016 ms) 2024-09-16T05:10:34.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-16T05:10:34.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2106 ms) 2024-09-16T05:10:34.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-16T05:10:34.105 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: ./src/test/librados/snapshots_cxx.cc:460: Skipped 2024-09-16T05:10:34.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: 2024-09-16T05:10:34.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-09-16T05:10:34.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-16T05:10:34.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi149-29688-7 2024-09-16T05:10:34.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-16T05:10:34.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17874 ms) 2024-09-16T05:10:34.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (37992 ms total) 2024-09-16T05:10:34.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: 2024-09-16T05:10:34.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-16T05:10:34.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-16T05:10:34.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (36 ms) 2024-09-16T05:10:34.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-16T05:10:34.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (39993 ms) 2024-09-16T05:10:34.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (40029 ms total) 2024-09-16T05:10:34.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: 2024-09-16T05:10:34.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-16T05:10:34.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-16T05:10:34.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2996 ms) 2024-09-16T05:10:34.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-16T05:10:34.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (1968 ms) 2024-09-16T05:10:34.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-16T05:10:34.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2010 ms) 2024-09-16T05:10:41.178 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshot api_aio: Running main() from gmock_main.cc 2024-09-16T05:10:41.178 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-09-16T05:10:41.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [----------] Global test environment set-up. 2024-09-16T05:10:41.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-09-16T05:10:41.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-16T05:10:41.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.TooBig (3226 ms) 2024-09-16T05:10:41.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-16T05:10:41.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (2954 ms) 2024-09-16T05:10:41.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-16T05:10:41.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3056 ms) 2024-09-16T05:10:41.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-16T05:10:41.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3050 ms) 2024-09-16T05:10:41.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-16T05:10:41.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2508 ms) 2024-09-16T05:10:41.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-16T05:10:41.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3042 ms) 2024-09-16T05:10:41.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-16T05:10:41.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3039 ms) 2024-09-16T05:10:41.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-16T05:10:41.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (2922 ms) 2024-09-16T05:10:41.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-16T05:10:41.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3018 ms) 2024-09-16T05:10:41.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-16T05:10:41.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3021 ms) 2024-09-16T05:10:41.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-16T05:10:41.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2968 ms) 2024-09-16T05:10:41.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-16T05:10:41.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3056 ms) 2024-09-16T05:10:41.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-16T05:10:41.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2933 ms) 2024-09-16T05:10:41.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-16T05:10:41.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3020 ms) 2024-09-16T05:10:41.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-16T05:10:41.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.Flush (3026 ms) 2024-09-16T05:10:41.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-16T05:10:41.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2948 ms) 2024-09-16T05:10:41.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-16T05:10:41.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3064 ms) 2024-09-16T05:10:41.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-16T05:10:41.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3008 ms) 2024-09-16T05:10:41.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-16T05:10:41.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (17733 ms) 2024-09-16T05:10:41.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-16T05:10:41.182 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (2217 ms) 2024-09-16T05:10:41.182 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-09-16T05:10:41.182 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (23046 ms) 2024-09-16T05:10:41.182 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-16T05:10:41.182 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3007 ms) 2024-09-16T05:10:41.182 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-16T05:10:41.182 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2947 ms) 2024-09-16T05:10:41.182 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-16T05:10:41.182 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3011 ms) 2024-09-16T05:10:41.182 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-16T05:10:41.182 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3000 ms) 2024-09-16T05:10:41.182 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-16T05:10:41.182 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3065 ms) 2024-09-16T05:10:41.182 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [----------] 26 tests from LibRadosAio (111885 ms total) 2024-09-16T05:10:41.183 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: 2024-09-16T05:10:41.183 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-16T05:10:53.137 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWritscPP (94756 ms total) 2024-09-16T05:10:53.137 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: 2024-09-16T05:10:53.137 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-16T05:10:53.137 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-16T05:10:53.138 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (378 ms) 2024-09-16T05:10:53.138 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (378 ms total) 2024-09-16T05:10:53.138 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: 2024-09-16T05:10:53.138 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-16T05:10:53.138 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-16T05:10:53.138 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (54 ms) 2024-09-16T05:10:53.138 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-16T05:10:53.138 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (5 ms) 2024-09-16T05:10:53.138 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (59 ms total) 2024-09-16T05:10:53.138 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: 2024-09-16T05:10:53.138 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-16T05:10:53.138 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-16T05:10:53.138 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (11 ms) 2024-09-16T05:10:53.139 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-16T05:10:53.139 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (7 ms) 2024-09-16T05:10:53.139 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (18 ms total) 2024-09-16T05:10:53.139 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: 2024-09-16T05:10:53.139 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-16T05:10:53.139 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-16T05:10:53.139 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (56 ms) 2024-09-16T05:10:53.139 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-16T05:10:53.139 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (7 ms) 2024-09-16T05:10:53.139 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (63 ms total) 2024-09-16T05:10:53.139 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: 2024-09-16T05:10:53.139 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-16T05:10:53.139 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-16T05:10:53.139 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1023 ms) 2024-09-16T05:10:53.140 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1023 ms total) 2024-09-16T05:10:53.140 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: 2024-09-16T05:10:53.140 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-16T05:10:53.140 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (123757 ms total) 2024-09-16T05:10:53.140 INFO:tasks.workunit.client.0.smithi149.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-16T05:10:57.168 INFO:tasks.workunit.client.0.smithi149.stdout:s_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-16T05:10:57.168 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2022 ms) 2024-09-16T05:10:57.168 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8996 ms total) 2024-09-16T05:10:57.169 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: 2024-09-16T05:10:57.169 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-16T05:10:57.169 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-16T05:10:57.169 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4038 ms) 2024-09-16T05:10:57.169 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-16T05:10:57.169 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3994 ms) 2024-09-16T05:10:57.169 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-16T05:10:57.169 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3938 ms) 2024-09-16T05:10:57.169 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11971 ms total) 2024-09-16T05:10:57.169 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: 2024-09-16T05:10:57.169 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-16T05:10:57.169 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (127761 ms total) 2024-09-16T05:10:57.169 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-16T05:10:57.169 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-09-16T05:10:57.170 INFO:tasks.workunit.client.0.smithi149.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-16T05:11:02.123 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-09-16T05:11:02.123 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-16T05:11:02.123 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-16T05:11:02.124 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-16T05:11:02.124 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-16T05:11:02.124 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (3222 ms) 2024-09-16T05:11:02.124 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-16T05:11:02.124 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (16643 ms) 2024-09-16T05:11:02.124 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-16T05:11:02.124 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (5944 ms) 2024-09-16T05:11:02.124 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-16T05:11:02.124 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2988 ms) 2024-09-16T05:11:02.124 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-16T05:11:02.124 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3071 ms) 2024-09-16T05:11:02.124 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-16T05:11:02.124 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2971 ms) 2024-09-16T05:11:02.125 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-16T05:11:02.125 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2941 ms) 2024-09-16T05:11:02.125 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-16T05:11:02.125 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3027 ms) 2024-09-16T05:11:02.125 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-16T05:11:02.125 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2981 ms) 2024-09-16T05:11:02.125 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-16T05:11:02.125 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3030 ms) 2024-09-16T05:11:02.125 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-16T05:11:02.125 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2993 ms) 2024-09-16T05:11:02.125 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-16T05:11:02.125 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3043 ms) 2024-09-16T05:11:02.125 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-16T05:11:02.125 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3013 ms) 2024-09-16T05:11:02.126 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-16T05:11:02.126 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (16944 ms) 2024-09-16T05:11:02.126 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-16T05:11:02.126 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (24041 ms) 2024-09-16T05:11:02.126 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-16T05:11:02.126 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3007 ms) 2024-09-16T05:11:02.126 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-16T05:11:02.126 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2947 ms) 2024-09-16T05:11:02.126 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-16T05:11:02.126 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3022 ms) 2024-09-16T05:11:02.126 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-16T05:11:02.126 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2989 ms) 2024-09-16T05:11:02.126 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-16T05:11:02.126 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3068 ms) 2024-09-16T05:11:02.127 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-16T05:11:02.127 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2937 ms) 2024-09-16T05:11:02.127 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-16T05:11:02.127 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2994 ms) 2024-09-16T05:11:02.127 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-16T05:11:02.127 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3074 ms) 2024-09-16T05:11:02.127 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-16T05:11:02.127 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2934 ms) 2024-09-16T05:11:02.127 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-16T05:11:02.127 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3010 ms) 2024-09-16T05:11:02.127 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-16T05:11:02.127 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3043 ms) 2024-09-16T05:11:02.127 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-16T05:11:49.406 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAsor()=9:89d3ae78:::11:head expected=9:89d3ae78:::11:head 2024-09-16T05:11:49.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:89d3ae78:::11:head -> 11 2024-09-16T05:11:49.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=11 expected=11 2024-09-16T05:11:49.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:863748b0:::15:head 2024-09-16T05:11:49.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cursor()=9:863748b0:::15:head expected=9:863748b0:::15:head 2024-09-16T05:11:49.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:863748b0:::15:head -> 15 2024-09-16T05:11:49.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=15 expected=15 2024-09-16T05:11:49.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:6cac518f:::0:head 2024-09-16T05:11:49.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cursor()=9:6cac518f:::0:head expected=9:6cac518f:::0:head 2024-09-16T05:11:49.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:6cac518f:::0:head -> 0 2024-09-16T05:11:49.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=0 expected=0 2024-09-16T05:11:49.407 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:62a1935d:::14:head 2024-09-16T05:11:49.408 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cursor()=9:62a1935d:::14:head expected=9:62a1935d:::14:head 2024-09-16T05:11:49.408 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:62a1935d:::14:head -> 14 2024-09-16T05:11:49.408 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=14 expected=14 2024-09-16T05:11:49.408 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:5c6b0b28:::7:head 2024-09-16T05:11:49.408 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cursor()=9:5c6b0b28:::7:head expected=9:5c6b0b28:::7:head 2024-09-16T05:11:49.408 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:5c6b0b28:::7:head -> 7 2024-09-16T05:11:49.408 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=7 expected=7 2024-09-16T05:11:49.408 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:566253c9:::13:head 2024-09-16T05:11:49.408 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cursor()=9:566253c9:::13:head expected=9:566253c9:::13:head 2024-09-16T05:11:49.408 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:566253c9:::13:head -> 13 2024-09-16T05:11:49.408 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=13 expected=13 2024-09-16T05:11:49.408 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:52ea6a34:::10:head 2024-09-16T05:11:49.408 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cursor()=9:52ea6a34:::10:head expected=9:52ea6a34:::10:head 2024-09-16T05:11:49.409 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:52ea6a34:::10:head -> 10 2024-09-16T05:11:49.409 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=10 expected=10 2024-09-16T05:11:49.409 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : seek to 9:02547ec2:::1:head 2024-09-16T05:11:49.409 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : cursor()=9:02547ec2:::1:head expected=9:02547ec2:::1:head 2024-09-16T05:11:49.409 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: > 9:02547ec2:::1:head -> 1 2024-09-16T05:11:49.409 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: : entry=1 expected=1 2024-09-16T05:11:49.409 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (295 ms) 2024-09-16T05:11:49.409 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-16T05:11:49.409 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (125937 ms) 2024-09-16T05:11:49.409 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-16T05:11:49.409 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: split 0/5 -> MIN 9:33333333::::head 2024-09-16T05:11:49.409 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: split 1/5 -> 9:33333333::::head 9:66666666::::head 2024-09-16T05:11:49.409 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: split 2/5 -> 9:66666666::::head 9:99999999::::head 2024-09-16T05:11:49.410 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: split 3/5 -> 9:99999999::::head 9:cccccccc::::head 2024-09-16T05:11:49.410 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: split 4/5 -> 9:cccccccc::::head MAX 2024-09-16T05:11:49.410 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (46014 ms) 2024-09-16T05:11:49.410 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [----------] 7 tests from LibRadosList (172884 ms total) 2024-09-16T05:11:49.410 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 2024-09-16T05:11:49.410 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-16T05:11:49.410 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-16T05:11:49.410 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1077 ms) 2024-09-16T05:11:49.410 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-16T05:11:49.410 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: myset foo1,foo2,foo3 2024-09-16T05:11:49.411 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo1 2024-09-16T05:11:49.411 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo2 2024-09-16T05:11:49.411 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo3 2024-09-16T05:11:49.411 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: myset foo1,foo4,foo5 2024-09-16T05:11:49.411 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo4 2024-09-16T05:11:49.411 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo5 2024-09-16T05:11:49.411 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo1 2024-09-16T05:11:49.411 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: myset foo6,foo7 2024-09-16T05:11:49.411 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo7 2024-09-16T05:11:49.411 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: foo6 2024-09-16T05:11:49.412 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-16T05:11:49.412 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: ns1:foo4 2024-09-16T05:11:49.412 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: ns1:foo5 2024-09-16T05:11:49.412 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: ns2:foo7 2024-09-16T05:11:49.412 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: ns2:foo6 2024-09-16T05:11:49.412 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: ns1:foo1 2024-09-16T05:11:49.412 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: :foo1 2024-09-16T05:11:49.412 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: :foo2 2024-09-16T05:11:49.412 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: :foo3 2024-09-16T05:11:49.412 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (87 ms) 2024-09-16T05:11:49.413 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-09-16T05:11:49.413 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 1 0 2024-09-16T05:11:49.413 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 10 0 2024-09-16T05:11:49.413 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 13 0 2024-09-16T05:11:55.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 7 0 2024-09-16T05:11:55.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 14 0 2024-09-16T05:11:55.179 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 0 0 2024-09-16T05:11:55.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 15 0 2024-09-16T05:11:55.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 11 0 2024-09-16T05:11:55.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 5 0 2024-09-16T05:11:55.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 8 0 2024-09-16T05:11:55.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 6 0 2024-09-16T05:11:55.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 3 0 2024-09-16T05:11:55.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 4 0 2024-09-16T05:11:55.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 12 0 2024-09-16T05:11:55.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 9 0 2024-09-16T05:11:55.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 2 0 2024-09-16T05:11:55.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-16T05:11:55.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (148 ms) 2024-09-16T05:11:55.180 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [----------] 3 tests from LibRadosListEC (1312 ms total) 2024-09-16T05:11:55.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 2024-09-16T05:11:55.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-16T05:11:55.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-16T05:11:55.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2945 ms) 2024-09-16T05:11:55.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [----------] 1 test from LibRadosListNP (2945 ms total) 2024-09-16T05:11:55.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: 2024-09-16T05:11:55.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [----------] Global test environment tear-down 2024-09-16T05:11:55.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [==========] 11 tests from 3 test suites ran. (185839 ms total) 2024-09-16T05:11:55.181 INFO:tasks.workunit.client.0.smithi149.stdout: api_list: [ PASSED ] 11 tests. 2024-09-16T05:11:55.784 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:11:55.784 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T05:11:55.784 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T05:11:55.785 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T05:11:55.788 INFO:tasks.ceph.mon.b:Started 2024-09-16T05:11:55.788 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T05:11:55.788 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T05:11:55.789 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T05:12:10.789 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T05:12:10.790 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T05:12:10.790 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:12:10.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:12:10.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.921+0000 7feb6c073640 10 monclient: get_monmap_and_config 2024-09-16T05:12:10.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.921+0000 7feb6c073640 10 monclient: build_initial_monmap 2024-09-16T05:12:10.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.921+0000 7feb6c073640 10 monclient: monmap: 2024-09-16T05:12:10.927 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:10.927 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:10.927 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:12:10.926975+0000 2024-09-16T05:12:10.927 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:12:10.926975+0000 2024-09-16T05:12:10.927 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:10.927 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:10.927 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:12:10.927 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:12:10.927 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:12:10.927 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:12:10.928 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:12:10.928 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:10.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.921+0000 7feb6c073640 1 Processor -- start 2024-09-16T05:12:10.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.921+0000 7feb6c073640 1 -- start start 2024-09-16T05:12:10.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.921+0000 7feb6c073640 10 monclient: init 2024-09-16T05:12:10.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:10.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-16T05:12:10.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb641376e0 addr v1:172.21.15.149:6789/0 2024-09-16T05:12:10.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb641370a0 0x7feb64137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:10.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 10 monclient(hunting): picked mon.noname-b con 0x7feb641370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:10.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7feb640a5d00 0x7feb640a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:10.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 10 monclient(hunting): picked mon.noname-c con 0x7feb640a5d00 addr v2:172.21.15.149:3300/0 2024-09-16T05:12:10.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb6409bd20 con 0x7feb641376e0 2024-09-16T05:12:10.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:10.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7feb64138000 con 0x7feb640a5d00 2024-09-16T05:12:10.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:10.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7feb6409c5c0 con 0x7feb641370a0 2024-09-16T05:12:10.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:10.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 10 monclient(hunting): authenticate will time out at 1215.556156s 2024-09-16T05:12:10.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb641370a0 0x7feb64137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:10.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb641370a0 0x7feb64137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:59114/0 (socket says 172.21.15.114:59114) 2024-09-16T05:12:10.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 1 -- 172.21.15.114:0/459951880 learned_addr learned my addr 172.21.15.114:0/459951880 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:10.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 10 monclient(hunting): get_auth_request con 0x7feb641370a0 auth_method 0 2024-09-16T05:12:10.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:10.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:10.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:10.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:10.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:10.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:10.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 10 monclient(hunting): handle_auth_done global_id 54443 payload 274 2024-09-16T05:12:10.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6a610640 1 --2- 172.21.15.114:0/459951880 >> v2:172.21.15.149:3300/0 conn(0x7feb640a5d00 0x7feb640a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:10.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 1 -- 172.21.15.114:0/459951880 >> v2:172.21.15.149:3300/0 conn(0x7feb640a5d00 msgr2=0x7feb640a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:10.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 1 --2- 172.21.15.114:0/459951880 >> v2:172.21.15.149:3300/0 conn(0x7feb640a5d00 0x7feb640a4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:10.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 1 -- 172.21.15.114:0/459951880 >> v1:172.21.15.149:6789/0 conn(0x7feb641376e0 legacy=0x7feb640a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:10.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 10 monclient: _finish_hunting 0 2024-09-16T05:12:10.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 1 monclient: found mon.noname-b 2024-09-16T05:12:10.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:10.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:12:10.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 1 -- 172.21.15.114:0/459951880 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6409c000 con 0x7feb641370a0 2024-09-16T05:12:10.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6960e640 1 --2- 172.21.15.114:0/459951880 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb641370a0 0x7feb64137ac0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7feb54009870 tx=0x7feb5401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=571132f9f8100a2c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:10.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb68e0d640 1 -- 172.21.15.114:0/459951880 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb540b0070 con 0x7feb641370a0 2024-09-16T05:12:10.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb68e0d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:10.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb68e0d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:12:10.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb68e0d640 10 monclient: dump: 2024-09-16T05:12:10.934 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:10.934 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:10.934 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:10.934 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:10.934 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:10.934 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:10.934 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:10.935 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:10.935 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:10.935 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:10.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb68e0d640 10 monclient: _finish_auth 0 2024-09-16T05:12:10.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb68e0d640 10 monclient: _check_auth_tickets 2024-09-16T05:12:10.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb68e0d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:10.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb68e0d640 1 -- 172.21.15.114:0/459951880 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb540ab600 con 0x7feb641370a0 2024-09-16T05:12:10.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb68e0d640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:10.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb68e0d640 1 -- 172.21.15.114:0/459951880 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb540ab9d0 con 0x7feb641370a0 2024-09-16T05:12:10.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 5 monclient: authenticate success, global_id 54443 2024-09-16T05:12:10.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 10 monclient: get_monmap_and_config success 2024-09-16T05:12:10.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 4 set_mon_vals no callback set 2024-09-16T05:12:10.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb68e0d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:10.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb68e0d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:10.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb68e0d640 10 monclient: dump: 2024-09-16T05:12:10.936 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:10.936 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:10.936 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:10.936 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:10.936 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:10.936 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:10.936 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:10.936 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:10.936 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:10.936 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:10.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 10 monclient: shutdown 2024-09-16T05:12:10.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:10.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 1 -- 172.21.15.114:0/459951880 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb641370a0 msgr2=0x7feb64137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:10.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 1 --2- 172.21.15.114:0/459951880 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb641370a0 0x7feb64137ac0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7feb54009870 tx=0x7feb5401c3e0 comp rx=0 tx=0).stop 2024-09-16T05:12:10.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 1 -- 172.21.15.114:0/459951880 shutdown_connections 2024-09-16T05:12:10.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 1 --2- 172.21.15.114:0/459951880 >> v2:172.21.15.149:3300/0 conn(0x7feb640a5d00 0x7feb640a4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:10.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 1 --2- 172.21.15.114:0/459951880 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb641370a0 0x7feb64137ac0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:10.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 1 -- 172.21.15.114:0/459951880 >> 172.21.15.114:0/459951880 conn(0x7feb640a9a40 msgr2=0x7feb640a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:10.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 1 -- 172.21.15.114:0/459951880 shutdown_connections 2024-09-16T05:12:10.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.925+0000 7feb6c073640 1 -- 172.21.15.114:0/459951880 wait complete. 2024-09-16T05:12:10.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient: build_initial_monmap 2024-09-16T05:12:10.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient: monmap: 2024-09-16T05:12:10.942 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:10.942 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:12:10.942 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:12:10.942 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:12:10.942 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:10.942 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:10.943 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:12:10.943 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:12:10.943 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:12:10.943 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:10.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 librados: starting msgr at 2024-09-16T05:12:10.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 librados: starting objecter 2024-09-16T05:12:10.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 Processor -- start 2024-09-16T05:12:10.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 -- start start 2024-09-16T05:12:10.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 librados: setting wanted keys 2024-09-16T05:12:10.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 librados: calling monclient init 2024-09-16T05:12:10.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient: init 2024-09-16T05:12:10.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:10.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:12:10.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb640a5d00 0x7feb641cf090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:10.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient(hunting): picked mon.noname-a con 0x7feb640a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:10.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feb641370a0 0x7feb64136190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:10.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient(hunting): picked mon.noname-b con 0x7feb641370a0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:10.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb641376e0 0x7feb6413c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:10.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient(hunting): picked mon.noname-c con 0x7feb641376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:10.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:10.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feb64141d80 con 0x7feb640a5d00 2024-09-16T05:12:10.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:10.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7feb64141c00 con 0x7feb641370a0 2024-09-16T05:12:10.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:10.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7feb64141f00 con 0x7feb641376e0 2024-09-16T05:12:10.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:10.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient(hunting): authenticate will time out at 1215.561167s 2024-09-16T05:12:10.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb641376e0 0x7feb6413c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:10.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb69e0f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb640a5d00 0x7feb641cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:10.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb641376e0 0x7feb6413c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:59126/0 (socket says 172.21.15.114:59126) 2024-09-16T05:12:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 1 -- 172.21.15.114:0/2882822988 learned_addr learned my addr 172.21.15.114:0/2882822988 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 10 monclient(hunting): get_auth_request con 0x7feb641376e0 auth_method 0 2024-09-16T05:12:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb69e0f640 10 monclient(hunting): get_auth_request con 0x7feb640a5d00 auth_method 0 2024-09-16T05:12:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb69e0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb69e0f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb69e0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb69e0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb69e0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb69e0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6960e640 1 --2- 172.21.15.114:0/2882822988 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feb641370a0 0x7feb64136190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6960e640 10 monclient(hunting): get_auth_request con 0x7feb641370a0 auth_method 0 2024-09-16T05:12:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6960e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6960e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6960e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 10 monclient(hunting): handle_auth_done global_id 54446 payload 824 2024-09-16T05:12:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 1 -- 172.21.15.114:0/2882822988 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feb641370a0 msgr2=0x7feb64136190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 1 --2- 172.21.15.114:0/2882822988 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feb641370a0 0x7feb64136190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 1 -- 172.21.15.114:0/2882822988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb640a5d00 msgr2=0x7feb641cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 1 --2- 172.21.15.114:0/2882822988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb640a5d00 0x7feb641cf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 10 monclient: _finish_hunting 0 2024-09-16T05:12:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 1 monclient: found mon.noname-c 2024-09-16T05:12:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:12:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 1 -- 172.21.15.114:0/2882822988 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb64073d20 con 0x7feb641376e0 2024-09-16T05:12:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6a610640 1 --2- 172.21.15.114:0/2882822988 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb641376e0 0x7feb6413c060 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7feb6000c820 tx=0x7feb6000cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 10 client.?.objecter ms_handle_connect 0x7feb641376e0 2024-09-16T05:12:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:12:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 1 -- 172.21.15.114:0/2882822988 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb60015070 con 0x7feb641376e0 2024-09-16T05:12:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:12:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 10 monclient: dump: 2024-09-16T05:12:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:10.949 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:12:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:12:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 1 -- 172.21.15.114:0/2882822988 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb60004220 con 0x7feb641376e0 2024-09-16T05:12:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 1 -- 172.21.15.114:0/2882822988 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6000ce10 con 0x7feb641376e0 2024-09-16T05:12:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 10 monclient: dump: 2024-09-16T05:12:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:10.950 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5bfff640 4 set_mon_vals no callback set 2024-09-16T05:12:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 5 monclient: authenticate success, global_id 54446 2024-09-16T05:12:10.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient: _renew_subs 2024-09-16T05:12:10.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:10.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 -- 172.21.15.114:0/2882822988 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb6411fec0 con 0x7feb641376e0 2024-09-16T05:12:10.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 client.54446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:12:10.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient: _renew_subs 2024-09-16T05:12:10.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:10.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 -- 172.21.15.114:0/2882822988 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feb640a25c0 con 0x7feb641376e0 2024-09-16T05:12:10.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 librados: init done 2024-09-16T05:12:10.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:10.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:10.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:10.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb6c073640 1 -- 172.21.15.114:0/2882822988 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb64137ac0 con 0x7feb641376e0 2024-09-16T05:12:10.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 1 -- 172.21.15.114:0/2882822988 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7feb60007830 con 0x7feb641376e0 2024-09-16T05:12:10.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 1 --2- 172.21.15.114:0/2882822988 >> v2:172.21.15.149:6812/1185531135 conn(0x7feb48037810 0x7feb48039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:10.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 1 -- 172.21.15.114:0/2882822988 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(177..177 src has 1..177) ==== 6231+0+0 (secure 0 0 0) 0x7feb60049020 con 0x7feb641376e0 2024-09-16T05:12:10.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 10 client.54446.objecter ms_dispatch 0x7feb64000ba0 osd_map(177..177 src has 1..177) 2024-09-16T05:12:10.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 3 client.54446.objecter handle_osd_map got epochs [177,177] > 0 2024-09-16T05:12:10.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 3 client.54446.objecter handle_osd_map decoding full epoch 177 2024-09-16T05:12:10.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb5a7fc640 20 client.54446.objecter dump_active .. 0 homeless 2024-09-16T05:12:10.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.929+0000 7feb69e0f640 1 --2- 172.21.15.114:0/2882822988 >> v2:172.21.15.149:6812/1185531135 conn(0x7feb48037810 0x7feb48039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:10.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.933+0000 7feb69e0f640 10 monclient: get_auth_request con 0x7feb48037810 auth_method 0 2024-09-16T05:12:10.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.933+0000 7feb5a7fc640 10 client.54446.objecter ms_handle_connect 0x7feb48037810 2024-09-16T05:12:10.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.933+0000 7feb69e0f640 1 --2- 172.21.15.114:0/2882822988 >> v2:172.21.15.149:6812/1185531135 conn(0x7feb48037810 0x7feb48039cd0 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 crypto rx=0x7feb5c005680 tx=0x7feb5c003dd0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:10.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.937+0000 7feb5a7fc640 1 -- 172.21.15.114:0/2882822988 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7feb64073d20 con 0x7feb641376e0 2024-09-16T05:12:10.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.937+0000 7feb5a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:10.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:10.937+0000 7feb5a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:12:11.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:12:11.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:12:11.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:11.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 1 -- 172.21.15.114:0/2882822988 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7feb6413e5e0 con 0x7feb641376e0 2024-09-16T05:12:11.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb5a7fc640 1 -- 172.21.15.114:0/2882822988 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7feb60021960 con 0x7feb641376e0 2024-09-16T05:12:11.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb5a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:12:11.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb5a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:12:11.161 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:12:11.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 10 librados: watch_flush enter 2024-09-16T05:12:11.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 10 librados: watch_flush exit 2024-09-16T05:12:11.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 20 client.54446.objecter shutdown clearing up homeless session... 2024-09-16T05:12:11.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 10 client.54446.objecter successfully canceled tick 2024-09-16T05:12:11.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 1 -- 172.21.15.114:0/2882822988 >> v2:172.21.15.149:6812/1185531135 conn(0x7feb48037810 msgr2=0x7feb48039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 1 --2- 172.21.15.114:0/2882822988 >> v2:172.21.15.149:6812/1185531135 conn(0x7feb48037810 0x7feb48039cd0 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 crypto rx=0x7feb5c005680 tx=0x7feb5c003dd0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 10 monclient: shutdown 2024-09-16T05:12:11.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:11.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 1 -- 172.21.15.114:0/2882822988 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb641376e0 msgr2=0x7feb6413c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 1 --2- 172.21.15.114:0/2882822988 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb641376e0 0x7feb6413c060 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7feb6000c820 tx=0x7feb6000cce0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 1 -- 172.21.15.114:0/2882822988 shutdown_connections 2024-09-16T05:12:11.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 1 --2- 172.21.15.114:0/2882822988 >> v2:172.21.15.149:6812/1185531135 conn(0x7feb48037810 0x7feb48039cd0 unknown :-1 s=CLOSED pgs=3451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 1 --2- 172.21.15.114:0/2882822988 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb641376e0 0x7feb6413c060 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 1 --2- 172.21.15.114:0/2882822988 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feb641370a0 0x7feb64136190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 1 --2- 172.21.15.114:0/2882822988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb640a5d00 0x7feb641cf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 1 -- 172.21.15.114:0/2882822988 >> 172.21.15.114:0/2882822988 conn(0x7feb640a9a40 msgr2=0x7feb640a7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:11.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 1 -- 172.21.15.114:0/2882822988 shutdown_connections 2024-09-16T05:12:11.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 1 -- 172.21.15.114:0/2882822988 wait complete. 2024-09-16T05:12:11.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.157+0000 7feb6c073640 1 librados: shutdown 2024-09-16T05:12:11.178 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:12:11.178 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:12:11.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:12:11.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 10 monclient: get_monmap_and_config 2024-09-16T05:12:11.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 10 monclient: build_initial_monmap 2024-09-16T05:12:11.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 10 monclient: monmap: 2024-09-16T05:12:11.296 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:11.296 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.296 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:12:11.296496+0000 2024-09-16T05:12:11.297 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:12:11.296496+0000 2024-09-16T05:12:11.297 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:11.297 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.297 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:12:11.297 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:12:11.297 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:12:11.297 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:12:11.297 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:12:11.297 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 1 Processor -- start 2024-09-16T05:12:11.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 1 -- start start 2024-09-16T05:12:11.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 10 monclient: init 2024-09-16T05:12:11.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:11.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-16T05:12:11.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb59809b6c0 addr v1:172.21.15.114:6789/0 2024-09-16T05:12:11.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5980a15d0 addr v1:172.21.15.149:6789/0 2024-09-16T05:12:11.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5980a19e0 0x7fb59813dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 10 monclient(hunting): picked mon.noname-b con 0x7fb5980a19e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:11.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb59809d4e0 con 0x7fb59809b6c0 2024-09-16T05:12:11.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb59809f430 con 0x7fb5980a15d0 2024-09-16T05:12:11.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb59809e390 con 0x7fb5980a19e0 2024-09-16T05:12:11.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:11.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb59d4e7640 10 monclient(hunting): authenticate will time out at 1215.925684s 2024-09-16T05:12:11.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb596ffd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb59809b6c0 0x7fb5980a0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38516/0 (socket says 172.21.15.114:38516) 2024-09-16T05:12:11.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5980a19e0 0x7fb59813dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb596ffd640 1 -- 172.21.15.114:0/995616245 learned_addr learned my addr 172.21.15.114:0/995616245 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:12:11.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 10 monclient(hunting): get_auth_request con 0x7fb5980a19e0 auth_method 0 2024-09-16T05:12:11.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb595ffb640 1 -- 172.21.15.114:0/995616245 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1848369716 0 0) 0x7fb59809d4e0 con 0x7fb59809b6c0 2024-09-16T05:12:11.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb595ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:12:11.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb595ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb595ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb595ffb640 10 monclient(hunting): my global_id is 27966 2024-09-16T05:12:11.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb595ffb640 1 -- 172.21.15.114:0/995616245 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb584003530 con 0x7fb59809b6c0 2024-09-16T05:12:11.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 10 monclient(hunting): handle_auth_done global_id 54449 payload 274 2024-09-16T05:12:11.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 1 -- 172.21.15.114:0/995616245 >> v1:172.21.15.149:6789/0 conn(0x7fb5980a15d0 legacy=0x7fb59813a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 1 -- 172.21.15.114:0/995616245 >> v1:172.21.15.114:6789/0 conn(0x7fb59809b6c0 legacy=0x7fb5980a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 10 monclient: _finish_hunting 0 2024-09-16T05:12:11.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 1 monclient: found mon.noname-b 2024-09-16T05:12:11.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:11.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:12:11.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.293+0000 7fb5977fe640 1 -- 172.21.15.114:0/995616245 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb59809c2f0 con 0x7fb5980a19e0 2024-09-16T05:12:11.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5977fe640 1 --2- 172.21.15.114:0/995616245 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5980a19e0 0x7fb59813dd80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb5900097f0 tx=0x7fb59001c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbe308d33d2839ad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb595ffb640 1 -- 172.21.15.114:0/995616245 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5900b0070 con 0x7fb5980a19e0 2024-09-16T05:12:11.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb595ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb595ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:12:11.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb595ffb640 10 monclient: dump: 2024-09-16T05:12:11.309 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.309 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.309 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.309 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.309 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.309 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.309 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.309 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.309 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.309 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb595ffb640 10 monclient: _finish_auth 0 2024-09-16T05:12:11.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb595ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:12:11.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb595ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:11.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 5 monclient: authenticate success, global_id 54449 2024-09-16T05:12:11.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:12:11.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb595ffb640 1 -- 172.21.15.114:0/995616245 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5900ab600 con 0x7fb5980a19e0 2024-09-16T05:12:11.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb595ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:11.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 10 monclient: get_monmap_and_config success 2024-09-16T05:12:11.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 4 set_mon_vals no callback set 2024-09-16T05:12:11.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb595ffb640 1 -- 172.21.15.114:0/995616245 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5900ab9b0 con 0x7fb5980a19e0 2024-09-16T05:12:11.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb595ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb595ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:11.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb595ffb640 10 monclient: dump: 2024-09-16T05:12:11.311 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.311 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.311 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.311 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.311 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.311 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.311 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.311 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.311 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.311 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 10 monclient: shutdown 2024-09-16T05:12:11.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:11.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 -- 172.21.15.114:0/995616245 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5980a19e0 msgr2=0x7fb59813dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 --2- 172.21.15.114:0/995616245 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5980a19e0 0x7fb59813dd80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb5900097f0 tx=0x7fb59001c3a0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 -- 172.21.15.114:0/995616245 shutdown_connections 2024-09-16T05:12:11.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 --2- 172.21.15.114:0/995616245 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5980a19e0 0x7fb59813dd80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 -- 172.21.15.114:0/995616245 >> 172.21.15.114:0/995616245 conn(0x7fb5980a1fb0 msgr2=0x7fb5980a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:11.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 -- 172.21.15.114:0/995616245 shutdown_connections 2024-09-16T05:12:11.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 -- 172.21.15.114:0/995616245 wait complete. 2024-09-16T05:12:11.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 10 monclient: build_initial_monmap 2024-09-16T05:12:11.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 10 monclient: monmap: 2024-09-16T05:12:11.312 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:11.312 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:12:11.312 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:12:11.312 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:12:11.312 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:11.312 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.313 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:12:11.313 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:12:11.313 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:12:11.313 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 librados: starting msgr at 2024-09-16T05:12:11.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 librados: starting objecter 2024-09-16T05:12:11.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 Processor -- start 2024-09-16T05:12:11.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 -- start start 2024-09-16T05:12:11.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 librados: setting wanted keys 2024-09-16T05:12:11.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 librados: calling monclient init 2024-09-16T05:12:11.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 10 monclient: init 2024-09-16T05:12:11.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:11.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:12:11.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59809b6c0 0x7fb598134610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 10 monclient(hunting): picked mon.noname-a con 0x7fb59809b6c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:11.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5980a15d0 0x7fb598134b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 10 monclient(hunting): picked mon.noname-c con 0x7fb5980a15d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:11.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5980a19e0 0x7fb598139a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 10 monclient(hunting): picked mon.noname-b con 0x7fb5980a19e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:11.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb598140c60 con 0x7fb59809b6c0 2024-09-16T05:12:11.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb598140ae0 con 0x7fb5980a19e0 2024-09-16T05:12:11.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb598140de0 con 0x7fb5980a15d0 2024-09-16T05:12:11.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:11.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb59d4e7640 10 monclient(hunting): authenticate will time out at 1215.931171s 2024-09-16T05:12:11.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb596ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59809b6c0 0x7fb598134610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5980a15d0 0x7fb598134b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb596ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59809b6c0 0x7fb598134610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36314/0 (socket says 172.21.15.114:36314) 2024-09-16T05:12:11.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5980a15d0 0x7fb598134b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:59138/0 (socket says 172.21.15.114:59138) 2024-09-16T05:12:11.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb596ffd640 1 -- 172.21.15.114:0/1686604109 learned_addr learned my addr 172.21.15.114:0/1686604109 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:11.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb596ffd640 10 monclient(hunting): get_auth_request con 0x7fb59809b6c0 auth_method 0 2024-09-16T05:12:11.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb596ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb596ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 10 monclient(hunting): get_auth_request con 0x7fb5980a15d0 auth_method 0 2024-09-16T05:12:11.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb596ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb596ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb596ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 10 monclient(hunting): handle_auth_done global_id 54452 payload 824 2024-09-16T05:12:11.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 1 -- 172.21.15.114:0/1686604109 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5980a19e0 msgr2=0x7fb598139a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5977fe640 1 --2- 172.21.15.114:0/1686604109 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5980a19e0 0x7fb598139a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 1 --2- 172.21.15.114:0/1686604109 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5980a19e0 0x7fb598139a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 1 -- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59809b6c0 msgr2=0x7fb598134610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 1 --2- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59809b6c0 0x7fb598134610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 10 monclient: _finish_hunting 0 2024-09-16T05:12:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 1 monclient: found mon.noname-c 2024-09-16T05:12:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:12:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 1 -- 172.21.15.114:0/1686604109 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb59813d5f0 con 0x7fb5980a15d0 2024-09-16T05:12:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5967fc640 1 --2- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5980a15d0 0x7fb598134b50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb588002290 tx=0x7fb58800cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb58f7fe640 10 client.?.objecter ms_handle_connect 0x7fb5980a15d0 2024-09-16T05:12:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb58f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:12:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb58f7fe640 1 -- 172.21.15.114:0/1686604109 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb588003bf0 con 0x7fb5980a15d0 2024-09-16T05:12:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb58f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb58f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:12:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb58f7fe640 10 monclient: dump: 2024-09-16T05:12:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.318 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb58f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:12:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb58f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:12:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb58f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:11.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb58f7fe640 1 -- 172.21.15.114:0/1686604109 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb588003db0 con 0x7fb5980a15d0 2024-09-16T05:12:11.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb58f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:11.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb58f7fe640 1 -- 172.21.15.114:0/1686604109 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb588005310 con 0x7fb5980a15d0 2024-09-16T05:12:11.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb58f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb58f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:11.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb58f7fe640 10 monclient: dump: 2024-09-16T05:12:11.319 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.319 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.319 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.319 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.319 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.319 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.319 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.320 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.297+0000 7fb5957fa640 4 set_mon_vals no callback set 2024-09-16T05:12:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb59d4e7640 5 monclient: authenticate success, global_id 54452 2024-09-16T05:12:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb59d4e7640 10 monclient: _renew_subs 2024-09-16T05:12:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb59d4e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb59d4e7640 1 -- 172.21.15.114:0/1686604109 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb59806ddb0 con 0x7fb5980a15d0 2024-09-16T05:12:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb59d4e7640 10 client.54452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:12:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb59d4e7640 10 monclient: _renew_subs 2024-09-16T05:12:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb59d4e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb59d4e7640 1 -- 172.21.15.114:0/1686604109 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb59813d8d0 con 0x7fb5980a15d0 2024-09-16T05:12:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb59d4e7640 1 librados: init done 2024-09-16T05:12:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb59d4e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb59d4e7640 1 --2- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59813d030 0x7fb598139f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb59d4e7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb59d4e7640 1 -- 172.21.15.114:0/1686604109 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb59809c660 con 0x7fb59813d030 2024-09-16T05:12:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 1 -- 172.21.15.114:0/1686604109 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb5880056c0 con 0x7fb5980a15d0 2024-09-16T05:12:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb596ffd640 1 --2- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59813d030 0x7fb598139f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb596ffd640 10 monclient: get_auth_request con 0x7fb59813d030 auth_method 0 2024-09-16T05:12:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb596ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb596ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb596ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb596ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb596ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 1 --2- 172.21.15.114:0/1686604109 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb578037810 0x7fb578039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 1 -- 172.21.15.114:0/1686604109 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(177..177 src has 1..177) ==== 6231+0+0 (secure 0 0 0) 0x7fb5880492a0 con 0x7fb5980a15d0 2024-09-16T05:12:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 10 client.54452.objecter ms_dispatch 0x7fb598000ba0 osd_map(177..177 src has 1..177) 2024-09-16T05:12:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 3 client.54452.objecter handle_osd_map got epochs [177,177] > 0 2024-09-16T05:12:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 3 client.54452.objecter handle_osd_map decoding full epoch 177 2024-09-16T05:12:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb596ffd640 10 monclient(hunting): handle_auth_done global_id 27972 payload 824 2024-09-16T05:12:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 20 client.54452.objecter dump_active .. 0 homeless 2024-09-16T05:12:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb596ffd640 1 -- 172.21.15.114:0/1686604109 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb59813ba20 con 0x7fb59813d030 2024-09-16T05:12:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 10 client.54452.objecter ms_handle_connect 0x7fb59813d030 2024-09-16T05:12:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 10 client.54452.objecter resend_mon_ops 2024-09-16T05:12:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb596ffd640 1 --2- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59813d030 0x7fb598139f50 secure :-1 s=READY pgs=8304 cs=0 l=1 rev1=1 crypto rx=0x7fb580000f70 tx=0x7fb58000fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb5977fe640 1 --2- 172.21.15.114:0/1686604109 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb578037810 0x7fb578039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 1 -- 172.21.15.114:0/1686604109 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5800058c0 con 0x7fb59813d030 2024-09-16T05:12:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:12:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 10 monclient: dump: 2024-09-16T05:12:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.324 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb5977fe640 10 monclient: get_auth_request con 0x7fb578037810 auth_method 0 2024-09-16T05:12:11.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 1 -- 172.21.15.114:0/1686604109 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb580018070 con 0x7fb59813d030 2024-09-16T05:12:11.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:11.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:12:11.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 1 -- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59813d030 msgr2=0x7fb598139f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 1 --2- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59813d030 0x7fb598139f50 secure :-1 s=READY pgs=8304 cs=0 l=1 rev1=1 crypto rx=0x7fb580000f70 tx=0x7fb58000fc00 comp rx=0 tx=0).stop 2024-09-16T05:12:11.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 1 -- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59813d030 msgr2=0x7fb598139f50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:11.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 1 --2- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59813d030 0x7fb598139f50 unknown :-1 s=CLOSED pgs=8304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb58f7fe640 10 client.54452.objecter ms_handle_connect 0x7fb578037810 2024-09-16T05:12:11.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.301+0000 7fb5977fe640 1 --2- 172.21.15.114:0/1686604109 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb578037810 0x7fb578039cd0 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7fb5900ad000 tx=0x7fb5900a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:12:11.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 1 --2- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59813a490 0x7fb59813a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 1 -- 172.21.15.114:0/1686604109 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb59813ade0 con 0x7fb59813a490 2024-09-16T05:12:11.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb596ffd640 1 --2- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59813a490 0x7fb59813a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb596ffd640 10 monclient: get_auth_request con 0x7fb59813a490 auth_method 0 2024-09-16T05:12:11.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb596ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb596ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb596ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb596ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb596ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb596ffd640 10 monclient(hunting): handle_auth_done global_id 27975 payload 824 2024-09-16T05:12:11.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb596ffd640 1 -- 172.21.15.114:0/1686604109 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb5980686f0 con 0x7fb59813a490 2024-09-16T05:12:11.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb58f7fe640 10 client.54452.objecter ms_handle_connect 0x7fb59813a490 2024-09-16T05:12:11.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb58f7fe640 10 client.54452.objecter resend_mon_ops 2024-09-16T05:12:11.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb596ffd640 1 --2- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59813a490 0x7fb59813a8a0 secure :-1 s=READY pgs=8305 cs=0 l=1 rev1=1 crypto rx=0x7fb580000f70 tx=0x7fb580001160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb58f7fe640 1 -- 172.21.15.114:0/1686604109 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5800072e0 con 0x7fb59813a490 2024-09-16T05:12:11.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb58f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb58f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:12:11.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb58f7fe640 10 monclient: dump: 2024-09-16T05:12:11.328 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.328 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.328 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.328 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.328 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.328 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.328 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.328 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.328 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.328 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb58f7fe640 1 -- 172.21.15.114:0/1686604109 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7fb5800072e0 con 0x7fb59813a490 2024-09-16T05:12:11.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb58f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:12:11.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb58f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:12:11.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb58f7fe640 1 -- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59813a490 msgr2=0x7fb59813a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb58f7fe640 1 --2- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59813a490 0x7fb59813a8a0 secure :-1 s=READY pgs=8305 cs=0 l=1 rev1=1 crypto rx=0x7fb580000f70 tx=0x7fb580001160 comp rx=0 tx=0).stop 2024-09-16T05:12:11.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb58f7fe640 1 -- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59813a490 msgr2=0x7fb59813a8a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:11.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb58f7fe640 1 --2- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb59813a490 0x7fb59813a8a0 secure :-1 s=CLOSED pgs=8305 cs=0 l=1 rev1=1 crypto rx=0x7fb580000f70 tx=0x7fb580001160 comp rx=0 tx=0).stop 2024-09-16T05:12:11.329 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:12:11.329 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:12:11.329 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:12:11.329 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:12:11.330 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 20, 2024-09-16T05:12:11.330 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 289298, 2024-09-16T05:12:11.330 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:12:11.330 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:12:11.330 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:12:11.330 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:12:11.330 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:11.330 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:12:11.330 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:11.330 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:12:11.330 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:12:11.330 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:11.330 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:11.331 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:11.331 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:11.331 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:11.331 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:11.331 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:11.331 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:11.331 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:11.331 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:11.331 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:11.331 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:11.331 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:12:11.331 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:12:11.331 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:11.332 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:11.332 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:11.332 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:11.332 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:11.332 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:11.332 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:11.332 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:11.332 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:11.332 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:11.332 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:11.332 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.333 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:12:11.333 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:12:11.333 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:12:11.333 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:12:11.333 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:12:11.333 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:12:11.333 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:11.333 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:11.333 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:12:11.333 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:12:11.333 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:12:11.333 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:12:11.333 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:12:11.333 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:12:11.334 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:12:11.334 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:11.334 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:12:11.334 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:11.334 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:11.334 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:11.334 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:11.334 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:11.334 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:11.334 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:11.334 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:11.334 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:11.334 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:11.335 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:11.335 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:11.335 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:12:11.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.335 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:12:11.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.335 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:12:11.335 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:12:11.335 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:11.335 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:11.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.335 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:11.335 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:12:11.335 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.336 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.336 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:11.336 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:12:11.336 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.336 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.336 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.336 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:12:11.336 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:12:11.336 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:11.336 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:11.337 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:11.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.337 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:12:11.337 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:12:11.337 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:11.337 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:11.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.337 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:11.337 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:12:11.337 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.337 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:11.338 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:12:11.338 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.338 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.338 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.338 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:12:11.338 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:12:11.338 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:11.338 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:11.338 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:11.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.338 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:12:11.338 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:12:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:12:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:12:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:12:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:12:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:12:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:12:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:12:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:12:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:12:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:12:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:11.342 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:12:11.342 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.342 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.342 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:12:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:12:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 10 librados: watch_flush enter 2024-09-16T05:12:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 10 librados: watch_flush exit 2024-09-16T05:12:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 20 client.54452.objecter shutdown clearing up homeless session... 2024-09-16T05:12:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 10 client.54452.objecter successfully canceled tick 2024-09-16T05:12:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 1 -- 172.21.15.114:0/1686604109 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb578037810 msgr2=0x7fb578039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 1 --2- 172.21.15.114:0/1686604109 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb578037810 0x7fb578039cd0 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7fb5900ad000 tx=0x7fb5900a6000 comp rx=0 tx=0).stop 2024-09-16T05:12:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 10 monclient: shutdown 2024-09-16T05:12:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 1 -- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5980a15d0 msgr2=0x7fb598134b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 1 --2- 172.21.15.114:0/1686604109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5980a15d0 0x7fb598134b50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb588002290 tx=0x7fb58800cb90 comp rx=0 tx=0).stop 2024-09-16T05:12:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb5977fe640 1 -- 172.21.15.114:0/1686604109 reap_dead start 2024-09-16T05:12:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb5977fe640 1 -- 172.21.15.114:0/1686604109 reap_dead start 2024-09-16T05:12:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 1 -- 172.21.15.114:0/1686604109 shutdown_connections 2024-09-16T05:12:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 1 -- 172.21.15.114:0/1686604109 >> 172.21.15.114:0/1686604109 conn(0x7fb5980a1fb0 msgr2=0x7fb598137860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 1 -- 172.21.15.114:0/1686604109 shutdown_connections 2024-09-16T05:12:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 1 -- 172.21.15.114:0/1686604109 wait complete. 2024-09-16T05:12:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.309+0000 7fb59d4e7640 1 librados: shutdown 2024-09-16T05:12:11.345 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:12:11.345 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:12:11.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:12:11.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 10 monclient: get_monmap_and_config 2024-09-16T05:12:11.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 10 monclient: build_initial_monmap 2024-09-16T05:12:11.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 10 monclient: monmap: 2024-09-16T05:12:11.465 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:11.465 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.465 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:12:11.464816+0000 2024-09-16T05:12:11.465 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:12:11.464816+0000 2024-09-16T05:12:11.465 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:11.465 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.465 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:12:11.466 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:12:11.466 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:12:11.466 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:12:11.466 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:12:11.466 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 1 Processor -- start 2024-09-16T05:12:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 1 -- start start 2024-09-16T05:12:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 10 monclient: init 2024-09-16T05:12:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T05:12:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f240c0a0ec0 0x7f240c09dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 10 monclient(hunting): picked mon.noname-a con 0x7f240c0a0ec0 addr v2:172.21.15.114:3300/0 2024-09-16T05:12:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f240c09e000 addr v1:172.21.15.149:6789/0 2024-09-16T05:12:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f240c139640 addr v1:172.21.15.114:6789/0 2024-09-16T05:12:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f240c09a760 con 0x7f240c139640 2024-09-16T05:12:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f240c0a1870 con 0x7f240c09e000 2024-09-16T05:12:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f240c0994b0 con 0x7f240c0a0ec0 2024-09-16T05:12:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2412ac8640 10 monclient(hunting): authenticate will time out at 1216.093980s 2024-09-16T05:12:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f240c0a0ec0 0x7f240c09dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2411065640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f240c139640 0x7f240c13ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38528/0 (socket says 172.21.15.114:38528) 2024-09-16T05:12:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2411065640 1 -- 172.21.15.114:0/1997019645 learned_addr learned my addr 172.21.15.114:0/1997019645 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:12:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 10 monclient(hunting): get_auth_request con 0x7f240c0a0ec0 auth_method 0 2024-09-16T05:12:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 10 monclient(hunting): handle_auth_done global_id 27978 payload 274 2024-09-16T05:12:11.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 1 -- 172.21.15.114:0/1997019645 >> v1:172.21.15.149:6789/0 conn(0x7f240c09e000 legacy=0x7f240c09e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 1 -- 172.21.15.114:0/1997019645 >> v1:172.21.15.114:6789/0 conn(0x7f240c139640 legacy=0x7f240c13ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 10 monclient: _finish_hunting 0 2024-09-16T05:12:11.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 1 monclient: found mon.noname-a 2024-09-16T05:12:11.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:11.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:12:11.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.461+0000 7f2410864640 1 -- 172.21.15.114:0/1997019645 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240c099790 con 0x7f240c0a0ec0 2024-09-16T05:12:11.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2410864640 1 --2- 172.21.15.114:0/1997019645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f240c0a0ec0 0x7f240c09dac0 secure :-1 s=READY pgs=8307 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0097c0 tx=0x7f23fc01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef190be9ee59f708 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240b7fe640 1 -- 172.21.15.114:0/1997019645 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc0b0070 con 0x7f240c0a0ec0 2024-09-16T05:12:11.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:12:11.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240b7fe640 10 monclient: dump: 2024-09-16T05:12:11.478 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.478 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.478 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.479 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.479 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.479 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.479 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.479 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.479 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.479 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:12:11.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:12:11.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:11.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 5 monclient: authenticate success, global_id 27978 2024-09-16T05:12:11.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:12:11.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240b7fe640 1 -- 172.21.15.114:0/1997019645 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23fc0ab600 con 0x7f240c0a0ec0 2024-09-16T05:12:11.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:11.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 10 monclient: get_monmap_and_config success 2024-09-16T05:12:11.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240b7fe640 1 -- 172.21.15.114:0/1997019645 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc0ab9d0 con 0x7f240c0a0ec0 2024-09-16T05:12:11.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 4 set_mon_vals no callback set 2024-09-16T05:12:11.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:12:11.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240b7fe640 10 monclient: dump: 2024-09-16T05:12:11.480 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.480 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.480 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.480 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.481 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.481 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.481 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.481 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.481 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.481 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 10 monclient: shutdown 2024-09-16T05:12:11.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:11.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 -- 172.21.15.114:0/1997019645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f240c0a0ec0 msgr2=0x7f240c09dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 --2- 172.21.15.114:0/1997019645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f240c0a0ec0 0x7f240c09dac0 secure :-1 s=READY pgs=8307 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0097c0 tx=0x7f23fc01c3f0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 -- 172.21.15.114:0/1997019645 shutdown_connections 2024-09-16T05:12:11.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 --2- 172.21.15.114:0/1997019645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f240c0a0ec0 0x7f240c09dac0 unknown :-1 s=CLOSED pgs=8307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 -- 172.21.15.114:0/1997019645 >> 172.21.15.114:0/1997019645 conn(0x7f240c0a1fb0 msgr2=0x7f240c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:11.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 -- 172.21.15.114:0/1997019645 shutdown_connections 2024-09-16T05:12:11.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 -- 172.21.15.114:0/1997019645 wait complete. 2024-09-16T05:12:11.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 10 monclient: build_initial_monmap 2024-09-16T05:12:11.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 10 monclient: monmap: 2024-09-16T05:12:11.482 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:11.482 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:12:11.482 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:12:11.482 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:12:11.482 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:11.482 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.482 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:12:11.482 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:12:11.482 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:12:11.482 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 librados: starting msgr at 2024-09-16T05:12:11.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 librados: starting objecter 2024-09-16T05:12:11.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 Processor -- start 2024-09-16T05:12:11.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 -- start start 2024-09-16T05:12:11.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 librados: setting wanted keys 2024-09-16T05:12:11.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 librados: calling monclient init 2024-09-16T05:12:11.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 10 monclient: init 2024-09-16T05:12:11.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:11.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:12:11.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f240c09e000 0x7f240c139010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 10 monclient(hunting): picked mon.noname-a con 0x7f240c09e000 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:11.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c0a0ec0 0x7f240c098aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 10 monclient(hunting): picked mon.noname-c con 0x7f240c0a0ec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:11.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f240c139640 0x7f240c139a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 10 monclient(hunting): picked mon.noname-b con 0x7f240c139640 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:11.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f240c13faf0 con 0x7f240c09e000 2024-09-16T05:12:11.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f240c13f970 con 0x7f240c139640 2024-09-16T05:12:11.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f240c13fc70 con 0x7f240c0a0ec0 2024-09-16T05:12:11.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:11.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2412ac8640 10 monclient(hunting): authenticate will time out at 1216.099579s 2024-09-16T05:12:11.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2410864640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f240c09e000 0x7f240c139010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2410864640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f240c09e000 0x7f240c139010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36352/0 (socket says 172.21.15.114:36352) 2024-09-16T05:12:11.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2410864640 1 -- 172.21.15.114:0/1715038 learned_addr learned my addr 172.21.15.114:0/1715038 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:11.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2410864640 10 monclient(hunting): get_auth_request con 0x7f240c09e000 auth_method 0 2024-09-16T05:12:11.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2410864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2410864640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2410864640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240bfff640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c0a0ec0 0x7f240c098aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240bfff640 10 monclient(hunting): get_auth_request con 0x7f240c0a0ec0 auth_method 0 2024-09-16T05:12:11.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2410864640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2410864640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2410864640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f240bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2410864640 10 monclient(hunting): handle_auth_done global_id 27984 payload 824 2024-09-16T05:12:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.465+0000 7f2411065640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f240c139640 0x7f240c139a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2410864640 1 -- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c0a0ec0 msgr2=0x7f240c098aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2410864640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c0a0ec0 0x7f240c098aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2410864640 1 -- 172.21.15.114:0/1715038 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f240c139640 msgr2=0x7f240c139a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2410864640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f240c139640 0x7f240c139a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2410864640 10 monclient: _finish_hunting 0 2024-09-16T05:12:11.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2410864640 1 monclient: found mon.noname-a 2024-09-16T05:12:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2410864640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2410864640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:12:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2410864640 1 -- 172.21.15.114:0/1715038 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240c093ac0 con 0x7f240c09e000 2024-09-16T05:12:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2411065640 10 monclient: get_auth_request con 0x7f240c139640 auth_method 0 2024-09-16T05:12:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2411065640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f240c139640 0x7f240c139a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:12:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2410864640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f240c09e000 0x7f240c139010 secure :-1 s=READY pgs=8308 cs=0 l=1 rev1=1 crypto rx=0x7f23fc019000 tx=0x7f23fc019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 client.?.objecter ms_handle_connect 0x7f240c09e000 2024-09-16T05:12:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:12:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 1 -- 172.21.15.114:0/1715038 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc0b8020 con 0x7f240c09e000 2024-09-16T05:12:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:12:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 monclient: dump: 2024-09-16T05:12:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.488 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 monclient: _finish_auth 0 2024-09-16T05:12:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 monclient: _check_auth_tickets 2024-09-16T05:12:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2412ac8640 5 monclient: authenticate success, global_id 27984 2024-09-16T05:12:11.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 1 -- 172.21.15.114:0/1715038 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23fc0b4610 con 0x7f240c09e000 2024-09-16T05:12:11.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2412ac8640 10 monclient: _renew_subs 2024-09-16T05:12:11.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2412ac8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:12:11.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2412ac8640 1 -- 172.21.15.114:0/1715038 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f240c093da0 con 0x7f240c09e000 2024-09-16T05:12:11.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:11.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 1 -- 172.21.15.114:0/1715038 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc0b0040 con 0x7f240c09e000 2024-09-16T05:12:11.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:12:11.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 monclient: dump: 2024-09-16T05:12:11.489 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.489 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.489 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.489 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.489 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.490 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.490 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.490 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.490 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.490 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2412ac8640 10 client.27984.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:12:11.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f240a7fc640 4 set_mon_vals no callback set 2024-09-16T05:12:11.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2412ac8640 10 monclient: _renew_subs 2024-09-16T05:12:11.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2412ac8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:12:11.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2412ac8640 1 -- 172.21.15.114:0/1715038 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f240c0a4b50 con 0x7f240c09e000 2024-09-16T05:12:11.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2412ac8640 1 librados: init done 2024-09-16T05:12:11.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2412ac8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:11.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2412ac8640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c139f60 0x7f240c13a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2412ac8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2412ac8640 1 -- 172.21.15.114:0/1715038 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f240c069390 con 0x7f240c139f60 2024-09-16T05:12:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f240bfff640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c139f60 0x7f240c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f240bfff640 10 monclient: get_auth_request con 0x7f240c139f60 auth_method 0 2024-09-16T05:12:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f240bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f240bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f240bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f240bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f240bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f240bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 1 -- 172.21.15.114:0/1715038 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f23fc0bb020 con 0x7f240c09e000 2024-09-16T05:12:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f240bfff640 10 monclient(hunting): handle_auth_done global_id 54458 payload 824 2024-09-16T05:12:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f240bfff640 1 -- 172.21.15.114:0/1715038 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f240c13a880 con 0x7f240c139f60 2024-09-16T05:12:11.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 1 --2- 172.21.15.114:0/1715038 >> v2:172.21.15.149:6812/1185531135 conn(0x7f23f8037810 0x7f23f8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 1 -- 172.21.15.114:0/1715038 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 6231+0+0 (secure 0 0 0) 0x7f23fc0e2e00 con 0x7f240c09e000 2024-09-16T05:12:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 client.27984.objecter ms_dispatch 0x7f240c000ba0 osd_map(177..177 src has 1..177) 2024-09-16T05:12:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 3 client.27984.objecter handle_osd_map got epochs [177,177] > 0 2024-09-16T05:12:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 3 client.27984.objecter handle_osd_map decoding full epoch 177 2024-09-16T05:12:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f240bfff640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c139f60 0x7f240c13a340 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2400000fc0 tx=0x7f240000f9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 20 client.27984.objecter dump_active .. 0 homeless 2024-09-16T05:12:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 client.27984.objecter ms_handle_connect 0x7f240c139f60 2024-09-16T05:12:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 client.27984.objecter resend_mon_ops 2024-09-16T05:12:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 1 -- 172.21.15.114:0/1715038 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24000052e0 con 0x7f240c139f60 2024-09-16T05:12:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 monclient: dump: 2024-09-16T05:12:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.492 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.493 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.493 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.493 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.493 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.493 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.493 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.493 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.493 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2411065640 1 --2- 172.21.15.114:0/1715038 >> v2:172.21.15.149:6812/1185531135 conn(0x7f23f8037810 0x7f23f8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2411065640 10 monclient: get_auth_request con 0x7f23f8037810 auth_method 0 2024-09-16T05:12:11.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 1 -- 172.21.15.114:0/1715038 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f24000052e0 con 0x7f240c139f60 2024-09-16T05:12:11.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:11.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:12:11.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 1 -- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c139f60 msgr2=0x7f240c13a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c139f60 0x7f240c13a340 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2400000fc0 tx=0x7f240000f9d0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 1 -- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c139f60 msgr2=0x7f240c13a340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:11.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c139f60 0x7f240c13a340 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2400000fc0 tx=0x7f240000f9d0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f24097fa640 10 client.27984.objecter ms_handle_connect 0x7f23f8037810 2024-09-16T05:12:11.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.469+0000 7f2411065640 1 --2- 172.21.15.114:0/1715038 >> v2:172.21.15.149:6812/1185531135 conn(0x7f23f8037810 0x7f23f8039cd0 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 crypto rx=0x7f240c0930a0 tx=0x7f2404009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:12:11.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c0709b0 0x7f240c070d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 1 -- 172.21.15.114:0/1715038 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f240c0937e0 con 0x7f240c0709b0 2024-09-16T05:12:11.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f240bfff640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c0709b0 0x7f240c070d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f240bfff640 10 monclient: get_auth_request con 0x7f240c0709b0 auth_method 0 2024-09-16T05:12:11.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f240bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f240bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f240bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f240bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f240bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f240bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f240bfff640 10 monclient(hunting): handle_auth_done global_id 54461 payload 824 2024-09-16T05:12:11.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f240bfff640 1 -- 172.21.15.114:0/1715038 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f240c09dac0 con 0x7f240c0709b0 2024-09-16T05:12:11.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f240bfff640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c0709b0 0x7f240c070d90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2400000fc0 tx=0x7f2400000f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f24097fa640 10 client.27984.objecter ms_handle_connect 0x7f240c0709b0 2024-09-16T05:12:11.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f24097fa640 10 client.27984.objecter resend_mon_ops 2024-09-16T05:12:11.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f24097fa640 1 -- 172.21.15.114:0/1715038 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240001edb0 con 0x7f240c0709b0 2024-09-16T05:12:11.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f24097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f24097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:11.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f24097fa640 10 monclient: dump: 2024-09-16T05:12:11.498 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.498 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.498 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.498 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.498 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.498 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.498 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.498 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.498 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.498 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f24097fa640 1 -- 172.21.15.114:0/1715038 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f240001e110 con 0x7f240c0709b0 2024-09-16T05:12:11.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f24097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:12:11.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f24097fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:12:11.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f24097fa640 1 -- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c0709b0 msgr2=0x7f240c070d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f24097fa640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c0709b0 0x7f240c070d90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2400000fc0 tx=0x7f2400000f30 comp rx=0 tx=0).stop 2024-09-16T05:12:11.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f24097fa640 1 -- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c0709b0 msgr2=0x7f240c070d90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:11.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f24097fa640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f240c0709b0 0x7f240c070d90 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2400000fc0 tx=0x7f2400000f30 comp rx=0 tx=0).stop 2024-09-16T05:12:11.499 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:12:11.499 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:12:11.499 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:12:11.499 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:12:11.499 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 20, 2024-09-16T05:12:11.499 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 145613, 2024-09-16T05:12:11.499 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:12:11.500 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:12:11.500 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:12:11.500 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:12:11.500 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:11.500 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:12:11.500 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:11.500 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:12:11.500 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:12:11.500 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:11.500 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:11.500 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:11.500 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:11.500 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:11.500 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:11.501 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:11.501 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:11.501 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:11.501 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:11.501 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:11.501 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:11.501 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:12:11.501 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:12:11.501 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:11.501 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:11.501 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:11.501 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:11.501 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:11.501 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:11.502 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:11.502 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:11.502 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:11.502 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:11.502 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:11.502 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.502 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.502 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:12:11.502 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:12:11.502 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:12:11.502 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:12:11.502 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:12:11.502 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:12:11.502 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:11.503 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:11.503 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:12:11.503 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:12:11.503 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:12:11.503 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:12:11.503 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:12:11.503 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:12:11.503 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:12:11.503 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:11.503 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:12:11.503 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:11.503 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:11.503 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:11.503 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:11.504 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:11.504 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:11.504 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:11.504 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:11.504 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:11.504 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:11.504 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:11.504 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:11.504 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:12:11.504 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.504 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:12:11.504 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.504 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:12:11.505 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:12:11.505 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:11.505 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:11.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.505 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:11.505 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:12:11.505 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.505 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:11.505 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:12:11.505 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.505 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.506 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.506 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:12:11.506 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:12:11.506 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:11.506 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:11.506 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:11.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.506 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:12:11.506 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:12:11.506 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:11.506 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:11.507 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.507 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:11.507 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:12:11.507 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.507 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.507 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:11.507 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:12:11.507 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.507 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.507 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.507 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:12:11.507 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:12:11.507 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:11.508 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:11.508 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:11.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.508 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.508 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:12:11.508 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:12:11.508 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:11.508 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:11.508 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.508 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:11.508 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:12:11.508 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.509 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.509 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:11.509 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:12:11.509 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.509 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.509 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.509 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.509 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:12:11.509 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:12:11.509 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:11.509 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:11.509 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:11.509 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.510 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.510 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.510 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:12:11.510 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:12:11.510 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.510 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:11.510 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:11.510 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:12:11.510 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.510 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:11.510 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:12:11.510 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.511 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:11.511 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:11.511 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:12:11.511 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.511 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:11.511 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:12:11.511 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.511 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:11.511 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:11.511 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 6 2024-09-16T05:12:11.511 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.511 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:11.511 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:12:11.511 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.512 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:11.512 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:11.512 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:12:11.512 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.512 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.512 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.512 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:12:11.512 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:12:11.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 10 librados: watch_flush enter 2024-09-16T05:12:11.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 10 librados: watch_flush exit 2024-09-16T05:12:11.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 20 client.27984.objecter shutdown clearing up homeless session... 2024-09-16T05:12:11.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 10 client.27984.objecter successfully canceled tick 2024-09-16T05:12:11.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 1 -- 172.21.15.114:0/1715038 >> v2:172.21.15.149:6812/1185531135 conn(0x7f23f8037810 msgr2=0x7f23f8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 1 --2- 172.21.15.114:0/1715038 >> v2:172.21.15.149:6812/1185531135 conn(0x7f23f8037810 0x7f23f8039cd0 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 crypto rx=0x7f240c0930a0 tx=0x7f2404009040 comp rx=0 tx=0).stop 2024-09-16T05:12:11.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 10 monclient: shutdown 2024-09-16T05:12:11.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:11.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 1 -- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f240c09e000 msgr2=0x7f240c139010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 1 --2- 172.21.15.114:0/1715038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f240c09e000 0x7f240c139010 secure :-1 s=READY pgs=8308 cs=0 l=1 rev1=1 crypto rx=0x7f23fc019000 tx=0x7f23fc019190 comp rx=0 tx=0).stop 2024-09-16T05:12:11.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2411065640 1 -- 172.21.15.114:0/1715038 reap_dead start 2024-09-16T05:12:11.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2411065640 1 -- 172.21.15.114:0/1715038 reap_dead start 2024-09-16T05:12:11.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 1 -- 172.21.15.114:0/1715038 shutdown_connections 2024-09-16T05:12:11.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 1 -- 172.21.15.114:0/1715038 >> 172.21.15.114:0/1715038 conn(0x7f240c0a1fb0 msgr2=0x7f240c09c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:11.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 1 -- 172.21.15.114:0/1715038 shutdown_connections 2024-09-16T05:12:11.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 1 -- 172.21.15.114:0/1715038 wait complete. 2024-09-16T05:12:11.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.477+0000 7f2412ac8640 1 librados: shutdown 2024-09-16T05:12:11.515 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:12:11.515 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:12:11.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:12:11.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 10 monclient: get_monmap_and_config 2024-09-16T05:12:11.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 10 monclient: build_initial_monmap 2024-09-16T05:12:11.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 10 monclient: monmap: 2024-09-16T05:12:11.629 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:11.629 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.629 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:12:11.628938+0000 2024-09-16T05:12:11.629 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:12:11.628938+0000 2024-09-16T05:12:11.629 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:11.629 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.629 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:12:11.629 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:12:11.629 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:12:11.630 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:12:11.630 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:12:11.630 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 1 Processor -- start 2024-09-16T05:12:11.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 1 -- start start 2024-09-16T05:12:11.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 10 monclient: init 2024-09-16T05:12:11.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:11.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-16T05:12:11.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f696409b6c0 addr v1:172.21.15.114:6789/0 2024-09-16T05:12:11.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f69640a15d0 0x7f696413a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 10 monclient(hunting): picked mon.noname-c con 0x7f69640a15d0 addr v2:172.21.15.149:3300/0 2024-09-16T05:12:11.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f69640a19e0 0x7f696413cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 10 monclient(hunting): picked mon.noname-a con 0x7f69640a19e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:12:11.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696409d4e0 con 0x7f696409b6c0 2024-09-16T05:12:11.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f696409e390 con 0x7f69640a19e0 2024-09-16T05:12:11.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f696409ece0 con 0x7f69640a15d0 2024-09-16T05:12:11.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:11.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6969aea640 10 monclient(hunting): authenticate will time out at 1216.258167s 2024-09-16T05:12:11.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f69637fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f696409b6c0 0x7f69640a0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38534/0 (socket says 172.21.15.114:38534) 2024-09-16T05:12:11.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f69637fe640 1 -- 172.21.15.114:0/4048036423 learned_addr learned my addr 172.21.15.114:0/4048036423 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:12:11.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 1 --2- 172.21.15.114:0/4048036423 >> v2:172.21.15.114:3300/0 conn(0x7f69640a19e0 0x7f696413cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 10 monclient(hunting): get_auth_request con 0x7f69640a19e0 auth_method 0 2024-09-16T05:12:11.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6962ffd640 1 --2- 172.21.15.114:0/4048036423 >> v2:172.21.15.149:3300/0 conn(0x7f69640a15d0 0x7f696413a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6962ffd640 10 monclient(hunting): get_auth_request con 0x7f69640a15d0 auth_method 0 2024-09-16T05:12:11.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6962ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6962ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 10 monclient(hunting): handle_auth_done global_id 27987 payload 274 2024-09-16T05:12:11.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 1 -- 172.21.15.114:0/4048036423 >> v2:172.21.15.149:3300/0 conn(0x7f69640a15d0 msgr2=0x7f696413a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:11.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 1 --2- 172.21.15.114:0/4048036423 >> v2:172.21.15.149:3300/0 conn(0x7f69640a15d0 0x7f696413a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 1 -- 172.21.15.114:0/4048036423 >> v1:172.21.15.114:6789/0 conn(0x7f696409b6c0 legacy=0x7f69640a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 10 monclient: _finish_hunting 0 2024-09-16T05:12:11.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 1 monclient: found mon.noname-a 2024-09-16T05:12:11.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:11.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f69627fc640 1 -- 172.21.15.114:0/4048036423 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 865908804 0 0) 0x7f696409d4e0 con 0x7f696409b6c0 2024-09-16T05:12:11.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:12:11.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f6963fff640 1 -- 172.21.15.114:0/4048036423 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696409f430 con 0x7f69640a19e0 2024-09-16T05:12:11.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.625+0000 7f69627fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:12:11.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6963fff640 1 --2- 172.21.15.114:0/4048036423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69640a19e0 0x7f696413cba0 secure :-1 s=READY pgs=8310 cs=0 l=1 rev1=1 crypto rx=0x7f696409c630 tx=0x7f695c01c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f89569aa50624df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69627fc640 1 -- 172.21.15.114:0/4048036423 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c0b2070 con 0x7f69640a19e0 2024-09-16T05:12:11.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:12:11.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69627fc640 10 monclient: dump: 2024-09-16T05:12:11.642 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.642 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.643 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.643 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.643 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.643 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.643 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.643 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.643 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.643 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69627fc640 10 monclient: _finish_auth 0 2024-09-16T05:12:11.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69627fc640 10 monclient: _check_auth_tickets 2024-09-16T05:12:11.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:11.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 5 monclient: authenticate success, global_id 27987 2024-09-16T05:12:11.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:12:11.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69627fc640 1 -- 172.21.15.114:0/4048036423 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f695c0a5830 con 0x7f69640a19e0 2024-09-16T05:12:11.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69627fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:11.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient: get_monmap_and_config success 2024-09-16T05:12:11.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 4 set_mon_vals no callback set 2024-09-16T05:12:11.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69627fc640 1 -- 172.21.15.114:0/4048036423 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c0a5c00 con 0x7f69640a19e0 2024-09-16T05:12:11.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:12:11.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69627fc640 10 monclient: dump: 2024-09-16T05:12:11.644 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.644 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.644 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.644 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.644 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.644 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.645 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.645 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.645 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.645 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient: shutdown 2024-09-16T05:12:11.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:11.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 -- 172.21.15.114:0/4048036423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69640a19e0 msgr2=0x7f696413cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 --2- 172.21.15.114:0/4048036423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69640a19e0 0x7f696413cba0 secure :-1 s=READY pgs=8310 cs=0 l=1 rev1=1 crypto rx=0x7f696409c630 tx=0x7f695c01c880 comp rx=0 tx=0).stop 2024-09-16T05:12:11.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 -- 172.21.15.114:0/4048036423 shutdown_connections 2024-09-16T05:12:11.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 --2- 172.21.15.114:0/4048036423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69640a19e0 0x7f696413cba0 unknown :-1 s=CLOSED pgs=8310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 --2- 172.21.15.114:0/4048036423 >> v2:172.21.15.149:3300/0 conn(0x7f69640a15d0 0x7f696413a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 -- 172.21.15.114:0/4048036423 >> 172.21.15.114:0/4048036423 conn(0x7f69640a1fb0 msgr2=0x7f69640a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:11.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 -- 172.21.15.114:0/4048036423 shutdown_connections 2024-09-16T05:12:11.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 -- 172.21.15.114:0/4048036423 wait complete. 2024-09-16T05:12:11.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient: build_initial_monmap 2024-09-16T05:12:11.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient: monmap: 2024-09-16T05:12:11.646 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:11.646 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:12:11.646 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:12:11.646 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:12:11.646 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:11.646 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.646 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:12:11.646 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:12:11.646 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:12:11.646 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 librados: starting msgr at 2024-09-16T05:12:11.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 librados: starting objecter 2024-09-16T05:12:11.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 Processor -- start 2024-09-16T05:12:11.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 -- start start 2024-09-16T05:12:11.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 librados: setting wanted keys 2024-09-16T05:12:11.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 librados: calling monclient init 2024-09-16T05:12:11.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient: init 2024-09-16T05:12:11.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:11.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:12:11.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f696409b6c0 0x7f6964139df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient(hunting): picked mon.noname-c con 0x7f696409b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:11.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69640a15d0 0x7f696413a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient(hunting): picked mon.noname-a con 0x7f69640a15d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:11.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69640a19e0 0x7f696413a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient(hunting): picked mon.noname-b con 0x7f69640a19e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:11.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f696413f900 con 0x7f69640a15d0 2024-09-16T05:12:11.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f696409cad0 con 0x7f69640a19e0 2024-09-16T05:12:11.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f696413fa80 con 0x7f696409b6c0 2024-09-16T05:12:11.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:11.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient(hunting): authenticate will time out at 1216.263082s 2024-09-16T05:12:11.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69640a15d0 0x7f696413a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69637fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f696409b6c0 0x7f6964139df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69637fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f696409b6c0 0x7f6964139df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:59172/0 (socket says 172.21.15.114:59172) 2024-09-16T05:12:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69640a15d0 0x7f696413a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36382/0 (socket says 172.21.15.114:36382) 2024-09-16T05:12:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69637fe640 1 -- 172.21.15.114:0/2939706599 learned_addr learned my addr 172.21.15.114:0/2939706599 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69637fe640 10 monclient(hunting): get_auth_request con 0x7f696409b6c0 auth_method 0 2024-09-16T05:12:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 10 monclient(hunting): get_auth_request con 0x7f69640a15d0 auth_method 0 2024-09-16T05:12:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6963fff640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69640a19e0 0x7f696413a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6963fff640 10 monclient(hunting): get_auth_request con 0x7f69640a19e0 auth_method 0 2024-09-16T05:12:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6963fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6963fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6963fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 10 monclient(hunting): handle_auth_done global_id 27993 payload 824 2024-09-16T05:12:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 1 -- 172.21.15.114:0/2939706599 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f696409b6c0 msgr2=0x7f6964139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f696409b6c0 0x7f6964139df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 1 -- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69640a19e0 msgr2=0x7f696413a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69640a19e0 0x7f696413a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:12:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 1 monclient: found mon.noname-a 2024-09-16T05:12:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:12:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 1 -- 172.21.15.114:0/2939706599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696412af80 con 0x7f69640a15d0 2024-09-16T05:12:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f69637fe640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f696409b6c0 0x7f6964139df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:12:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6962ffd640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69640a15d0 0x7f696413a330 secure :-1 s=READY pgs=8311 cs=0 l=1 rev1=1 crypto rx=0x7f695400d5a0 tx=0x7f695400da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f694bfff640 10 client.?.objecter ms_handle_connect 0x7f69640a15d0 2024-09-16T05:12:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f694bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:12:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f694bfff640 1 -- 172.21.15.114:0/2939706599 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6954005bb0 con 0x7f69640a15d0 2024-09-16T05:12:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f694bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f694bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:12:11.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f694bfff640 10 monclient: dump: 2024-09-16T05:12:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.652 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f694bfff640 10 monclient: _finish_auth 0 2024-09-16T05:12:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f694bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:12:11.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f694bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f694bfff640 1 -- 172.21.15.114:0/2939706599 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6954018070 con 0x7f69640a15d0 2024-09-16T05:12:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 5 monclient: authenticate success, global_id 27993 2024-09-16T05:12:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient: _renew_subs 2024-09-16T05:12:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:12:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6969aea640 1 -- 172.21.15.114:0/2939706599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f696412b260 con 0x7f69640a15d0 2024-09-16T05:12:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f694bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f694bfff640 1 -- 172.21.15.114:0/2939706599 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6954006050 con 0x7f69640a15d0 2024-09-16T05:12:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f694bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f694bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:12:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f694bfff640 10 monclient: dump: 2024-09-16T05:12:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.653 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.654 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f6969aea640 10 client.27993.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:12:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f6969aea640 10 monclient: _renew_subs 2024-09-16T05:12:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.629+0000 7f6961ffb640 4 set_mon_vals no callback set 2024-09-16T05:12:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f6969aea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:12:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f6969aea640 1 -- 172.21.15.114:0/2939706599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f696413b780 con 0x7f69640a15d0 2024-09-16T05:12:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f6969aea640 1 librados: init done 2024-09-16T05:12:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f6969aea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:11.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f6969aea640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69641ccb20 0x7f69641ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f6969aea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f6969aea640 1 -- 172.21.15.114:0/2939706599 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f6964069390 con 0x7f69641ccb20 2024-09-16T05:12:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 1 -- 172.21.15.114:0/2939706599 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f69540061f0 con 0x7f69640a15d0 2024-09-16T05:12:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 1 --2- 172.21.15.114:0/2939706599 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6950037810 0x7f6950039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f69637fe640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69641ccb20 0x7f69641ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f69637fe640 10 monclient: get_auth_request con 0x7f69641ccb20 auth_method 0 2024-09-16T05:12:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f69637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f69637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f69637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 1 -- 172.21.15.114:0/2939706599 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 6231+0+0 (secure 0 0 0) 0x7f6954049170 con 0x7f69640a15d0 2024-09-16T05:12:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 10 client.27993.objecter ms_dispatch 0x7f6964000ba0 osd_map(177..177 src has 1..177) 2024-09-16T05:12:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 3 client.27993.objecter handle_osd_map got epochs [177,177] > 0 2024-09-16T05:12:11.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 3 client.27993.objecter handle_osd_map decoding full epoch 177 2024-09-16T05:12:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 20 client.27993.objecter dump_active .. 0 homeless 2024-09-16T05:12:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f69637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f69637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f69637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f6963fff640 1 --2- 172.21.15.114:0/2939706599 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6950037810 0x7f6950039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f6963fff640 10 monclient: get_auth_request con 0x7f6950037810 auth_method 0 2024-09-16T05:12:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f69637fe640 10 monclient(hunting): handle_auth_done global_id 64111 payload 824 2024-09-16T05:12:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f69637fe640 1 -- 172.21.15.114:0/2939706599 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f69640a0ec0 con 0x7f69641ccb20 2024-09-16T05:12:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 10 client.27993.objecter ms_handle_connect 0x7f69641ccb20 2024-09-16T05:12:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 10 client.27993.objecter resend_mon_ops 2024-09-16T05:12:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f69637fe640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69641ccb20 0x7f69641ccf00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f69580018c0 tx=0x7f6958001cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 10 client.27993.objecter ms_handle_connect 0x7f6950037810 2024-09-16T05:12:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f6963fff640 1 --2- 172.21.15.114:0/2939706599 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6950037810 0x7f6950039cd0 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7f695c0af000 tx=0x7f695c0a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 1 -- 172.21.15.114:0/2939706599 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695800fa90 con 0x7f69641ccb20 2024-09-16T05:12:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:12:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 10 monclient: dump: 2024-09-16T05:12:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.657 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 1 -- 172.21.15.114:0/2939706599 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6958018070 con 0x7f69641ccb20 2024-09-16T05:12:11.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:11.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:12:11.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 1 -- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69641ccb20 msgr2=0x7f69641ccf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69641ccb20 0x7f69641ccf00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f69580018c0 tx=0x7f6958001cc0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 1 -- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69641ccb20 msgr2=0x7f69641ccf00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:11.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.633+0000 7f694bfff640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69641ccb20 0x7f69641ccf00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:12:11.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69641cd440 0x7f6964070780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 1 -- 172.21.15.114:0/2939706599 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f696412aae0 con 0x7f69641cd440 2024-09-16T05:12:11.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f69637fe640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69641cd440 0x7f6964070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f69637fe640 10 monclient: get_auth_request con 0x7f69641cd440 auth_method 0 2024-09-16T05:12:11.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f69637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f69637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f69637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f69637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f69637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f69637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f69637fe640 10 monclient(hunting): handle_auth_done global_id 64114 payload 824 2024-09-16T05:12:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f69637fe640 1 -- 172.21.15.114:0/2939706599 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f69640686f0 con 0x7f69641cd440 2024-09-16T05:12:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f694bfff640 10 client.27993.objecter ms_handle_connect 0x7f69641cd440 2024-09-16T05:12:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f694bfff640 10 client.27993.objecter resend_mon_ops 2024-09-16T05:12:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f69637fe640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69641cd440 0x7f6964070780 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f69580018c0 tx=0x7f6958001090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f694bfff640 1 -- 172.21.15.114:0/2939706599 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69580138d0 con 0x7f69641cd440 2024-09-16T05:12:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f694bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f694bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:12:11.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f694bfff640 10 monclient: dump: 2024-09-16T05:12:11.662 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.662 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.662 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.662 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.662 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.662 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.662 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.662 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.662 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.662 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f694bfff640 1 -- 172.21.15.114:0/2939706599 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f69580138d0 con 0x7f69641cd440 2024-09-16T05:12:11.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f694bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:12:11.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f694bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:12:11.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f694bfff640 1 -- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69641cd440 msgr2=0x7f6964070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f694bfff640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69641cd440 0x7f6964070780 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f69580018c0 tx=0x7f6958001090 comp rx=0 tx=0).stop 2024-09-16T05:12:11.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f694bfff640 1 -- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69641cd440 msgr2=0x7f6964070780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:11.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f694bfff640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69641cd440 0x7f6964070780 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f69580018c0 tx=0x7f6958001090 comp rx=0 tx=0).stop 2024-09-16T05:12:11.663 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:12:11.664 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:12:11.664 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:12:11.664 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:12:11.664 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 20, 2024-09-16T05:12:11.664 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 15779, 2024-09-16T05:12:11.664 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:12:11.664 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:12:11.664 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:12:11.664 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:12:11.664 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:11.664 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:12:11.665 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:11.665 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:12:11.665 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:12:11.665 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:11.665 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:11.665 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:11.665 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:11.665 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:11.665 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:11.665 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:11.665 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:11.665 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:11.665 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:11.665 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:12:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:12:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:12:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:12:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:12:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:12:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:12:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:12:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:12:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:12:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:12:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:12:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:12:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:12:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:12:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:12:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:12:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:12:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:12:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:12:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:12:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:12:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:12:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:12:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:12:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:12:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:12:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:12:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:12:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:12:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:12:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:12:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:12:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:12:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:12:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:12:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:12:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:12:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:12:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:12:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:12:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:11.676 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:11.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:11.676 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:12:11.676 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:12:11.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 10 librados: watch_flush enter 2024-09-16T05:12:11.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 10 librados: watch_flush exit 2024-09-16T05:12:11.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 20 client.27993.objecter shutdown clearing up homeless session... 2024-09-16T05:12:11.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 10 client.27993.objecter successfully canceled tick 2024-09-16T05:12:11.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 1 -- 172.21.15.114:0/2939706599 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6950037810 msgr2=0x7f6950039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 1 --2- 172.21.15.114:0/2939706599 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6950037810 0x7f6950039cd0 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7f695c0af000 tx=0x7f695c0a7070 comp rx=0 tx=0).stop 2024-09-16T05:12:11.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 10 monclient: shutdown 2024-09-16T05:12:11.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:11.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 1 -- 172.21.15.114:0/2939706599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69640a15d0 msgr2=0x7f696413a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 1 --2- 172.21.15.114:0/2939706599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69640a15d0 0x7f696413a330 secure :-1 s=READY pgs=8311 cs=0 l=1 rev1=1 crypto rx=0x7f695400d5a0 tx=0x7f695400da60 comp rx=0 tx=0).stop 2024-09-16T05:12:11.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6963fff640 1 -- 172.21.15.114:0/2939706599 reap_dead start 2024-09-16T05:12:11.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6963fff640 1 -- 172.21.15.114:0/2939706599 reap_dead start 2024-09-16T05:12:11.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 1 -- 172.21.15.114:0/2939706599 shutdown_connections 2024-09-16T05:12:11.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 1 -- 172.21.15.114:0/2939706599 >> 172.21.15.114:0/2939706599 conn(0x7f69640a1fb0 msgr2=0x7f69640a4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:11.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 1 -- 172.21.15.114:0/2939706599 shutdown_connections 2024-09-16T05:12:11.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 1 -- 172.21.15.114:0/2939706599 wait complete. 2024-09-16T05:12:11.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.641+0000 7f6969aea640 1 librados: shutdown 2024-09-16T05:12:11.679 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:12:11.679 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:12:11.679 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:12:11.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.785+0000 7f8053085640 10 monclient: get_monmap_and_config 2024-09-16T05:12:11.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.785+0000 7f8053085640 10 monclient: build_initial_monmap 2024-09-16T05:12:11.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.785+0000 7f8053085640 10 monclient: monmap: 2024-09-16T05:12:11.791 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:11.791 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.792 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:12:11.791245+0000 2024-09-16T05:12:11.792 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:12:11.791245+0000 2024-09-16T05:12:11.792 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:11.792 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.792 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:12:11.792 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:12:11.792 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:12:11.792 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:12:11.792 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:12:11.792 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.785+0000 7f8053085640 1 Processor -- start 2024-09-16T05:12:11.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.785+0000 7f8053085640 1 -- start start 2024-09-16T05:12:11.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.785+0000 7f8053085640 10 monclient: init 2024-09-16T05:12:11.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:11.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-16T05:12:11.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f804c09b6c0 0x7f804c0a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 10 monclient(hunting): picked mon.noname-c con 0x7f804c09b6c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:12:11.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804c0a1400 0x7f804c0a1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 10 monclient(hunting): picked mon.noname-b con 0x7f804c0a1400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:11.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f804c139640 0x7f804c13ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 10 monclient(hunting): picked mon.noname-a con 0x7f804c139640 addr v2:172.21.15.114:3300/0 2024-09-16T05:12:11.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f804c09e390 con 0x7f804c139640 2024-09-16T05:12:11.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f804c0a1d50 con 0x7f804c09b6c0 2024-09-16T05:12:11.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f804c09ece0 con 0x7f804c0a1400 2024-09-16T05:12:11.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:11.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 10 monclient(hunting): authenticate will time out at 1216.420467s 2024-09-16T05:12:11.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804c0a1400 0x7f804c0a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f804c139640 0x7f804c13ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804c0a1400 0x7f804c0a1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:59176/0 (socket says 172.21.15.114:59176) 2024-09-16T05:12:11.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804bfff640 1 -- 172.21.15.114:0/3098082261 learned_addr learned my addr 172.21.15.114:0/3098082261 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:11.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 10 monclient(hunting): get_auth_request con 0x7f804c139640 auth_method 0 2024-09-16T05:12:11.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804bfff640 10 monclient(hunting): get_auth_request con 0x7f804c0a1400 auth_method 0 2024-09-16T05:12:11.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8050e21640 1 --2- 172.21.15.114:0/3098082261 >> v2:172.21.15.149:3300/0 conn(0x7f804c09b6c0 0x7f804c0a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8050e21640 10 monclient(hunting): get_auth_request con 0x7f804c09b6c0 auth_method 0 2024-09-16T05:12:11.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8050e21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8050e21640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8050e21640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 10 monclient(hunting): handle_auth_done global_id 27996 payload 274 2024-09-16T05:12:11.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 1 -- 172.21.15.114:0/3098082261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804c0a1400 msgr2=0x7f804c0a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:11.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 1 --2- 172.21.15.114:0/3098082261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804c0a1400 0x7f804c0a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 1 -- 172.21.15.114:0/3098082261 >> v2:172.21.15.149:3300/0 conn(0x7f804c09b6c0 msgr2=0x7f804c0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:11.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 1 --2- 172.21.15.114:0/3098082261 >> v2:172.21.15.149:3300/0 conn(0x7f804c09b6c0 0x7f804c0a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 10 monclient: _finish_hunting 0 2024-09-16T05:12:11.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 1 monclient: found mon.noname-a 2024-09-16T05:12:11.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:11.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:12:11.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 1 -- 172.21.15.114:0/3098082261 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f804c09d4e0 con 0x7f804c139640 2024-09-16T05:12:11.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804bfff640 1 --2- 172.21.15.114:0/3098082261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804c0a1400 0x7f804c0a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:12:11.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8051622640 1 --2- 172.21.15.114:0/3098082261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804c139640 0x7f804c13ba30 secure :-1 s=READY pgs=8312 cs=0 l=1 rev1=1 crypto rx=0x7f804c0a2f70 tx=0x7f804401c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aed63514273d485 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804b7fe640 1 -- 172.21.15.114:0/3098082261 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80440ac070 con 0x7f804c139640 2024-09-16T05:12:11.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:12:11.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804b7fe640 10 monclient: dump: 2024-09-16T05:12:11.804 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.804 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.804 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.804 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.804 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.804 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.804 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.804 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.804 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.804 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:12:11.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:12:11.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:11.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804b7fe640 1 -- 172.21.15.114:0/3098082261 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80440a6680 con 0x7f804c139640 2024-09-16T05:12:11.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:11.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804b7fe640 1 -- 172.21.15.114:0/3098082261 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80440a6a50 con 0x7f804c139640 2024-09-16T05:12:11.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 5 monclient: authenticate success, global_id 27996 2024-09-16T05:12:11.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 10 monclient: get_monmap_and_config success 2024-09-16T05:12:11.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 4 set_mon_vals no callback set 2024-09-16T05:12:11.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:12:11.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f804b7fe640 10 monclient: dump: 2024-09-16T05:12:11.805 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.806 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.806 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.806 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.806 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.806 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.806 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.806 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.806 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.806 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 10 monclient: shutdown 2024-09-16T05:12:11.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:11.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.789+0000 7f8053085640 1 -- 172.21.15.114:0/3098082261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804c139640 msgr2=0x7f804c13ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 --2- 172.21.15.114:0/3098082261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804c139640 0x7f804c13ba30 secure :-1 s=READY pgs=8312 cs=0 l=1 rev1=1 crypto rx=0x7f804c0a2f70 tx=0x7f804401c7f0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 -- 172.21.15.114:0/3098082261 shutdown_connections 2024-09-16T05:12:11.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 --2- 172.21.15.114:0/3098082261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804c139640 0x7f804c13ba30 unknown :-1 s=CLOSED pgs=8312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 --2- 172.21.15.114:0/3098082261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804c0a1400 0x7f804c0a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 --2- 172.21.15.114:0/3098082261 >> v2:172.21.15.149:3300/0 conn(0x7f804c09b6c0 0x7f804c0a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 -- 172.21.15.114:0/3098082261 >> 172.21.15.114:0/3098082261 conn(0x7f804c0a1fb0 msgr2=0x7f804c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:11.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 -- 172.21.15.114:0/3098082261 shutdown_connections 2024-09-16T05:12:11.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 -- 172.21.15.114:0/3098082261 wait complete. 2024-09-16T05:12:11.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient: build_initial_monmap 2024-09-16T05:12:11.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient: monmap: 2024-09-16T05:12:11.807 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:11.807 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:12:11.807 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:12:11.807 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:12:11.808 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:11.808 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.808 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:12:11.808 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:12:11.808 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:12:11.808 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 librados: starting msgr at 2024-09-16T05:12:11.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 librados: starting objecter 2024-09-16T05:12:11.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 Processor -- start 2024-09-16T05:12:11.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 -- start start 2024-09-16T05:12:11.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 librados: setting wanted keys 2024-09-16T05:12:11.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 librados: calling monclient init 2024-09-16T05:12:11.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient: init 2024-09-16T05:12:11.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:11.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:12:11.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f804c09b6c0 0x7f804c098b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient(hunting): picked mon.noname-b con 0x7f804c09b6c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:11.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804c0a1400 0x7f804c08e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient(hunting): picked mon.noname-c con 0x7f804c0a1400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:11.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804c139640 0x7f804c08ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient(hunting): picked mon.noname-a con 0x7f804c139640 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:11.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f804c13e790 con 0x7f804c139640 2024-09-16T05:12:11.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f804c09cad0 con 0x7f804c09b6c0 2024-09-16T05:12:11.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:11.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f804c13e910 con 0x7f804c0a1400 2024-09-16T05:12:11.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:11.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient(hunting): authenticate will time out at 1216.425858s 2024-09-16T05:12:11.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804c139640 0x7f804c08ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f804bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804c0a1400 0x7f804c08e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804c139640 0x7f804c08ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36390/0 (socket says 172.21.15.114:36390) 2024-09-16T05:12:11.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 1 -- 172.21.15.114:0/974031590 learned_addr learned my addr 172.21.15.114:0/974031590 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:11.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 10 monclient(hunting): get_auth_request con 0x7f804c139640 auth_method 0 2024-09-16T05:12:11.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f804bfff640 10 monclient(hunting): get_auth_request con 0x7f804c0a1400 auth_method 0 2024-09-16T05:12:11.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f804bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:11.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f804bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f804bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f804bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f804bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f804bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8050e21640 1 --2- 172.21.15.114:0/974031590 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f804c09b6c0 0x7f804c098b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 10 monclient(hunting): handle_auth_done global_id 27999 payload 824 2024-09-16T05:12:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 1 -- 172.21.15.114:0/974031590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804c0a1400 msgr2=0x7f804c08e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 1 --2- 172.21.15.114:0/974031590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804c0a1400 0x7f804c08e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 1 -- 172.21.15.114:0/974031590 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f804c09b6c0 msgr2=0x7f804c098b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 1 --2- 172.21.15.114:0/974031590 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f804c09b6c0 0x7f804c098b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 10 monclient: _finish_hunting 0 2024-09-16T05:12:11.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 1 monclient: found mon.noname-a 2024-09-16T05:12:11.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:11.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:12:11.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 1 -- 172.21.15.114:0/974031590 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f804c139cd0 con 0x7f804c139640 2024-09-16T05:12:11.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8050e21640 10 monclient: get_auth_request con 0x7f804c09b6c0 auth_method 0 2024-09-16T05:12:11.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8050e21640 1 --2- 172.21.15.114:0/974031590 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f804c09b6c0 0x7f804c098b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:12:11.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8051622640 1 --2- 172.21.15.114:0/974031590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804c139640 0x7f804c08ecc0 secure :-1 s=READY pgs=8313 cs=0 l=1 rev1=1 crypto rx=0x7f8044009d90 tx=0x7f8044019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 10 client.?.objecter ms_handle_connect 0x7f804c139640 2024-09-16T05:12:11.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:12:11.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 1 -- 172.21.15.114:0/974031590 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80440c0070 con 0x7f804c139640 2024-09-16T05:12:11.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:12:11.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 10 monclient: dump: 2024-09-16T05:12:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.813 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 10 monclient: _finish_auth 0 2024-09-16T05:12:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 10 monclient: _check_auth_tickets 2024-09-16T05:12:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 1 -- 172.21.15.114:0/974031590 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80440a6480 con 0x7f804c139640 2024-09-16T05:12:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 1 -- 172.21.15.114:0/974031590 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80440ac050 con 0x7f804c139640 2024-09-16T05:12:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 5 monclient: authenticate success, global_id 27999 2024-09-16T05:12:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:12:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 10 monclient: dump: 2024-09-16T05:12:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:11.815 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:11.815 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:11.815 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:11.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient: _renew_subs 2024-09-16T05:12:11.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f804affd640 4 set_mon_vals no callback set 2024-09-16T05:12:11.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:12:11.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 -- 172.21.15.114:0/974031590 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f804c0906a0 con 0x7f804c139640 2024-09-16T05:12:11.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 client.27999.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:12:11.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient: _renew_subs 2024-09-16T05:12:11.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:12:11.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 -- 172.21.15.114:0/974031590 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f804c0924f0 con 0x7f804c139640 2024-09-16T05:12:11.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 librados: init done 2024-09-16T05:12:11.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:11.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:11.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:12:11.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f8053085640 1 -- 172.21.15.114:0/974031590 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f804c0720c0 con 0x7f804c139640 2024-09-16T05:12:11.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 1 -- 172.21.15.114:0/974031590 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f80440c3070 con 0x7f804c139640 2024-09-16T05:12:11.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.793+0000 7f80497fa640 1 --2- 172.21.15.114:0/974031590 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8038037810 0x7f8038039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:11.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.797+0000 7f80497fa640 1 -- 172.21.15.114:0/974031590 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 6231+0+0 (secure 0 0 0) 0x7f80440e3a80 con 0x7f804c139640 2024-09-16T05:12:11.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.797+0000 7f80497fa640 10 client.27999.objecter ms_dispatch 0x7f804c000ba0 osd_map(177..177 src has 1..177) 2024-09-16T05:12:11.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.797+0000 7f80497fa640 3 client.27999.objecter handle_osd_map got epochs [177,177] > 0 2024-09-16T05:12:11.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.797+0000 7f80497fa640 3 client.27999.objecter handle_osd_map decoding full epoch 177 2024-09-16T05:12:11.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.797+0000 7f80497fa640 20 client.27999.objecter dump_active .. 0 homeless 2024-09-16T05:12:11.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.797+0000 7f8050e21640 1 --2- 172.21.15.114:0/974031590 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8038037810 0x7f8038039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:11.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.797+0000 7f8050e21640 10 monclient: get_auth_request con 0x7f8038037810 auth_method 0 2024-09-16T05:12:11.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.797+0000 7f80497fa640 10 client.27999.objecter ms_handle_connect 0x7f8038037810 2024-09-16T05:12:11.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.797+0000 7f8050e21640 1 --2- 172.21.15.114:0/974031590 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8038037810 0x7f8038039cd0 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7f80400097a0 tx=0x7f8040008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:11.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.801+0000 7f80497fa640 1 -- 172.21.15.114:0/974031590 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f804c139cd0 con 0x7f804c139640 2024-09-16T05:12:11.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.801+0000 7f80497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:11.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:11.801+0000 7f80497fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:12:12.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.029+0000 7f8053085640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:12:12.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.029+0000 7f8053085640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:12:12.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.029+0000 7f8053085640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:12:12.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.029+0000 7f8053085640 1 -- 172.21.15.114:0/974031590 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f804c0a0e40 con 0x7f804c139640 2024-09-16T05:12:12.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.033+0000 7f80497fa640 1 -- 172.21.15.114:0/974031590 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f8044016500 con 0x7f804c139640 2024-09-16T05:12:12.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.033+0000 7f80497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:12:12.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.033+0000 7f80497fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:12:12.038 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.033+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.038 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.033+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=1,health=11,kv=12,logm=64} crc {auth=2472553844,config=1747058506,health=2912582278,kv=2757161437,logm=853256401}) 2024-09-16T05:12:12.038 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.033+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=12,kv=11,logm=63} crc {auth=2080234965,config=2586702516,health=3824680933,kv=3030643143,logm=3263316642}) 2024-09-16T05:12:12.039 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.033+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.039 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.033+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=1,health=11,kv=12,logm=64} crc {auth=2472553844,config=1747058506,health=2912582278,kv=2757161437,logm=853256401}) 2024-09-16T05:12:12.039 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.033+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=1,health=12,kv=12,logm=64} crc {auth=926094473,config=1228700967,health=3824680933,kv=2757161437,logm=1340201899}) 2024-09-16T05:12:12.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.033+0000 7f8053085640 10 librados: watch_flush enter 2024-09-16T05:12:12.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.033+0000 7f8053085640 10 librados: watch_flush exit 2024-09-16T05:12:12.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.033+0000 7f8053085640 20 client.27999.objecter shutdown clearing up homeless session... 2024-09-16T05:12:12.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.033+0000 7f8053085640 10 client.27999.objecter successfully canceled tick 2024-09-16T05:12:12.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f8053085640 1 -- 172.21.15.114:0/974031590 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8038037810 msgr2=0x7f8038039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:12.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f8053085640 1 --2- 172.21.15.114:0/974031590 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8038037810 0x7f8038039cd0 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7f80400097a0 tx=0x7f8040008040 comp rx=0 tx=0).stop 2024-09-16T05:12:12.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f8053085640 10 monclient: shutdown 2024-09-16T05:12:12.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f8053085640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:12.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f8053085640 1 -- 172.21.15.114:0/974031590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804c139640 msgr2=0x7f804c08ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:12.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f8053085640 1 --2- 172.21.15.114:0/974031590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804c139640 0x7f804c08ecc0 secure :-1 s=READY pgs=8313 cs=0 l=1 rev1=1 crypto rx=0x7f8044009d90 tx=0x7f8044019590 comp rx=0 tx=0).stop 2024-09-16T05:12:12.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f8053085640 1 -- 172.21.15.114:0/974031590 shutdown_connections 2024-09-16T05:12:12.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f8053085640 1 --2- 172.21.15.114:0/974031590 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8038037810 0x7f8038039cd0 unknown :-1 s=CLOSED pgs=3455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:12.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f8053085640 1 --2- 172.21.15.114:0/974031590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804c139640 0x7f804c08ecc0 unknown :-1 s=CLOSED pgs=8313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:12.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f8053085640 1 --2- 172.21.15.114:0/974031590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804c0a1400 0x7f804c08e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:12.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f8053085640 1 --2- 172.21.15.114:0/974031590 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f804c09b6c0 0x7f804c098b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:12.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f8053085640 1 -- 172.21.15.114:0/974031590 >> 172.21.15.114:0/974031590 conn(0x7f804c0a1fb0 msgr2=0x7f804c138a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:12.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f8053085640 1 -- 172.21.15.114:0/974031590 shutdown_connections 2024-09-16T05:12:12.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f8053085640 1 -- 172.21.15.114:0/974031590 wait complete. 2024-09-16T05:12:12.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f8053085640 1 librados: shutdown 2024-09-16T05:12:12.042 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.042 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632969604}) 2024-09-16T05:12:12.043 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2111781774}) 2024-09-16T05:12:12.043 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.043 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632969604}) 2024-09-16T05:12:12.043 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3169915561}) 2024-09-16T05:12:12.043 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.043 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445830735}) 2024-09-16T05:12:12.043 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1726016814}) 2024-09-16T05:12:12.044 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.044 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445830735}) 2024-09-16T05:12:12.044 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.037+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3559656019}) 2024-09-16T05:12:12.044 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.041+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.045 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.041+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3793028289}) 2024-09-16T05:12:12.045 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.041+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3706944129}) 2024-09-16T05:12:12.045 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.041+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.045 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.041+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3793028289}) 2024-09-16T05:12:12.045 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.041+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=213843178}) 2024-09-16T05:12:12.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.041+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.041+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3081807395}) 2024-09-16T05:12:12.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.041+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588400315}) 2024-09-16T05:12:12.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.041+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.047 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.041+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3081807395}) 2024-09-16T05:12:12.047 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.041+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3207396420}) 2024-09-16T05:12:12.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306078817}) 2024-09-16T05:12:12.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1895035098}) 2024-09-16T05:12:12.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306078817}) 2024-09-16T05:12:12.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2335453522}) 2024-09-16T05:12:12.049 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.050 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2681863680}) 2024-09-16T05:12:12.050 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1529133852}) 2024-09-16T05:12:12.050 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.050 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2681863680}) 2024-09-16T05:12:12.050 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2430414831}) 2024-09-16T05:12:12.051 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.051 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439911513}) 2024-09-16T05:12:12.051 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2867945442}) 2024-09-16T05:12:12.051 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.051 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439911513}) 2024-09-16T05:12:12.051 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.045+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=793507597}) 2024-09-16T05:12:12.053 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.049+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.053 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.049+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1316292290}) 2024-09-16T05:12:12.053 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.049+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2502479330}) 2024-09-16T05:12:12.053 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.049+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.053 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.049+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1316292290}) 2024-09-16T05:12:12.053 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.049+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228692219}) 2024-09-16T05:12:12.054 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.049+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.054 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.049+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412440720}) 2024-09-16T05:12:12.054 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.049+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3907390866}) 2024-09-16T05:12:12.054 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.049+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.055 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.049+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412440720}) 2024-09-16T05:12:12.055 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.049+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3929596984}) 2024-09-16T05:12:12.056 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.056 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3305306905}) 2024-09-16T05:12:12.056 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3001038493}) 2024-09-16T05:12:12.056 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.056 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3305306905}) 2024-09-16T05:12:12.056 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4264701797}) 2024-09-16T05:12:12.058 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:12:12.058 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.058 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4032047536}) 2024-09-16T05:12:12.058 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=199370444}) 2024-09-16T05:12:12.058 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.059 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4032047536}) 2024-09-16T05:12:12.059 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2154818418}) 2024-09-16T05:12:12.059 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.059 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184673211}) 2024-09-16T05:12:12.059 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1886887129}) 2024-09-16T05:12:12.059 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.060 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184673211}) 2024-09-16T05:12:12.060 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.053+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2576786958}) 2024-09-16T05:12:12.061 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.057+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.061 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.057+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=397557568}) 2024-09-16T05:12:12.061 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.057+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3784535873}) 2024-09-16T05:12:12.061 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.057+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.062 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.057+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=397557568}) 2024-09-16T05:12:12.062 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.057+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1860747194}) 2024-09-16T05:12:12.063 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.057+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.063 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.057+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=960378620}) 2024-09-16T05:12:12.063 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.057+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3253461110}) 2024-09-16T05:12:12.063 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.057+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.063 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.057+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=960378620}) 2024-09-16T05:12:12.063 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.057+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=19489418}) 2024-09-16T05:12:12.064 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.064 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1945765772}) 2024-09-16T05:12:12.064 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1150499473}) 2024-09-16T05:12:12.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1945765772}) 2024-09-16T05:12:12.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=956902096}) 2024-09-16T05:12:12.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2990773059}) 2024-09-16T05:12:12.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1050611482}) 2024-09-16T05:12:12.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2990773059}) 2024-09-16T05:12:12.067 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3019362780}) 2024-09-16T05:12:12.067 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.068 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3571600623}) 2024-09-16T05:12:12.069 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936331016}) 2024-09-16T05:12:12.069 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.069 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3571600623}) 2024-09-16T05:12:12.069 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.061+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1323621333}) 2024-09-16T05:12:12.070 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.065+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.070 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.065+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3776889716}) 2024-09-16T05:12:12.070 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.065+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=270707622}) 2024-09-16T05:12:12.070 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.065+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.070 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.065+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3776889716}) 2024-09-16T05:12:12.070 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.065+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3681148595}) 2024-09-16T05:12:12.071 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.065+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.071 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.065+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1452648531}) 2024-09-16T05:12:12.071 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.065+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3105176014}) 2024-09-16T05:12:12.071 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.065+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.071 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.065+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1452648531}) 2024-09-16T05:12:12.071 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.065+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=285695351}) 2024-09-16T05:12:12.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.069+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.069+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356904633}) 2024-09-16T05:12:12.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.069+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=548840671}) 2024-09-16T05:12:12.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.069+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.069+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356904633}) 2024-09-16T05:12:12.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.069+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4102210861}) 2024-09-16T05:12:12.074 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.069+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.074 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.069+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515472842}) 2024-09-16T05:12:12.074 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.069+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=139260341}) 2024-09-16T05:12:12.074 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.069+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.075 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.069+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515472842}) 2024-09-16T05:12:12.075 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.069+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=388286273}) 2024-09-16T05:12:12.076 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.073+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.076 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.073+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613692878}) 2024-09-16T05:12:12.076 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.073+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1755892824}) 2024-09-16T05:12:12.076 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.073+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.076 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.073+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613692878}) 2024-09-16T05:12:12.076 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.073+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2302567176}) 2024-09-16T05:12:12.078 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.073+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.078 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.073+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102575905}) 2024-09-16T05:12:12.078 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.073+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=78385772,mdsmap=1078762840}) 2024-09-16T05:12:12.078 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.073+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.078 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.073+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102575905}) 2024-09-16T05:12:12.078 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.073+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3617929883}) 2024-09-16T05:12:12.080 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.077+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.080 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.077+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=1,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=30} crc {logm=1815763340,mdsmap=2485929294,mgr=529707881,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=2447354312,mon_config_key=1587861323,monmap=3045472550,osd_metadata=487805116,osd_pg_creating=2313498167,osd_snap=26388699,osdmap=2619867721}) 2024-09-16T05:12:12.081 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.077+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=2,osd_metadata=6,osd_snap=24,osdmap=27} crc {logm=1815763340,mdsmap=2485929294,mgr=3253566820,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=1025172728,mon_config_key=1578821364,monmap=3371862038,osd_metadata=487805116,osd_snap=2667843241,osdmap=589628495}) 2024-09-16T05:12:12.081 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.077+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.081 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.077+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=1,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=30} crc {logm=1815763340,mdsmap=2485929294,mgr=529707881,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=2447354312,mon_config_key=1587861323,monmap=3045472550,osd_metadata=487805116,osd_pg_creating=2313498167,osd_snap=26388699,osdmap=2619867721}) 2024-09-16T05:12:12.081 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.077+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=1,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,osd_metadata=6,osd_snap=23,osdmap=29} crc {logm=1815763340,mdsmap=2485929294,mgr=2796660337,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=2193104044,mon_config_key=2314223147,monmap=3045472550,osd_metadata=487805116,osd_snap=3022997552,osdmap=2773960488}) 2024-09-16T05:12:12.083 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.077+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.083 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.077+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3020640332}) 2024-09-16T05:12:12.083 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.077+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1212468846}) 2024-09-16T05:12:12.083 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.077+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.083 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.077+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3020640332}) 2024-09-16T05:12:12.083 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.077+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3255554234}) 2024-09-16T05:12:12.087 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.081+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.087 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.081+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1939876493}) 2024-09-16T05:12:12.087 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.081+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4216888910}) 2024-09-16T05:12:12.087 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.081+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.087 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.081+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1939876493}) 2024-09-16T05:12:12.087 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.081+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3564016822}) 2024-09-16T05:12:12.089 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.085+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.089 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.085+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=46} crc {osdmap=314069903}) 2024-09-16T05:12:12.090 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.085+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=44} crc {osdmap=2179758549}) 2024-09-16T05:12:12.090 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.085+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:12:12.091 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.085+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=46} crc {osdmap=314069903}) 2024-09-16T05:12:12.091 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:12:12.085+0000 7f721f361640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=45} crc {osdmap=3272138522}) 2024-09-16T05:12:13.058 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:12:13.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:12:13.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 10 monclient: get_monmap_and_config 2024-09-16T05:12:13.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 10 monclient: build_initial_monmap 2024-09-16T05:12:13.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 10 monclient: monmap: 2024-09-16T05:12:13.190 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:13.190 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.190 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:12:13.190286+0000 2024-09-16T05:12:13.190 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:12:13.190286+0000 2024-09-16T05:12:13.190 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:13.191 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.191 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:12:13.191 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:12:13.191 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:12:13.191 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:12:13.191 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:12:13.191 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 1 Processor -- start 2024-09-16T05:12:13.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 1 -- start start 2024-09-16T05:12:13.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 10 monclient: init 2024-09-16T05:12:13.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:13.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-16T05:12:13.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f10641376e0 0x7f1064137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 10 monclient(hunting): picked mon.noname-c con 0x7f10641376e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:12:13.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f10641370a0 0x7f10640a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 10 monclient(hunting): picked mon.noname-a con 0x7f10641370a0 addr v2:172.21.15.114:3300/0 2024-09-16T05:12:13.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10640a5d00 addr v1:172.21.15.114:6789/0 2024-09-16T05:12:13.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f106409bd20 con 0x7f10640a5d00 2024-09-16T05:12:13.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f1064138000 con 0x7f10641370a0 2024-09-16T05:12:13.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f106409c5c0 con 0x7f10641376e0 2024-09-16T05:12:13.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:13.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1069d62640 10 monclient(hunting): authenticate will time out at 1217.819510s 2024-09-16T05:12:13.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1062ffd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f10641370a0 0x7f10640a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1062ffd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f10641370a0 0x7f10640a4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36406/0 (socket says 172.21.15.114:36406) 2024-09-16T05:12:13.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1062ffd640 1 -- 172.21.15.114:0/899875765 learned_addr learned my addr 172.21.15.114:0/899875765 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:13.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1062ffd640 10 monclient(hunting): get_auth_request con 0x7f10641370a0 auth_method 0 2024-09-16T05:12:13.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1062ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1062ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1062ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1062ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1062ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1062ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1062ffd640 10 monclient(hunting): handle_auth_done global_id 28008 payload 274 2024-09-16T05:12:13.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f1062ffd640 1 -- 172.21.15.114:0/899875765 >> v2:172.21.15.149:3300/0 conn(0x7f10641376e0 msgr2=0x7f1064137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:13.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.185+0000 7f10637fe640 1 --2- 172.21.15.114:0/899875765 >> v2:172.21.15.149:3300/0 conn(0x7f10641376e0 0x7f1064137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1062ffd640 1 --2- 172.21.15.114:0/899875765 >> v2:172.21.15.149:3300/0 conn(0x7f10641376e0 0x7f1064137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1062ffd640 1 -- 172.21.15.114:0/899875765 >> v1:172.21.15.114:6789/0 conn(0x7f10640a5d00 legacy=0x7f10640a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1062ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:12:13.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1062ffd640 1 monclient: found mon.noname-a 2024-09-16T05:12:13.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1062ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:13.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1062ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:12:13.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1062ffd640 1 -- 172.21.15.114:0/899875765 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106409c000 con 0x7f10641370a0 2024-09-16T05:12:13.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f10637fe640 10 monclient: get_auth_request con 0x7f10641376e0 auth_method 0 2024-09-16T05:12:13.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f10637fe640 1 --2- 172.21.15.114:0/899875765 >> v2:172.21.15.149:3300/0 conn(0x7f10641376e0 0x7f1064137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:12:13.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1062ffd640 1 --2- 172.21.15.114:0/899875765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10641370a0 0x7f10640a4270 secure :-1 s=READY pgs=8316 cs=0 l=1 rev1=1 crypto rx=0x7f10640a7410 tx=0x7f105401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5790e03672da571 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f10627fc640 1 -- 172.21.15.114:0/899875765 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10540b0070 con 0x7f10641370a0 2024-09-16T05:12:13.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f10627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f10627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:12:13.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f10627fc640 10 monclient: dump: 2024-09-16T05:12:13.202 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.202 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.202 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.202 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.202 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.202 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.202 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.202 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.202 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.202 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f10627fc640 10 monclient: _finish_auth 0 2024-09-16T05:12:13.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f10627fc640 10 monclient: _check_auth_tickets 2024-09-16T05:12:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f10627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f10627fc640 1 -- 172.21.15.114:0/899875765 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10540ab600 con 0x7f10641370a0 2024-09-16T05:12:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f10627fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f10627fc640 1 -- 172.21.15.114:0/899875765 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10540ab9d0 con 0x7f10641370a0 2024-09-16T05:12:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1069d62640 5 monclient: authenticate success, global_id 28008 2024-09-16T05:12:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1069d62640 10 monclient: get_monmap_and_config success 2024-09-16T05:12:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1069d62640 4 set_mon_vals no callback set 2024-09-16T05:12:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f10627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f10627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:12:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f10627fc640 10 monclient: dump: 2024-09-16T05:12:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.203 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.204 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1069d62640 10 monclient: shutdown 2024-09-16T05:12:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1069d62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1069d62640 1 -- 172.21.15.114:0/899875765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10641370a0 msgr2=0x7f10640a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1069d62640 1 --2- 172.21.15.114:0/899875765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10641370a0 0x7f10640a4270 secure :-1 s=READY pgs=8316 cs=0 l=1 rev1=1 crypto rx=0x7f10640a7410 tx=0x7f105401c3e0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1069d62640 1 -- 172.21.15.114:0/899875765 shutdown_connections 2024-09-16T05:12:13.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1069d62640 1 --2- 172.21.15.114:0/899875765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10641370a0 0x7f10640a4270 unknown :-1 s=CLOSED pgs=8316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1069d62640 1 --2- 172.21.15.114:0/899875765 >> v2:172.21.15.149:3300/0 conn(0x7f10641376e0 0x7f1064137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1069d62640 1 -- 172.21.15.114:0/899875765 >> 172.21.15.114:0/899875765 conn(0x7f10640a9a40 msgr2=0x7f10640a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:13.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1069d62640 1 -- 172.21.15.114:0/899875765 shutdown_connections 2024-09-16T05:12:13.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1069d62640 1 -- 172.21.15.114:0/899875765 wait complete. 2024-09-16T05:12:13.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1069d62640 10 monclient: build_initial_monmap 2024-09-16T05:12:13.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.189+0000 7f1069d62640 10 monclient: monmap: 2024-09-16T05:12:13.205 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:13.205 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:12:13.205 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:12:13.205 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:12:13.205 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:13.206 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.206 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:12:13.206 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:12:13.206 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:12:13.206 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 librados: starting msgr at 2024-09-16T05:12:13.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 librados: starting objecter 2024-09-16T05:12:13.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 Processor -- start 2024-09-16T05:12:13.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 -- start start 2024-09-16T05:12:13.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 librados: setting wanted keys 2024-09-16T05:12:13.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 librados: calling monclient init 2024-09-16T05:12:13.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient: init 2024-09-16T05:12:13.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:12:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f10640a5d00 0x7f1064135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient(hunting): picked mon.noname-b con 0x7f10640a5d00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10641370a0 0x7f1064136020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient(hunting): picked mon.noname-a con 0x7f10641370a0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10641376e0 0x7f106413be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient(hunting): picked mon.noname-c con 0x7f10641376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1064141d80 con 0x7f10641370a0 2024-09-16T05:12:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f1064141c00 con 0x7f10640a5d00 2024-09-16T05:12:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1064141f00 con 0x7f10641376e0 2024-09-16T05:12:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:13.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient(hunting): authenticate will time out at 1217.824732s 2024-09-16T05:12:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10641370a0 0x7f1064136020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10641370a0 0x7f1064136020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36416/0 (socket says 172.21.15.114:36416) 2024-09-16T05:12:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 1 -- 172.21.15.114:0/3945177348 learned_addr learned my addr 172.21.15.114:0/3945177348 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 10 monclient(hunting): get_auth_request con 0x7f10641370a0 auth_method 0 2024-09-16T05:12:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1063fff640 1 --2- 172.21.15.114:0/3945177348 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10641376e0 0x7f106413be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1063fff640 10 monclient(hunting): get_auth_request con 0x7f10641376e0 auth_method 0 2024-09-16T05:12:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1063fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1063fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1063fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1063fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1063fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1063fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f10637fe640 1 --2- 172.21.15.114:0/3945177348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f10640a5d00 0x7f1064135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 10 monclient(hunting): handle_auth_done global_id 28014 payload 824 2024-09-16T05:12:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 1 -- 172.21.15.114:0/3945177348 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10641376e0 msgr2=0x7f106413be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 1 --2- 172.21.15.114:0/3945177348 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10641376e0 0x7f106413be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 1 -- 172.21.15.114:0/3945177348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f10640a5d00 msgr2=0x7f1064135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 1 --2- 172.21.15.114:0/3945177348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f10640a5d00 0x7f1064135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:12:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 1 monclient: found mon.noname-a 2024-09-16T05:12:13.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:12:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 1 -- 172.21.15.114:0/3945177348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10640a8e60 con 0x7f10641370a0 2024-09-16T05:12:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f10637fe640 10 monclient: get_auth_request con 0x7f10640a5d00 auth_method 0 2024-09-16T05:12:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f10637fe640 1 --2- 172.21.15.114:0/3945177348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f10640a5d00 0x7f1064135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:12:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1062ffd640 1 --2- 172.21.15.114:0/3945177348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10641370a0 0x7f1064136020 secure :-1 s=READY pgs=8317 cs=0 l=1 rev1=1 crypto rx=0x7f1054018ff0 tx=0x7f1054019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 10 client.?.objecter ms_handle_connect 0x7f10641370a0 2024-09-16T05:12:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:12:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 1 -- 172.21.15.114:0/3945177348 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10540b9070 con 0x7f10641370a0 2024-09-16T05:12:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:12:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 10 monclient: dump: 2024-09-16T05:12:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.211 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 10 monclient: _finish_auth 0 2024-09-16T05:12:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:12:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 1 -- 172.21.15.114:0/3945177348 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10540b4600 con 0x7f10641370a0 2024-09-16T05:12:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 5 monclient: authenticate success, global_id 28014 2024-09-16T05:12:13.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient: _renew_subs 2024-09-16T05:12:13.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:12:13.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 -- 172.21.15.114:0/3945177348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1064073d20 con 0x7f10641370a0 2024-09-16T05:12:13.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:13.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 1 -- 172.21.15.114:0/3945177348 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10540b0040 con 0x7f10641370a0 2024-09-16T05:12:13.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:12:13.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 10 monclient: dump: 2024-09-16T05:12:13.212 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.212 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.212 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.212 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.212 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.212 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.213 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1061ffb640 4 set_mon_vals no callback set 2024-09-16T05:12:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 client.28014.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:12:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient: _renew_subs 2024-09-16T05:12:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:12:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 -- 172.21.15.114:0/3945177348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f106411fec0 con 0x7f10641370a0 2024-09-16T05:12:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 librados: init done 2024-09-16T05:12:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:12:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f1069d62640 1 -- 172.21.15.114:0/3945177348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10640a2500 con 0x7f10641370a0 2024-09-16T05:12:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 1 -- 172.21.15.114:0/3945177348 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f10540bc070 con 0x7f10641370a0 2024-09-16T05:12:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 1 --2- 172.21.15.114:0/3945177348 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1044037810 0x7f1044039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 1 -- 172.21.15.114:0/3945177348 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(180..180 src has 1..180) ==== 6023+0+0 (secure 0 0 0) 0x7f10540e23d0 con 0x7f10641370a0 2024-09-16T05:12:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 10 client.28014.objecter ms_dispatch 0x7f1064000ba0 osd_map(180..180 src has 1..180) 2024-09-16T05:12:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 3 client.28014.objecter handle_osd_map got epochs [180,180] > 0 2024-09-16T05:12:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 3 client.28014.objecter handle_osd_map decoding full epoch 180 2024-09-16T05:12:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 20 client.28014.objecter dump_active .. 0 homeless 2024-09-16T05:12:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f10637fe640 1 --2- 172.21.15.114:0/3945177348 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1044037810 0x7f1044039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f10637fe640 10 monclient: get_auth_request con 0x7f1044037810 auth_method 0 2024-09-16T05:12:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f10637fe640 1 --2- 172.21.15.114:0/3945177348 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1044037810 0x7f1044039cd0 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 crypto rx=0x7f104c008930 tx=0x7f104c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.193+0000 7f105bfff640 10 client.28014.objecter ms_handle_connect 0x7f1044037810 2024-09-16T05:12:13.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.197+0000 7f105bfff640 1 -- 172.21.15.114:0/3945177348 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f1064073d20 con 0x7f10641370a0 2024-09-16T05:12:13.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.197+0000 7f105bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:13.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.197+0000 7f105bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:12:13.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:12:13.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:12:13.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:12:13.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 1 -- 172.21.15.114:0/3945177348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1064137ac0 con 0x7f10641370a0 2024-09-16T05:12:13.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f105bfff640 1 -- 172.21.15.114:0/3945177348 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f10640a8e60 con 0x7f10641370a0 2024-09-16T05:12:13.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f105bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:12:13.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f105bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:12:13.433 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:12:13.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 10 librados: watch_flush enter 2024-09-16T05:12:13.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 10 librados: watch_flush exit 2024-09-16T05:12:13.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 20 client.28014.objecter shutdown clearing up homeless session... 2024-09-16T05:12:13.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 10 client.28014.objecter successfully canceled tick 2024-09-16T05:12:13.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 1 -- 172.21.15.114:0/3945177348 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1044037810 msgr2=0x7f1044039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 1 --2- 172.21.15.114:0/3945177348 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1044037810 0x7f1044039cd0 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 crypto rx=0x7f104c008930 tx=0x7f104c008040 comp rx=0 tx=0).stop 2024-09-16T05:12:13.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 10 monclient: shutdown 2024-09-16T05:12:13.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:13.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 1 -- 172.21.15.114:0/3945177348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10641370a0 msgr2=0x7f1064136020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 1 --2- 172.21.15.114:0/3945177348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10641370a0 0x7f1064136020 secure :-1 s=READY pgs=8317 cs=0 l=1 rev1=1 crypto rx=0x7f1054018ff0 tx=0x7f1054019180 comp rx=0 tx=0).stop 2024-09-16T05:12:13.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 1 -- 172.21.15.114:0/3945177348 shutdown_connections 2024-09-16T05:12:13.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 1 --2- 172.21.15.114:0/3945177348 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1044037810 0x7f1044039cd0 unknown :-1 s=CLOSED pgs=3458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 1 --2- 172.21.15.114:0/3945177348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10641370a0 0x7f1064136020 unknown :-1 s=CLOSED pgs=8317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 1 --2- 172.21.15.114:0/3945177348 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10641376e0 0x7f106413be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 1 --2- 172.21.15.114:0/3945177348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f10640a5d00 0x7f1064135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 1 -- 172.21.15.114:0/3945177348 >> 172.21.15.114:0/3945177348 conn(0x7f10640a9a40 msgr2=0x7f10640a7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:13.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 1 -- 172.21.15.114:0/3945177348 shutdown_connections 2024-09-16T05:12:13.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 1 -- 172.21.15.114:0/3945177348 wait complete. 2024-09-16T05:12:13.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.429+0000 7f1069d62640 1 librados: shutdown 2024-09-16T05:12:13.452 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:12:13.452 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:12:13.452 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:12:13.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:12:13.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 10 monclient: get_monmap_and_config 2024-09-16T05:12:13.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 10 monclient: build_initial_monmap 2024-09-16T05:12:13.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 10 monclient: monmap: 2024-09-16T05:12:13.568 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:13.568 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.569 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:12:13.568649+0000 2024-09-16T05:12:13.569 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:12:13.568649+0000 2024-09-16T05:12:13.569 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:13.569 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.569 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:12:13.569 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:12:13.569 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:12:13.569 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:12:13.569 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:12:13.569 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 1 Processor -- start 2024-09-16T05:12:13.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 1 -- start start 2024-09-16T05:12:13.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 10 monclient: init 2024-09-16T05:12:13.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:13.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T05:12:13.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f539c1376e0 0x7f539c137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 10 monclient(hunting): picked mon.noname-c con 0x7f539c1376e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:12:13.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f539c1370a0 0x7f539c0a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 10 monclient(hunting): picked mon.noname-b con 0x7f539c1370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:13.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f539c0a5d00 0x7f539c0a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 10 monclient(hunting): picked mon.noname-a con 0x7f539c0a5d00 addr v2:172.21.15.114:3300/0 2024-09-16T05:12:13.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f539c138000 con 0x7f539c0a5d00 2024-09-16T05:12:13.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f539c09c5c0 con 0x7f539c1376e0 2024-09-16T05:12:13.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f539c09bd20 con 0x7f539c1370a0 2024-09-16T05:12:13.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:13.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f53a1fae640 10 monclient(hunting): authenticate will time out at 1218.197915s 2024-09-16T05:12:13.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f539c1370a0 0x7f539c0a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539bfff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f539c0a5d00 0x7f539c0a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539bfff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f539c0a5d00 0x7f539c0a47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36422/0 (socket says 172.21.15.114:36422) 2024-09-16T05:12:13.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 10 monclient(hunting): get_auth_request con 0x7f539c1370a0 auth_method 0 2024-09-16T05:12:13.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539bfff640 1 -- 172.21.15.114:0/2828689486 learned_addr learned my addr 172.21.15.114:0/2828689486 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:13.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539bfff640 10 monclient(hunting): get_auth_request con 0x7f539c0a5d00 auth_method 0 2024-09-16T05:12:13.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539b7fe640 1 --2- 172.21.15.114:0/2828689486 >> v2:172.21.15.149:3300/0 conn(0x7f539c1376e0 0x7f539c137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 10 monclient(hunting): handle_auth_done global_id 54482 payload 274 2024-09-16T05:12:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 1 -- 172.21.15.114:0/2828689486 >> v2:172.21.15.149:3300/0 conn(0x7f539c1376e0 msgr2=0x7f539c137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 1 --2- 172.21.15.114:0/2828689486 >> v2:172.21.15.149:3300/0 conn(0x7f539c1376e0 0x7f539c137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 1 -- 172.21.15.114:0/2828689486 >> v2:172.21.15.114:3300/0 conn(0x7f539c0a5d00 msgr2=0x7f539c0a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 1 --2- 172.21.15.114:0/2828689486 >> v2:172.21.15.114:3300/0 conn(0x7f539c0a5d00 0x7f539c0a47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 10 monclient: _finish_hunting 0 2024-09-16T05:12:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 1 monclient: found mon.noname-b 2024-09-16T05:12:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:13.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:12:13.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539affd640 1 -- 172.21.15.114:0/2828689486 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f539c09c000 con 0x7f539c1370a0 2024-09-16T05:12:13.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539b7fe640 10 monclient: get_auth_request con 0x7f539c1376e0 auth_method 0 2024-09-16T05:12:13.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539b7fe640 1 --2- 172.21.15.114:0/2828689486 >> v2:172.21.15.149:3300/0 conn(0x7f539c1376e0 0x7f539c137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:12:13.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.565+0000 7f539bfff640 1 --2- 172.21.15.114:0/2828689486 >> v2:172.21.15.114:3300/0 conn(0x7f539c0a5d00 0x7f539c0a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:12:13.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539affd640 1 --2- 172.21.15.114:0/2828689486 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f539c1370a0 0x7f539c0a4270 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f539c0a7410 tx=0x7f538401c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f44e63fabbd5e2b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539a7fc640 1 -- 172.21.15.114:0/2828689486 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53840ac070 con 0x7f539c1370a0 2024-09-16T05:12:13.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:12:13.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539a7fc640 10 monclient: dump: 2024-09-16T05:12:13.582 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.583 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.583 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.583 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.583 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.583 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.583 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.583 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.583 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.583 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:12:13.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:12:13.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:13.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539a7fc640 1 -- 172.21.15.114:0/2828689486 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53840a6680 con 0x7f539c1370a0 2024-09-16T05:12:13.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:13.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539a7fc640 1 -- 172.21.15.114:0/2828689486 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53840a6a50 con 0x7f539c1370a0 2024-09-16T05:12:13.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 5 monclient: authenticate success, global_id 54482 2024-09-16T05:12:13.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient: get_monmap_and_config success 2024-09-16T05:12:13.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 4 set_mon_vals no callback set 2024-09-16T05:12:13.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:13.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539a7fc640 10 monclient: dump: 2024-09-16T05:12:13.584 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.584 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.584 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.584 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.585 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.585 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.585 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.585 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.585 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.585 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient: shutdown 2024-09-16T05:12:13.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:13.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 -- 172.21.15.114:0/2828689486 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f539c1370a0 msgr2=0x7f539c0a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 --2- 172.21.15.114:0/2828689486 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f539c1370a0 0x7f539c0a4270 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f539c0a7410 tx=0x7f538401c7f0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 -- 172.21.15.114:0/2828689486 shutdown_connections 2024-09-16T05:12:13.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 --2- 172.21.15.114:0/2828689486 >> v2:172.21.15.114:3300/0 conn(0x7f539c0a5d00 0x7f539c0a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 --2- 172.21.15.114:0/2828689486 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f539c1370a0 0x7f539c0a4270 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 --2- 172.21.15.114:0/2828689486 >> v2:172.21.15.149:3300/0 conn(0x7f539c1376e0 0x7f539c137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 -- 172.21.15.114:0/2828689486 >> 172.21.15.114:0/2828689486 conn(0x7f539c0a9a40 msgr2=0x7f539c0a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:13.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 -- 172.21.15.114:0/2828689486 shutdown_connections 2024-09-16T05:12:13.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 -- 172.21.15.114:0/2828689486 wait complete. 2024-09-16T05:12:13.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient: build_initial_monmap 2024-09-16T05:12:13.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient: monmap: 2024-09-16T05:12:13.586 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:13.586 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:12:13.586 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:12:13.586 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:12:13.586 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:13.586 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.586 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:12:13.586 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:12:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:12:13.587 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 librados: starting msgr at 2024-09-16T05:12:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 librados: starting objecter 2024-09-16T05:12:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 Processor -- start 2024-09-16T05:12:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 -- start start 2024-09-16T05:12:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 librados: setting wanted keys 2024-09-16T05:12:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 librados: calling monclient init 2024-09-16T05:12:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient: init 2024-09-16T05:12:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:12:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f539c0a5d00 0x7f539c1cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient(hunting): picked mon.noname-b con 0x7f539c0a5d00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c1370a0 0x7f539c1cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient(hunting): picked mon.noname-a con 0x7f539c1370a0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f539c1376e0 0x7f539c1d0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient(hunting): picked mon.noname-c con 0x7f539c1376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f539c140f80 con 0x7f539c1370a0 2024-09-16T05:12:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f539c140e00 con 0x7f539c0a5d00 2024-09-16T05:12:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f539c141100 con 0x7f539c1376e0 2024-09-16T05:12:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient(hunting): authenticate will time out at 1218.203020s 2024-09-16T05:12:13.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c1370a0 0x7f539c1cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f539c1376e0 0x7f539c1d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f539c1376e0 0x7f539c1d0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:59212/0 (socket says 172.21.15.114:59212) 2024-09-16T05:12:13.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c1370a0 0x7f539c1cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36432/0 (socket says 172.21.15.114:36432) 2024-09-16T05:12:13.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 1 -- 172.21.15.114:0/1986639271 learned_addr learned my addr 172.21.15.114:0/1986639271 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:13.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 10 monclient(hunting): get_auth_request con 0x7f539c1376e0 auth_method 0 2024-09-16T05:12:13.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539affd640 10 monclient(hunting): get_auth_request con 0x7f539c1370a0 auth_method 0 2024-09-16T05:12:13.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539b7fe640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f539c0a5d00 0x7f539c1cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539b7fe640 10 monclient(hunting): get_auth_request con 0x7f539c0a5d00 auth_method 0 2024-09-16T05:12:13.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 10 monclient(hunting): handle_auth_done global_id 54485 payload 824 2024-09-16T05:12:13.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 1 -- 172.21.15.114:0/1986639271 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f539c0a5d00 msgr2=0x7f539c1cf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f539c0a5d00 0x7f539c1cf5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 1 -- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c1370a0 msgr2=0x7f539c1cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c1370a0 0x7f539c1cfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:12:13.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 1 monclient: found mon.noname-c 2024-09-16T05:12:13.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:13.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:12:13.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 1 -- 172.21.15.114:0/1986639271 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f539c136ed0 con 0x7f539c1376e0 2024-09-16T05:12:13.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539affd640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c1370a0 0x7f539c1cfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:12:13.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f539bfff640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f539c1376e0 0x7f539c1d0040 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f538c00e980 tx=0x7f538c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5393fff640 10 client.?.objecter ms_handle_connect 0x7f539c1376e0 2024-09-16T05:12:13.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5393fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:12:13.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5393fff640 1 -- 172.21.15.114:0/1986639271 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c017070 con 0x7f539c1376e0 2024-09-16T05:12:13.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5393fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5393fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:12:13.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5393fff640 10 monclient: dump: 2024-09-16T05:12:13.592 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.592 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.592 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.592 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.592 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.592 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.593 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.593 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.593 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.593 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5393fff640 10 monclient: _finish_auth 0 2024-09-16T05:12:13.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5393fff640 10 monclient: _check_auth_tickets 2024-09-16T05:12:13.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5393fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:13.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 5 monclient: authenticate success, global_id 54485 2024-09-16T05:12:13.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient: _renew_subs 2024-09-16T05:12:13.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5393fff640 1 -- 172.21.15.114:0/1986639271 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f538c00ca00 con 0x7f539c1376e0 2024-09-16T05:12:13.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:13.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 -- 172.21.15.114:0/1986639271 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f539c0a2490 con 0x7f539c1376e0 2024-09-16T05:12:13.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5393fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:13.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5393fff640 1 -- 172.21.15.114:0/1986639271 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c00cdd0 con 0x7f539c1376e0 2024-09-16T05:12:13.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5393fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5393fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:13.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5393fff640 10 monclient: dump: 2024-09-16T05:12:13.594 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.594 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.594 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.594 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.594 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.594 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.594 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.594 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.595 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.595 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f5399ffb640 4 set_mon_vals no callback set 2024-09-16T05:12:13.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 client.54485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:12:13.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient: _renew_subs 2024-09-16T05:12:13.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:13.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.569+0000 7f53a1fae640 1 -- 172.21.15.114:0/1986639271 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f539c13cf20 con 0x7f539c1376e0 2024-09-16T05:12:13.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f53a1fae640 1 librados: init done 2024-09-16T05:12:13.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 1 -- 172.21.15.114:0/1986639271 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f538c013470 con 0x7f539c1376e0 2024-09-16T05:12:13.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f53a1fae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:13.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f53a1fae640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c13dc60 0x7f539c133940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f53a1fae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f53a1fae640 1 -- 172.21.15.114:0/1986639271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f539c073d80 con 0x7f539c13dc60 2024-09-16T05:12:13.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f539b7fe640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c13dc60 0x7f539c133940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 1 --2- 172.21.15.114:0/1986639271 >> v2:172.21.15.149:6812/1185531135 conn(0x7f537c037810 0x7f537c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f539b7fe640 10 monclient: get_auth_request con 0x7f539c13dc60 auth_method 0 2024-09-16T05:12:13.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f539b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f539b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f539b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 1 -- 172.21.15.114:0/1986639271 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(180..180 src has 1..180) ==== 6023+0+0 (secure 0 0 0) 0x7f538c048be0 con 0x7f539c1376e0 2024-09-16T05:12:13.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 10 client.54485.objecter ms_dispatch 0x7f539c000ba0 osd_map(180..180 src has 1..180) 2024-09-16T05:12:13.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f539b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f539b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 3 client.54485.objecter handle_osd_map got epochs [180,180] > 0 2024-09-16T05:12:13.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 3 client.54485.objecter handle_osd_map decoding full epoch 180 2024-09-16T05:12:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f539b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 20 client.54485.objecter dump_active .. 0 homeless 2024-09-16T05:12:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f539b7fe640 10 monclient(hunting): handle_auth_done global_id 28023 payload 824 2024-09-16T05:12:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f539b7fe640 1 -- 172.21.15.114:0/1986639271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f539c137ac0 con 0x7f539c13dc60 2024-09-16T05:12:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f539affd640 1 --2- 172.21.15.114:0/1986639271 >> v2:172.21.15.149:6812/1185531135 conn(0x7f537c037810 0x7f537c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f539affd640 10 monclient: get_auth_request con 0x7f537c037810 auth_method 0 2024-09-16T05:12:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 10 client.54485.objecter ms_handle_connect 0x7f539c13dc60 2024-09-16T05:12:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f539b7fe640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c13dc60 0x7f539c133940 secure :-1 s=READY pgs=8318 cs=0 l=1 rev1=1 crypto rx=0x7f539400f480 tx=0x7f539400f940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 10 client.54485.objecter resend_mon_ops 2024-09-16T05:12:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 1 -- 172.21.15.114:0/1986639271 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5394018070 con 0x7f539c13dc60 2024-09-16T05:12:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:12:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 10 monclient: dump: 2024-09-16T05:12:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.598 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 1 -- 172.21.15.114:0/1986639271 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5394014890 con 0x7f539c13dc60 2024-09-16T05:12:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:12:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 1 -- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c13dc60 msgr2=0x7f539c133940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c13dc60 0x7f539c133940 secure :-1 s=READY pgs=8318 cs=0 l=1 rev1=1 crypto rx=0x7f539400f480 tx=0x7f539400f940 comp rx=0 tx=0).stop 2024-09-16T05:12:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 1 -- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c13dc60 msgr2=0x7f539c133940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c13dc60 0x7f539c133940 unknown :-1 s=CLOSED pgs=8318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f5393fff640 10 client.54485.objecter ms_handle_connect 0x7f537c037810 2024-09-16T05:12:13.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.573+0000 7f539affd640 1 --2- 172.21.15.114:0/1986639271 >> v2:172.21.15.149:6812/1185531135 conn(0x7f537c037810 0x7f537c039cd0 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7f53840ae000 tx=0x7f53840a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.577+0000 7f53a1fae640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:12:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.577+0000 7f53a1fae640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c13d0f0 0x7f539c02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.577+0000 7f53a1fae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.577+0000 7f53a1fae640 1 -- 172.21.15.114:0/1986639271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f539c09f280 con 0x7f539c13d0f0 2024-09-16T05:12:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f539b7fe640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c13d0f0 0x7f539c02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f539b7fe640 10 monclient: get_auth_request con 0x7f539c13d0f0 auth_method 0 2024-09-16T05:12:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f539b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f539b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f539b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f539b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f539b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f539b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f539b7fe640 10 monclient(hunting): handle_auth_done global_id 28026 payload 824 2024-09-16T05:12:13.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f539b7fe640 1 -- 172.21.15.114:0/1986639271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f539c068820 con 0x7f539c13d0f0 2024-09-16T05:12:13.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f5393fff640 10 client.54485.objecter ms_handle_connect 0x7f539c13d0f0 2024-09-16T05:12:13.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f5393fff640 10 client.54485.objecter resend_mon_ops 2024-09-16T05:12:13.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f539b7fe640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c13d0f0 0x7f539c02c650 secure :-1 s=READY pgs=8319 cs=0 l=1 rev1=1 crypto rx=0x7f539400f480 tx=0x7f53940046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f5393fff640 1 -- 172.21.15.114:0/1986639271 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53940135f0 con 0x7f539c13d0f0 2024-09-16T05:12:13.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f5393fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f5393fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:12:13.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f5393fff640 10 monclient: dump: 2024-09-16T05:12:13.602 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.602 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.602 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.602 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.602 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.602 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.602 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.603 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.603 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.603 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f5393fff640 1 -- 172.21.15.114:0/1986639271 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f53940135f0 con 0x7f539c13d0f0 2024-09-16T05:12:13.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f5393fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:12:13.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f5393fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:12:13.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f5393fff640 1 -- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c13d0f0 msgr2=0x7f539c02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f5393fff640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c13d0f0 0x7f539c02c650 secure :-1 s=READY pgs=8319 cs=0 l=1 rev1=1 crypto rx=0x7f539400f480 tx=0x7f53940046d0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f5393fff640 1 -- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c13d0f0 msgr2=0x7f539c02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:13.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f5393fff640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f539c13d0f0 0x7f539c02c650 unknown :-1 s=CLOSED pgs=8319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.604 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:12:13.604 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:12:13.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:12:13.604 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:12:13.604 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 20, 2024-09-16T05:12:13.604 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 291570, 2024-09-16T05:12:13.604 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:12:13.604 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:12:13.604 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:12:13.604 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:12:13.604 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:13.604 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:12:13.605 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:13.605 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:12:13.605 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:12:13.605 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:13.605 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:13.605 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:13.605 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:13.605 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:13.605 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:13.605 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:13.605 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:13.605 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:13.605 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:13.606 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:13.606 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:13.606 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:12:13.606 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:12:13.606 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:13.606 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:13.606 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:13.606 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:13.606 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:13.606 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:13.606 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:13.606 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:13.606 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:13.607 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:13.607 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:13.607 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.607 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:12:13.607 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:12:13.607 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:12:13.607 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:12:13.607 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:12:13.607 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:12:13.607 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:13.607 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:13.607 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:12:13.607 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:12:13.608 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:12:13.608 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:12:13.608 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:12:13.608 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:12:13.608 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:12:13.608 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:13.608 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:12:13.608 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:13.608 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:13.608 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:13.608 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:13.608 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:13.608 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:13.609 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:13.609 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:13.609 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:13.609 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:13.609 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:13.609 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:13.609 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:12:13.609 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.609 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:12:13.609 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:12:13.609 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:12:13.609 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:13.610 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:13.610 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.610 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:13.610 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:12:13.610 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.610 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.610 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.610 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:13.610 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:12:13.610 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.610 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.610 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.611 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:12:13.611 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:12:13.611 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:13.611 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:13.611 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:13.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.611 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.611 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:12:13.611 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:12:13.611 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:13.611 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:13.611 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.611 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:13.612 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:12:13.612 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.612 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.612 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:13.612 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:12:13.612 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.612 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.612 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.612 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:12:13.612 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:12:13.612 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:13.613 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:13.613 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:13.613 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.613 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:12:13.613 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:12:13.613 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:13.613 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:13.613 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.613 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:13.613 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:12:13.613 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.613 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.613 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.614 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:13.614 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:12:13.614 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.614 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.614 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.614 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.614 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:12:13.614 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:12:13.614 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:13.614 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:13.614 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:13.614 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.614 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.615 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.615 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:12:13.615 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:12:13.615 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.615 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:13.615 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:13.615 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:12:13.615 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.615 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:13.615 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:12:13.615 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.615 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:13.615 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:13.615 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:12:13.616 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.616 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:13.616 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:12:13.616 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.616 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:13.616 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:13.616 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:12:13.616 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.616 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.616 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.616 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:12:13.616 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:12:13.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 10 librados: watch_flush enter 2024-09-16T05:12:13.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 10 librados: watch_flush exit 2024-09-16T05:12:13.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 20 client.54485.objecter shutdown clearing up homeless session... 2024-09-16T05:12:13.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 10 client.54485.objecter successfully canceled tick 2024-09-16T05:12:13.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 1 -- 172.21.15.114:0/1986639271 >> v2:172.21.15.149:6812/1185531135 conn(0x7f537c037810 msgr2=0x7f537c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 1 --2- 172.21.15.114:0/1986639271 >> v2:172.21.15.149:6812/1185531135 conn(0x7f537c037810 0x7f537c039cd0 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7f53840ae000 tx=0x7f53840a7000 comp rx=0 tx=0).stop 2024-09-16T05:12:13.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 10 monclient: shutdown 2024-09-16T05:12:13.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:13.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 1 -- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f539c1376e0 msgr2=0x7f539c1d0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f539c1376e0 0x7f539c1d0040 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f538c00e980 tx=0x7f538c00ee40 comp rx=0 tx=0).stop 2024-09-16T05:12:13.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f539bfff640 1 -- 172.21.15.114:0/1986639271 reap_dead start 2024-09-16T05:12:13.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 1 -- 172.21.15.114:0/1986639271 shutdown_connections 2024-09-16T05:12:13.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 1 --2- 172.21.15.114:0/1986639271 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f539c1376e0 0x7f539c1d0040 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 1 -- 172.21.15.114:0/1986639271 >> 172.21.15.114:0/1986639271 conn(0x7f539c0a9a40 msgr2=0x7f539c1367c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:13.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 1 -- 172.21.15.114:0/1986639271 shutdown_connections 2024-09-16T05:12:13.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 1 -- 172.21.15.114:0/1986639271 wait complete. 2024-09-16T05:12:13.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.581+0000 7f53a1fae640 1 librados: shutdown 2024-09-16T05:12:13.620 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:12:13.620 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:12:13.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:12:13.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 10 monclient: get_monmap_and_config 2024-09-16T05:12:13.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 10 monclient: build_initial_monmap 2024-09-16T05:12:13.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 10 monclient: monmap: 2024-09-16T05:12:13.734 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:13.734 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.734 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:12:13.733699+0000 2024-09-16T05:12:13.734 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:12:13.733699+0000 2024-09-16T05:12:13.734 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:13.734 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.734 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:12:13.734 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:12:13.734 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:12:13.735 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:12:13.735 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:12:13.735 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 1 Processor -- start 2024-09-16T05:12:13.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 1 -- start start 2024-09-16T05:12:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 10 monclient: init 2024-09-16T05:12:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T05:12:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fac3809b6c0 0x7fac380a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 10 monclient(hunting): picked mon.noname-c con 0x7fac3809b6c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:12:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac380a1400 addr v1:172.21.15.114:6789/0 2024-09-16T05:12:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac380a1810 addr v1:172.21.15.149:6789/0 2024-09-16T05:12:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac3809d4e0 con 0x7fac380a1400 2024-09-16T05:12:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac3809f430 con 0x7fac380a1810 2024-09-16T05:12:13.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fac3809e390 con 0x7fac3809b6c0 2024-09-16T05:12:13.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:13.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 10 monclient(hunting): authenticate will time out at 1218.362815s 2024-09-16T05:12:13.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac377fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fac380a1400 0x7fac38139640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38562/0 (socket says 172.21.15.114:38562) 2024-09-16T05:12:13.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac377fe640 1 -- 172.21.15.114:0/59747703 learned_addr learned my addr 172.21.15.114:0/59747703 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:12:13.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac37fff640 1 --2- 172.21.15.114:0/59747703 >> v2:172.21.15.149:3300/0 conn(0x7fac3809b6c0 0x7fac380a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac37fff640 10 monclient(hunting): get_auth_request con 0x7fac3809b6c0 auth_method 0 2024-09-16T05:12:13.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 -- 172.21.15.114:0/59747703 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4160624033 0 0) 0x7fac3809d4e0 con 0x7fac380a1400 2024-09-16T05:12:13.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:12:13.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient(hunting): my global_id is 28029 2024-09-16T05:12:13.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 -- 172.21.15.114:0/59747703 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac20003530 con 0x7fac380a1400 2024-09-16T05:12:13.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 -- 172.21.15.114:0/59747703 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2830259276 0 0) 0x7fac20003530 con 0x7fac380a1400 2024-09-16T05:12:13.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 -- 172.21.15.114:0/59747703 >> v2:172.21.15.149:3300/0 conn(0x7fac3809b6c0 msgr2=0x7fac380a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:13.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 --2- 172.21.15.114:0/59747703 >> v2:172.21.15.149:3300/0 conn(0x7fac3809b6c0 0x7fac380a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 -- 172.21.15.114:0/59747703 >> v1:172.21.15.149:6789/0 conn(0x7fac380a1810 legacy=0x7fac3813cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:12:13.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:12:13.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:12:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 -- 172.21.15.114:0/59747703 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac380a1cc0 con 0x7fac380a1400 2024-09-16T05:12:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient: _finish_auth 0 2024-09-16T05:12:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:12:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 5 monclient: authenticate success, global_id 28029 2024-09-16T05:12:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 -- 172.21.15.114:0/59747703 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7fac2c001820 con 0x7fac380a1400 2024-09-16T05:12:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:12:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient: dump: 2024-09-16T05:12:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.758 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.759 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:12:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 -- 172.21.15.114:0/59747703 >> v1:172.21.15.114:6789/0 conn(0x7fac380a1400 legacy=0x7fac38139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:12:13.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 --2- 172.21.15.114:0/59747703 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac200039c0 0x7fac20003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient(hunting): picked mon.b con 0x7fac200039c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 --2- 172.21.15.114:0/59747703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac20004810 0x7fac20004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient(hunting): picked mon.a con 0x7fac20004810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 --2- 172.21.15.114:0/59747703 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac200052f0 0x7fac2000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient(hunting): picked mon.c con 0x7fac200052f0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 -- 172.21.15.114:0/59747703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fac3809e390 con 0x7fac20004810 2024-09-16T05:12:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 -- 172.21.15.114:0/59747703 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fac2000c1c0 con 0x7fac200039c0 2024-09-16T05:12:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 1 -- 172.21.15.114:0/59747703 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fac2000c3c0 con 0x7fac200052f0 2024-09-16T05:12:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac36ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:12:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3e6c1640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:12:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac377fe640 1 --2- 172.21.15.114:0/59747703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac20004810 0x7fac20004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac377fe640 10 monclient(hunting): get_auth_request con 0x7fac20004810 auth_method 0 2024-09-16T05:12:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac377fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:12:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.729+0000 7fac3cc5e640 1 --2- 172.21.15.114:0/59747703 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac200052f0 0x7fac2000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3cc5e640 10 monclient(hunting): get_auth_request con 0x7fac200052f0 auth_method 0 2024-09-16T05:12:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3cc5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3cc5e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3cc5e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:12:13.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:12:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac37fff640 1 --2- 172.21.15.114:0/59747703 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac200039c0 0x7fac20003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac37fff640 10 monclient(hunting): get_auth_request con 0x7fac200039c0 auth_method 0 2024-09-16T05:12:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac37fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:12:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3cc5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3cc5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3cc5e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:12:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 10 monclient(hunting): handle_auth_done global_id 28029 payload 293 2024-09-16T05:12:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 1 -- 172.21.15.114:0/59747703 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac200052f0 msgr2=0x7fac2000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:13.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 1 --2- 172.21.15.114:0/59747703 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac200052f0 0x7fac2000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 1 -- 172.21.15.114:0/59747703 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac200039c0 msgr2=0x7fac20003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 1 --2- 172.21.15.114:0/59747703 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac200039c0 0x7fac20003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 10 monclient: _finish_hunting 0 2024-09-16T05:12:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 1 monclient: found mon.a 2024-09-16T05:12:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:12:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 1 -- 172.21.15.114:0/59747703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac20003530 con 0x7fac20004810 2024-09-16T05:12:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 10 monclient: _finish_auth 0 2024-09-16T05:12:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 10 monclient: _check_auth_tickets 2024-09-16T05:12:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3cc5e640 1 -- 172.21.15.114:0/59747703 reap_dead start 2024-09-16T05:12:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac37fff640 1 --2- 172.21.15.114:0/59747703 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac200039c0 0x7fac20003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:12:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac377fe640 1 --2- 172.21.15.114:0/59747703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac20004810 0x7fac20004c90 secure :-1 s=READY pgs=8321 cs=0 l=1 rev1=1 crypto rx=0x7fac3809d6b0 tx=0x7fac2c01cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4004396aae6cab server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac36ffd640 1 -- 172.21.15.114:0/59747703 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c007d50 con 0x7fac20004810 2024-09-16T05:12:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:12:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac36ffd640 10 monclient: dump: 2024-09-16T05:12:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.764 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.765 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac36ffd640 1 -- 172.21.15.114:0/59747703 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac2c0a9070 con 0x7fac20004810 2024-09-16T05:12:13.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:12:13.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac36ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:13.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 10 monclient: get_monmap_and_config success 2024-09-16T05:12:13.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 4 set_mon_vals no callback set 2024-09-16T05:12:13.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac36ffd640 1 -- 172.21.15.114:0/59747703 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0074b0 con 0x7fac20004810 2024-09-16T05:12:13.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:12:13.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac36ffd640 10 monclient: dump: 2024-09-16T05:12:13.765 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.765 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.765 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.766 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.766 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.766 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.766 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.766 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.766 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.766 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 10 monclient: shutdown 2024-09-16T05:12:13.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:13.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 1 -- 172.21.15.114:0/59747703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac20004810 msgr2=0x7fac20004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 1 --2- 172.21.15.114:0/59747703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac20004810 0x7fac20004c90 secure :-1 s=READY pgs=8321 cs=0 l=1 rev1=1 crypto rx=0x7fac3809d6b0 tx=0x7fac2c01cbe0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 1 -- 172.21.15.114:0/59747703 shutdown_connections 2024-09-16T05:12:13.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 1 --2- 172.21.15.114:0/59747703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac20004810 0x7fac20004c90 unknown :-1 s=CLOSED pgs=8321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 1 -- 172.21.15.114:0/59747703 >> 172.21.15.114:0/59747703 conn(0x7fac380a1fb0 msgr2=0x7fac380a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:13.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 1 -- 172.21.15.114:0/59747703 shutdown_connections 2024-09-16T05:12:13.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 1 -- 172.21.15.114:0/59747703 wait complete. 2024-09-16T05:12:13.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 10 monclient: build_initial_monmap 2024-09-16T05:12:13.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 10 monclient: monmap: 2024-09-16T05:12:13.767 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:13.767 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:12:13.767 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:12:13.767 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:12:13.767 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:13.767 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.767 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:12:13.767 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:12:13.767 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:12:13.768 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 1 librados: starting msgr at 2024-09-16T05:12:13.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.733+0000 7fac3e6c1640 1 librados: starting objecter 2024-09-16T05:12:13.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 1 Processor -- start 2024-09-16T05:12:13.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 1 -- start start 2024-09-16T05:12:13.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 1 librados: setting wanted keys 2024-09-16T05:12:13.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 1 librados: calling monclient init 2024-09-16T05:12:13.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient: init 2024-09-16T05:12:13.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:13.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:12:13.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac20004810 0x7fac38133590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient(hunting): picked mon.noname-b con 0x7fac20004810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:13.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac38133ad0 0x7fac38133f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient(hunting): picked mon.noname-c con 0x7fac38133ad0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:13.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac381cf320 0x7fac381cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient(hunting): picked mon.noname-a con 0x7fac381cf320 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:13.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fac3809cad0 con 0x7fac381cf320 2024-09-16T05:12:13.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fac3809baa0 con 0x7fac20004810 2024-09-16T05:12:13.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fac38137e40 con 0x7fac38133ad0 2024-09-16T05:12:13.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:13.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient(hunting): authenticate will time out at 1218.368628s 2024-09-16T05:12:13.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac38133ad0 0x7fac38133f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3cc5e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac381cf320 0x7fac381cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac38133ad0 0x7fac38133f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:59234/0 (socket says 172.21.15.114:59234) 2024-09-16T05:12:13.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 1 -- 172.21.15.114:0/2880549542 learned_addr learned my addr 172.21.15.114:0/2880549542 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:13.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3cc5e640 10 monclient(hunting): get_auth_request con 0x7fac381cf320 auth_method 0 2024-09-16T05:12:13.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3cc5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3cc5e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3cc5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 10 monclient(hunting): get_auth_request con 0x7fac38133ad0 auth_method 0 2024-09-16T05:12:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3cc5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3cc5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3cc5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac20004810 0x7fac38133590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 10 monclient(hunting): get_auth_request con 0x7fac20004810 auth_method 0 2024-09-16T05:12:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 10 monclient(hunting): handle_auth_done global_id 54488 payload 824 2024-09-16T05:12:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 1 -- 172.21.15.114:0/2880549542 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac20004810 msgr2=0x7fac38133590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac20004810 0x7fac38133590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 1 -- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac381cf320 msgr2=0x7fac381cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac381cf320 0x7fac381cf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 10 monclient: _finish_hunting 0 2024-09-16T05:12:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 1 monclient: found mon.noname-c 2024-09-16T05:12:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:12:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 1 -- 172.21.15.114:0/2880549542 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3809ad20 con 0x7fac38133ad0 2024-09-16T05:12:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3cc5e640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac381cf320 0x7fac381cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:12:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac377fe640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac38133ad0 0x7fac38133f30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fac2c000f00 tx=0x7fac20005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 client.?.objecter ms_handle_connect 0x7fac38133ad0 2024-09-16T05:12:13.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:12:13.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 1 -- 172.21.15.114:0/2880549542 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0bc070 con 0x7fac38133ad0 2024-09-16T05:12:13.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:12:13.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 monclient: dump: 2024-09-16T05:12:13.773 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.773 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.773 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.773 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.773 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.773 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.773 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.773 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.774 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.774 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 monclient: _finish_auth 0 2024-09-16T05:12:13.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:12:13.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:13.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 1 -- 172.21.15.114:0/2880549542 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac2c000f80 con 0x7fac38133ad0 2024-09-16T05:12:13.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 5 monclient: authenticate success, global_id 54488 2024-09-16T05:12:13.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient: _renew_subs 2024-09-16T05:12:13.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:13.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 1 -- 172.21.15.114:0/2880549542 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac3809b2e0 con 0x7fac38133ad0 2024-09-16T05:12:13.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 1 -- 172.21.15.114:0/2880549542 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c007d50 con 0x7fac38133ad0 2024-09-16T05:12:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 monclient: dump: 2024-09-16T05:12:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.776 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.776 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.776 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.776 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.776 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.777 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.777 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.777 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac367fc640 4 set_mon_vals no callback set 2024-09-16T05:12:13.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 client.54488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:12:13.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient: _renew_subs 2024-09-16T05:12:13.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:13.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 1 -- 172.21.15.114:0/2880549542 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fac3809b000 con 0x7fac38133ad0 2024-09-16T05:12:13.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 1 librados: init done 2024-09-16T05:12:13.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:13.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac381cfce0 0x7fac3812b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3e6c1640 1 -- 172.21.15.114:0/2880549542 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fac381351c0 con 0x7fac381cfce0 2024-09-16T05:12:13.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac381cfce0 0x7fac3812b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 10 monclient: get_auth_request con 0x7fac381cfce0 auth_method 0 2024-09-16T05:12:13.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 1 -- 172.21.15.114:0/2880549542 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fac2c0bf070 con 0x7fac38133ad0 2024-09-16T05:12:13.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 10 monclient(hunting): handle_auth_done global_id 54491 payload 824 2024-09-16T05:12:13.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 1 -- 172.21.15.114:0/2880549542 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac38138170 con 0x7fac381cfce0 2024-09-16T05:12:13.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 1 --2- 172.21.15.114:0/2880549542 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac24037810 0x7fac24039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 1 -- 172.21.15.114:0/2880549542 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(180..180 src has 1..180) ==== 6023+0+0 (secure 0 0 0) 0x7fac2c0e21c0 con 0x7fac38133ad0 2024-09-16T05:12:13.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 client.54488.objecter ms_dispatch 0x7fac38000ba0 osd_map(180..180 src has 1..180) 2024-09-16T05:12:13.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 3 client.54488.objecter handle_osd_map got epochs [180,180] > 0 2024-09-16T05:12:13.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 3 client.54488.objecter handle_osd_map decoding full epoch 180 2024-09-16T05:12:13.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac37fff640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac381cfce0 0x7fac3812b580 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fac380a2fc0 tx=0x7fac28013ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 20 client.54488.objecter dump_active .. 0 homeless 2024-09-16T05:12:13.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 client.54488.objecter ms_handle_connect 0x7fac381cfce0 2024-09-16T05:12:13.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 client.54488.objecter resend_mon_ops 2024-09-16T05:12:13.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 1 -- 172.21.15.114:0/2880549542 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2801c070 con 0x7fac381cfce0 2024-09-16T05:12:13.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:13.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 monclient: dump: 2024-09-16T05:12:13.784 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.784 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.784 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.784 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.785 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.786 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3cc5e640 1 --2- 172.21.15.114:0/2880549542 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac24037810 0x7fac24039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3cc5e640 10 monclient: get_auth_request con 0x7fac24037810 auth_method 0 2024-09-16T05:12:13.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 1 -- 172.21.15.114:0/2880549542 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fac280158e0 con 0x7fac381cfce0 2024-09-16T05:12:13.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:13.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:12:13.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 1 -- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac381cfce0 msgr2=0x7fac3812b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac381cfce0 0x7fac3812b580 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fac380a2fc0 tx=0x7fac28013ae0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 1 -- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac381cfce0 msgr2=0x7fac3812b580 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:13.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac381cfce0 0x7fac3812b580 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fac380a2fc0 tx=0x7fac28013ae0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac3cc5e640 1 --2- 172.21.15.114:0/2880549542 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac24037810 0x7fac24039cd0 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7fac3806bd50 tx=0x7fac30009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.737+0000 7fac34ff9640 10 client.54488.objecter ms_handle_connect 0x7fac24037810 2024-09-16T05:12:13.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:12:13.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3812bac0 0x7fac3812bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 1 -- 172.21.15.114:0/2880549542 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fac3809c6a0 con 0x7fac3812bac0 2024-09-16T05:12:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac37fff640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3812bac0 0x7fac3812bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac37fff640 10 monclient: get_auth_request con 0x7fac3812bac0 auth_method 0 2024-09-16T05:12:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac37fff640 10 monclient(hunting): handle_auth_done global_id 54494 payload 824 2024-09-16T05:12:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac37fff640 1 -- 172.21.15.114:0/2880549542 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fac38098b60 con 0x7fac3812bac0 2024-09-16T05:12:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac34ff9640 10 client.54488.objecter ms_handle_connect 0x7fac3812bac0 2024-09-16T05:12:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac37fff640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3812bac0 0x7fac3812bea0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fac28008010 tx=0x7fac28004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac34ff9640 10 client.54488.objecter resend_mon_ops 2024-09-16T05:12:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac34ff9640 1 -- 172.21.15.114:0/2880549542 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac280063d0 con 0x7fac3812bac0 2024-09-16T05:12:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac34ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac34ff9640 10 monclient: dump: 2024-09-16T05:12:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.791 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac34ff9640 1 -- 172.21.15.114:0/2880549542 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fac280063d0 con 0x7fac3812bac0 2024-09-16T05:12:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac34ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:12:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac34ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:12:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac34ff9640 1 -- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3812bac0 msgr2=0x7fac3812bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac34ff9640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3812bac0 0x7fac3812bea0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fac28008010 tx=0x7fac28004740 comp rx=0 tx=0).stop 2024-09-16T05:12:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac34ff9640 1 -- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3812bac0 msgr2=0x7fac3812bea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:13.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac34ff9640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac3812bac0 0x7fac3812bea0 secure :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fac28008010 tx=0x7fac28004740 comp rx=0 tx=0).stop 2024-09-16T05:12:13.792 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:12:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:12:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:12:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:12:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 20, 2024-09-16T05:12:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 147881, 2024-09-16T05:12:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:12:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:12:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:12:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:12:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:12:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:12:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:12:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:12:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:12:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:12:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:12:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:12:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:12:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:12:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:12:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:12:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:12:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:12:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:12:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:12:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:12:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:12:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:12:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:12:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:12:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:12:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:12:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:12:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:12:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:12:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:12:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:12:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:12:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:12:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:12:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:12:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:12:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:12:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:12:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:12:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:12:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:12:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:12:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:12:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:12:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:12:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:12:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:12:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:12:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:12:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.805 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:13.805 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:12:13.805 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.805 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:13.805 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:13.805 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:12:13.805 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.805 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.805 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:12:13.805 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:12:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 10 librados: watch_flush enter 2024-09-16T05:12:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 10 librados: watch_flush exit 2024-09-16T05:12:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 20 client.54488.objecter shutdown clearing up homeless session... 2024-09-16T05:12:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 10 client.54488.objecter successfully canceled tick 2024-09-16T05:12:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 1 -- 172.21.15.114:0/2880549542 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac24037810 msgr2=0x7fac24039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 1 --2- 172.21.15.114:0/2880549542 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac24037810 0x7fac24039cd0 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7fac3806bd50 tx=0x7fac30009040 comp rx=0 tx=0).stop 2024-09-16T05:12:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 10 monclient: shutdown 2024-09-16T05:12:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 1 -- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac38133ad0 msgr2=0x7fac38133f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 1 --2- 172.21.15.114:0/2880549542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac38133ad0 0x7fac38133f30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fac2c000f00 tx=0x7fac20005840 comp rx=0 tx=0).stop 2024-09-16T05:12:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3cc5e640 1 -- 172.21.15.114:0/2880549542 reap_dead start 2024-09-16T05:12:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3cc5e640 1 -- 172.21.15.114:0/2880549542 reap_dead start 2024-09-16T05:12:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 1 -- 172.21.15.114:0/2880549542 shutdown_connections 2024-09-16T05:12:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 1 -- 172.21.15.114:0/2880549542 >> 172.21.15.114:0/2880549542 conn(0x7fac380a1fb0 msgr2=0x7fac380a4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 1 -- 172.21.15.114:0/2880549542 shutdown_connections 2024-09-16T05:12:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 1 -- 172.21.15.114:0/2880549542 wait complete. 2024-09-16T05:12:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.745+0000 7fac3e6c1640 1 librados: shutdown 2024-09-16T05:12:13.809 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:12:13.809 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:12:13.809 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:12:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 10 monclient: get_monmap_and_config 2024-09-16T05:12:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 10 monclient: build_initial_monmap 2024-09-16T05:12:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 10 monclient: monmap: 2024-09-16T05:12:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:12:13.925665+0000 2024-09-16T05:12:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:12:13.925665+0000 2024-09-16T05:12:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:12:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:12:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:12:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:12:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:12:13.926 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 1 Processor -- start 2024-09-16T05:12:13.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 1 -- start start 2024-09-16T05:12:13.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 10 monclient: init 2024-09-16T05:12:13.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:13.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-16T05:12:13.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42981376e0 0x7f4298137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 10 monclient(hunting): picked mon.noname-b con 0x7f42981376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:13.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f42981370a0 0x7f42980a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 10 monclient(hunting): picked mon.noname-c con 0x7f42981370a0 addr v2:172.21.15.149:3300/0 2024-09-16T05:12:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42980a5d00 addr v1:172.21.15.149:6789/0 2024-09-16T05:12:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f429809bd20 con 0x7f42980a5d00 2024-09-16T05:12:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f4298138000 con 0x7f42981370a0 2024-09-16T05:12:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f429809c5c0 con 0x7f42981376e0 2024-09-16T05:12:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f429e8a7640 10 monclient(hunting): authenticate will time out at 1218.555630s 2024-09-16T05:12:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f4297fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42981376e0 0x7f4298137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f4297fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42981376e0 0x7f4298137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:59250/0 (socket says 172.21.15.114:59250) 2024-09-16T05:12:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f4297fff640 1 -- 172.21.15.114:0/3086677247 learned_addr learned my addr 172.21.15.114:0/3086677247 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f4297fff640 10 monclient(hunting): get_auth_request con 0x7f42981376e0 auth_method 0 2024-09-16T05:12:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f4297fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f4297fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f4297fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.921+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4297fff640 10 monclient(hunting): handle_auth_done global_id 54497 payload 274 2024-09-16T05:12:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f42977fe640 1 --2- 172.21.15.114:0/3086677247 >> v2:172.21.15.149:3300/0 conn(0x7f42981370a0 0x7f42980a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4297fff640 1 -- 172.21.15.114:0/3086677247 >> v2:172.21.15.149:3300/0 conn(0x7f42981370a0 msgr2=0x7f42980a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4297fff640 1 --2- 172.21.15.114:0/3086677247 >> v2:172.21.15.149:3300/0 conn(0x7f42981370a0 0x7f42980a4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4297fff640 1 -- 172.21.15.114:0/3086677247 >> v1:172.21.15.149:6789/0 conn(0x7f42980a5d00 legacy=0x7f42980a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4297fff640 10 monclient: _finish_hunting 0 2024-09-16T05:12:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4297fff640 1 monclient: found mon.noname-b 2024-09-16T05:12:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4297fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4297fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:12:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4297fff640 1 -- 172.21.15.114:0/3086677247 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f429809c000 con 0x7f42981376e0 2024-09-16T05:12:13.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f42977fe640 10 monclient: get_auth_request con 0x7f42981370a0 auth_method 0 2024-09-16T05:12:13.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f42977fe640 1 --2- 172.21.15.114:0/3086677247 >> v2:172.21.15.149:3300/0 conn(0x7f42981370a0 0x7f42980a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:12:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4297fff640 1 --2- 172.21.15.114:0/3086677247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42981376e0 0x7f4298137ac0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4288009870 tx=0x7f428801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=693622369d297b6b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4296ffd640 1 -- 172.21.15.114:0/3086677247 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42880b0070 con 0x7f42981376e0 2024-09-16T05:12:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4296ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4296ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:12:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4296ffd640 10 monclient: dump: 2024-09-16T05:12:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.939 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4296ffd640 10 monclient: _finish_auth 0 2024-09-16T05:12:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4296ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:12:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4296ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 5 monclient: authenticate success, global_id 54497 2024-09-16T05:12:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4296ffd640 1 -- 172.21.15.114:0/3086677247 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42880ab600 con 0x7f42981376e0 2024-09-16T05:12:13.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:12:13.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4296ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:13.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 10 monclient: get_monmap_and_config success 2024-09-16T05:12:13.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 4 set_mon_vals no callback set 2024-09-16T05:12:13.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4296ffd640 1 -- 172.21.15.114:0/3086677247 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42880ab9d0 con 0x7f42981376e0 2024-09-16T05:12:13.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4296ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4296ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:13.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f4296ffd640 10 monclient: dump: 2024-09-16T05:12:13.940 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.940 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.940 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.940 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.941 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.941 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.941 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.941 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.941 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.941 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 10 monclient: shutdown 2024-09-16T05:12:13.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:13.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 1 -- 172.21.15.114:0/3086677247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42981376e0 msgr2=0x7f4298137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 1 --2- 172.21.15.114:0/3086677247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42981376e0 0x7f4298137ac0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4288009870 tx=0x7f428801c3e0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 1 -- 172.21.15.114:0/3086677247 shutdown_connections 2024-09-16T05:12:13.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 1 --2- 172.21.15.114:0/3086677247 >> v2:172.21.15.149:3300/0 conn(0x7f42981370a0 0x7f42980a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 1 --2- 172.21.15.114:0/3086677247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42981376e0 0x7f4298137ac0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 1 -- 172.21.15.114:0/3086677247 >> 172.21.15.114:0/3086677247 conn(0x7f42980a9a40 msgr2=0x7f42980a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:13.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 1 -- 172.21.15.114:0/3086677247 shutdown_connections 2024-09-16T05:12:13.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 1 -- 172.21.15.114:0/3086677247 wait complete. 2024-09-16T05:12:13.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 10 monclient: build_initial_monmap 2024-09-16T05:12:13.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 10 monclient: monmap: 2024-09-16T05:12:13.942 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:13.942 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:12:13.942 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:12:13.942 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:12:13.942 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:13.942 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.942 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:12:13.942 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:12:13.942 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:12:13.942 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 1 librados: starting msgr at 2024-09-16T05:12:13.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.925+0000 7f429e8a7640 1 librados: starting objecter 2024-09-16T05:12:13.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 1 Processor -- start 2024-09-16T05:12:13.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 1 -- start start 2024-09-16T05:12:13.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 1 librados: setting wanted keys 2024-09-16T05:12:13.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 1 librados: calling monclient init 2024-09-16T05:12:13.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient: init 2024-09-16T05:12:13.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:13.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:12:13.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f42980a5d00 0x7f4298135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient(hunting): picked mon.noname-b con 0x7f42980a5d00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:13.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42981370a0 0x7f4298136020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient(hunting): picked mon.noname-c con 0x7f42981370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:13.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42981376e0 0x7f429813c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient(hunting): picked mon.noname-a con 0x7f42981376e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:13.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4298141d80 con 0x7f42981376e0 2024-09-16T05:12:13.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f4298141c00 con 0x7f42980a5d00 2024-09-16T05:12:13.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4298141f00 con 0x7f42981370a0 2024-09-16T05:12:13.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:13.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient(hunting): authenticate will time out at 1218.560538s 2024-09-16T05:12:13.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42981370a0 0x7f4298136020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42981370a0 0x7f4298136020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:59256/0 (socket says 172.21.15.114:59256) 2024-09-16T05:12:13.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429ce44640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42981376e0 0x7f429813c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 1 -- 172.21.15.114:0/3049114581 learned_addr learned my addr 172.21.15.114:0/3049114581 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:13.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 10 monclient(hunting): get_auth_request con 0x7f42981370a0 auth_method 0 2024-09-16T05:12:13.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429ce44640 10 monclient(hunting): get_auth_request con 0x7f42981376e0 auth_method 0 2024-09-16T05:12:13.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429ce44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429ce44640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429ce44640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429ce44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429ce44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429ce44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4297fff640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f42980a5d00 0x7f4298135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 10 monclient(hunting): handle_auth_done global_id 54500 payload 824 2024-09-16T05:12:13.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 1 -- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f42980a5d00 msgr2=0x7f4298135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f42980a5d00 0x7f4298135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 1 -- 172.21.15.114:0/3049114581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42981376e0 msgr2=0x7f429813c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42981376e0 0x7f429813c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 10 monclient: _finish_hunting 0 2024-09-16T05:12:13.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 1 monclient: found mon.noname-c 2024-09-16T05:12:13.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:13.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:12:13.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 1 -- 172.21.15.114:0/3049114581 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42980a8fc0 con 0x7f42981370a0 2024-09-16T05:12:13.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4297fff640 10 monclient: get_auth_request con 0x7f42980a5d00 auth_method 0 2024-09-16T05:12:13.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429ce44640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42981376e0 0x7f429813c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:12:13.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4297fff640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f42980a5d00 0x7f4298135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:12:13.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42977fe640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42981370a0 0x7f4298136020 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f428c00c7f0 tx=0x7f428c00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 client.?.objecter ms_handle_connect 0x7f42981370a0 2024-09-16T05:12:13.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:12:13.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 1 -- 172.21.15.114:0/3049114581 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f428c015070 con 0x7f42981370a0 2024-09-16T05:12:13.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:12:13.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 monclient: dump: 2024-09-16T05:12:13.947 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.947 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.947 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.947 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.948 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.948 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.948 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.948 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.948 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.948 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 monclient: _finish_auth 0 2024-09-16T05:12:13.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:12:13.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:13.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 5 monclient: authenticate success, global_id 54500 2024-09-16T05:12:13.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient: _renew_subs 2024-09-16T05:12:13.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 1 -- 172.21.15.114:0/3049114581 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f428c007a30 con 0x7f42981370a0 2024-09-16T05:12:13.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:13.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 1 -- 172.21.15.114:0/3049114581 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4298073d20 con 0x7f42981370a0 2024-09-16T05:12:13.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:13.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 1 -- 172.21.15.114:0/3049114581 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f428c007e00 con 0x7f42981370a0 2024-09-16T05:12:13.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:13.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 monclient: dump: 2024-09-16T05:12:13.949 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.949 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.949 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.949 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.949 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.949 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.949 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.949 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.950 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.950 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f42967fc640 4 set_mon_vals no callback set 2024-09-16T05:12:13.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 client.54500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:12:13.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient: _renew_subs 2024-09-16T05:12:13.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:13.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 1 -- 172.21.15.114:0/3049114581 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f429811fec0 con 0x7f42981370a0 2024-09-16T05:12:13.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 1 librados: init done 2024-09-16T05:12:13.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:13.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f429813c570 0x7f429813c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429e8a7640 1 -- 172.21.15.114:0/3049114581 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f42980a8a00 con 0x7f429813c570 2024-09-16T05:12:13.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 1 -- 172.21.15.114:0/3049114581 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f428c018070 con 0x7f42981370a0 2024-09-16T05:12:13.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4297fff640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f429813c570 0x7f429813c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 1 --2- 172.21.15.114:0/3049114581 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4284037810 0x7f4284039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4297fff640 10 monclient: get_auth_request con 0x7f429813c570 auth_method 0 2024-09-16T05:12:13.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4297fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4297fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4297fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 1 -- 172.21.15.114:0/3049114581 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(180..180 src has 1..180) ==== 6023+0+0 (secure 0 0 0) 0x7f428c049000 con 0x7f42981370a0 2024-09-16T05:12:13.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 client.54500.objecter ms_dispatch 0x7f4298000ba0 osd_map(180..180 src has 1..180) 2024-09-16T05:12:13.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 3 client.54500.objecter handle_osd_map got epochs [180,180] > 0 2024-09-16T05:12:13.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 3 client.54500.objecter handle_osd_map decoding full epoch 180 2024-09-16T05:12:13.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 20 client.54500.objecter dump_active .. 0 homeless 2024-09-16T05:12:13.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429ce44640 1 --2- 172.21.15.114:0/3049114581 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4284037810 0x7f4284039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429ce44640 10 monclient: get_auth_request con 0x7f4284037810 auth_method 0 2024-09-16T05:12:13.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4297fff640 10 monclient(hunting): handle_auth_done global_id 64144 payload 824 2024-09-16T05:12:13.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4297fff640 1 -- 172.21.15.114:0/3049114581 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4298137ac0 con 0x7f429813c570 2024-09-16T05:12:13.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 client.54500.objecter ms_handle_connect 0x7f429813c570 2024-09-16T05:12:13.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 client.54500.objecter resend_mon_ops 2024-09-16T05:12:13.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4297fff640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f429813c570 0x7f429813c950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4298073760 tx=0x7f428801ac20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 1 -- 172.21.15.114:0/3049114581 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42880b0040 con 0x7f429813c570 2024-09-16T05:12:13.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:12:13.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 monclient: dump: 2024-09-16T05:12:13.952 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.952 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.953 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.953 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.953 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.953 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.953 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.953 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.953 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.953 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f4294ff9640 10 client.54500.objecter ms_handle_connect 0x7f4284037810 2024-09-16T05:12:13.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.929+0000 7f429ce44640 1 --2- 172.21.15.114:0/3049114581 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4284037810 0x7f4284039cd0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f429000f220 tx=0x7f4290003d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.933+0000 7f4294ff9640 1 -- 172.21.15.114:0/3049114581 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f42880abd80 con 0x7f429813c570 2024-09-16T05:12:13.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.933+0000 7f4294ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:13.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.933+0000 7f4294ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:12:13.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.933+0000 7f4294ff9640 1 -- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f429813c570 msgr2=0x7f429813c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.933+0000 7f4294ff9640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f429813c570 0x7f429813c950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4298073760 tx=0x7f428801ac20 comp rx=0 tx=0).stop 2024-09-16T05:12:13.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.933+0000 7f4294ff9640 1 -- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f429813c570 msgr2=0x7f429813c950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:13.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.933+0000 7f4294ff9640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f429813c570 0x7f429813c950 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f429e8a7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:12:13.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f429e8a7640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f429813ce90 0x7f4298138190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:13.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f429e8a7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:13.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f429e8a7640 1 -- 172.21.15.114:0/3049114581 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f4298120c00 con 0x7f429813ce90 2024-09-16T05:12:13.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4297fff640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f429813ce90 0x7f4298138190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:13.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4297fff640 10 monclient: get_auth_request con 0x7f429813ce90 auth_method 0 2024-09-16T05:12:13.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4297fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:13.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4297fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:13.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4297fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:13.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:13.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:13.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:13.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4297fff640 10 monclient(hunting): handle_auth_done global_id 64147 payload 824 2024-09-16T05:12:13.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4297fff640 1 -- 172.21.15.114:0/3049114581 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4298068820 con 0x7f429813ce90 2024-09-16T05:12:13.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4297fff640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f429813ce90 0x7f4298138190 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f428801cc80 tx=0x7f4288019180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:13.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4294ff9640 10 client.54500.objecter ms_handle_connect 0x7f429813ce90 2024-09-16T05:12:13.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4294ff9640 10 client.54500.objecter resend_mon_ops 2024-09-16T05:12:13.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4294ff9640 1 -- 172.21.15.114:0/3049114581 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42880b0040 con 0x7f429813ce90 2024-09-16T05:12:13.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4294ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:13.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4294ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:12:13.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.937+0000 7f4294ff9640 10 monclient: dump: 2024-09-16T05:12:13.957 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:13.957 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:13.957 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.957 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:13.957 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:13.957 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:13.957 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:13.958 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:13.958 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:13.958 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:13.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f4294ff9640 1 -- 172.21.15.114:0/3049114581 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f42880beb60 con 0x7f429813ce90 2024-09-16T05:12:13.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f4294ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:12:13.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f4294ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:12:13.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f4294ff9640 1 -- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f429813ce90 msgr2=0x7f4298138190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f4294ff9640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f429813ce90 0x7f4298138190 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f428801cc80 tx=0x7f4288019180 comp rx=0 tx=0).stop 2024-09-16T05:12:13.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f4294ff9640 1 -- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f429813ce90 msgr2=0x7f4298138190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:13.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f4294ff9640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f429813ce90 0x7f4298138190 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.958 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:12:13.959 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:12:13.959 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:12:13.959 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:12:13.959 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 20, 2024-09-16T05:12:13.959 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 18075, 2024-09-16T05:12:13.959 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:12:13.959 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:12:13.959 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:12:13.959 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:12:13.959 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:13.959 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 3, 2024-09-16T05:12:13.959 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:13.959 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:12:13.960 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:12:13.960 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:13.960 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:13.960 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:13.960 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:13.960 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:13.960 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:13.960 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:13.960 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:13.960 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:13.960 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:13.960 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:13.960 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:13.961 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:12:13.961 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:12:13.961 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:13.961 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:13.961 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:13.961 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:13.961 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:13.961 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:13.961 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:13.961 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:13.961 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:13.961 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:13.961 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:13.961 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.962 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.962 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:12:13.962 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:12:13.962 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:12:13.962 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:12:13.962 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:12:13.962 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:12:13.962 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:13.962 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:13.962 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:12:13.962 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:12:13.962 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:12:13.962 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:12:13.962 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:12:13.963 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:12:13.963 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:12:13.963 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:13.963 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:12:13.963 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:13.963 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:13.963 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:13.963 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:13.963 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:13.963 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:13.963 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:13.963 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:13.963 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:13.963 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:13.964 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:13.964 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:13.964 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:12:13.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.964 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:12:13.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.964 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:12:13.964 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:12:13.964 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:13.964 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:13.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.964 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:13.964 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:12:13.965 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.965 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.965 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:13.965 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:12:13.965 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.965 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.965 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.965 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:12:13.965 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:12:13.965 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:13.965 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:13.965 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:13.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.966 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:12:13.966 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:12:13.966 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:13.966 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:13.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.966 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:13.966 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:12:13.966 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.966 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:13.967 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:12:13.967 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.967 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.967 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.967 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:12:13.967 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:12:13.967 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:13.967 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:13.967 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:13.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.967 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:12:13.967 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:12:13.968 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:13.968 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:13.968 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.968 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:13.968 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:12:13.968 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.968 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.968 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:13.968 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:12:13.968 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:13.968 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.968 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.969 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:12:13.969 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:12:13.969 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:13.969 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:13.969 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:13.969 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.969 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.969 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:12:13.969 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:12:13.969 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.969 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:13.969 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:13.969 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:12:13.970 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.970 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:13.970 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:12:13.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:13.970 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:13.970 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:13.970 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:12:13.970 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:13.970 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:13.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:13.970 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:12:13.970 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:12:13.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429e8a7640 10 librados: watch_flush enter 2024-09-16T05:12:13.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429e8a7640 10 librados: watch_flush exit 2024-09-16T05:12:13.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429e8a7640 20 client.54500.objecter shutdown clearing up homeless session... 2024-09-16T05:12:13.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429e8a7640 10 client.54500.objecter successfully canceled tick 2024-09-16T05:12:13.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429e8a7640 1 -- 172.21.15.114:0/3049114581 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4284037810 msgr2=0x7f4284039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429e8a7640 1 --2- 172.21.15.114:0/3049114581 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4284037810 0x7f4284039cd0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f429000f220 tx=0x7f4290003d50 comp rx=0 tx=0).stop 2024-09-16T05:12:13.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429e8a7640 10 monclient: shutdown 2024-09-16T05:12:13.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429e8a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:13.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429e8a7640 1 -- 172.21.15.114:0/3049114581 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42981370a0 msgr2=0x7f4298136020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:13.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429e8a7640 1 --2- 172.21.15.114:0/3049114581 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42981370a0 0x7f4298136020 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f428c00c7f0 tx=0x7f428c00ccb0 comp rx=0 tx=0).stop 2024-09-16T05:12:13.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429ce44640 1 -- 172.21.15.114:0/3049114581 reap_dead start 2024-09-16T05:12:13.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429ce44640 1 -- 172.21.15.114:0/3049114581 reap_dead start 2024-09-16T05:12:13.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429e8a7640 1 -- 172.21.15.114:0/3049114581 shutdown_connections 2024-09-16T05:12:13.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429e8a7640 1 -- 172.21.15.114:0/3049114581 >> 172.21.15.114:0/3049114581 conn(0x7f42980a9a40 msgr2=0x7f429809fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:13.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429e8a7640 1 -- 172.21.15.114:0/3049114581 shutdown_connections 2024-09-16T05:12:13.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429e8a7640 1 -- 172.21.15.114:0/3049114581 wait complete. 2024-09-16T05:12:13.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:13.941+0000 7f429e8a7640 1 librados: shutdown 2024-09-16T05:12:13.974 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:12:13.974 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T05:12:13.974 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T05:12:13.974 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T05:12:13.974 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T05:12:13.974 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T05:12:13.974 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:12:13.990 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T05:12:13.990 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:12:13.990 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:12:14.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 10 monclient: get_monmap_and_config 2024-09-16T05:12:14.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 10 monclient: build_initial_monmap 2024-09-16T05:12:14.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 10 monclient: monmap: 2024-09-16T05:12:14.142 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:14.142 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:14.142 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:12:14.142003+0000 2024-09-16T05:12:14.142 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:12:14.142003+0000 2024-09-16T05:12:14.142 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:14.142 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:14.142 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:12:14.142 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:12:14.143 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:12:14.143 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:12:14.143 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:12:14.143 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 1 Processor -- start 2024-09-16T05:12:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 1 -- start start 2024-09-16T05:12:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 10 monclient: init 2024-09-16T05:12:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-16T05:12:14.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effa4071da0 0x7effa4072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 10 monclient(hunting): picked mon.noname-b con 0x7effa4071da0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7effa40a1ab0 addr v1:172.21.15.149:6789/0 2024-09-16T05:12:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7effa4099cc0 addr v1:172.21.15.114:6789/0 2024-09-16T05:12:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effa4099910 con 0x7effa4099cc0 2024-09-16T05:12:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effa40a1870 con 0x7effa40a1ab0 2024-09-16T05:12:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7effa4073e20 con 0x7effa4071da0 2024-09-16T05:12:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effac0ad640 10 monclient(hunting): authenticate will time out at 1218.771222s 2024-09-16T05:12:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effaa64a640 1 -- >> v1:172.21.15.114:6789/0 conn(0x7effa4099cc0 legacy=0x7effa413e890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T05:12:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effa4071da0 0x7effa4072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effa4071da0 0x7effa4072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:59290/0 (socket says 172.21.15.114:59290) 2024-09-16T05:12:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 1 -- 172.21.15.114:0/610205270 learned_addr learned my addr 172.21.15.114:0/610205270 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 10 monclient(hunting): get_auth_request con 0x7effa4071da0 auth_method 0 2024-09-16T05:12:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 10 monclient(hunting): handle_auth_done global_id 54503 payload 274 2024-09-16T05:12:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 1 -- 172.21.15.114:0/610205270 >> v1:172.21.15.149:6789/0 conn(0x7effa40a1ab0 legacy=0x7effa40726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 1 -- 172.21.15.114:0/610205270 >> v1:172.21.15.114:6789/0 conn(0x7effa4099cc0 legacy=0x7effa413e890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:12:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 10 monclient: _finish_hunting 0 2024-09-16T05:12:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 1 monclient: found mon.noname-b 2024-09-16T05:12:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:12:14.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.137+0000 7effa9e49640 1 -- 172.21.15.114:0/610205270 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effa40a1120 con 0x7effa4071da0 2024-09-16T05:12:14.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effa9e49640 1 --2- 172.21.15.114:0/610205270 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effa4071da0 0x7effa4072180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7eff9c009e80 tx=0x7eff9c01c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4e6a490d0e48baa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:14.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effa8e47640 1 -- 172.21.15.114:0/610205270 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff9c0a57b0 con 0x7effa4071da0 2024-09-16T05:12:14.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effa8e47640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:14.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effa8e47640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:12:14.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effa8e47640 10 monclient: dump: 2024-09-16T05:12:14.156 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:14.156 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:14.156 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:14.156 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:14.156 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:14.156 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:14.156 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:14.156 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:14.156 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:14.156 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:14.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effa8e47640 10 monclient: _finish_auth 0 2024-09-16T05:12:14.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effa8e47640 10 monclient: _check_auth_tickets 2024-09-16T05:12:14.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effa8e47640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:14.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effa8e47640 1 -- 172.21.15.114:0/610205270 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff9c0a5950 con 0x7effa4071da0 2024-09-16T05:12:14.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 5 monclient: authenticate success, global_id 54503 2024-09-16T05:12:14.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:12:14.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effa8e47640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:14.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 10 monclient: get_monmap_and_config success 2024-09-16T05:12:14.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 4 set_mon_vals no callback set 2024-09-16T05:12:14.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effa8e47640 1 -- 172.21.15.114:0/610205270 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff9c0a5d20 con 0x7effa4071da0 2024-09-16T05:12:14.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effa8e47640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:14.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effa8e47640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:14.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effa8e47640 10 monclient: dump: 2024-09-16T05:12:14.158 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:14.158 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:14.158 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:14.158 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:14.158 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:14.158 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:14.158 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:14.158 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:14.158 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:14.158 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:14.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 10 monclient: shutdown 2024-09-16T05:12:14.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:14.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 1 -- 172.21.15.114:0/610205270 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effa4071da0 msgr2=0x7effa4072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:14.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 1 --2- 172.21.15.114:0/610205270 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effa4071da0 0x7effa4072180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7eff9c009e80 tx=0x7eff9c01c8d0 comp rx=0 tx=0).stop 2024-09-16T05:12:14.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 1 -- 172.21.15.114:0/610205270 shutdown_connections 2024-09-16T05:12:14.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 1 --2- 172.21.15.114:0/610205270 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effa4071da0 0x7effa4072180 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:14.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 1 -- 172.21.15.114:0/610205270 >> 172.21.15.114:0/610205270 conn(0x7effa40a1fb0 msgr2=0x7effa40a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:14.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 1 -- 172.21.15.114:0/610205270 shutdown_connections 2024-09-16T05:12:14.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 1 -- 172.21.15.114:0/610205270 wait complete. 2024-09-16T05:12:14.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 10 monclient: build_initial_monmap 2024-09-16T05:12:14.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 10 monclient: monmap: 2024-09-16T05:12:14.159 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:14.159 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:12:14.159 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:12:14.159 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:12:14.159 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:14.160 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:14.160 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:12:14.160 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:12:14.160 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:12:14.160 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:14.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 1 librados: starting msgr at 2024-09-16T05:12:14.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 1 librados: starting objecter 2024-09-16T05:12:14.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 1 Processor -- start 2024-09-16T05:12:14.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 1 -- start start 2024-09-16T05:12:14.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 1 librados: setting wanted keys 2024-09-16T05:12:14.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 1 librados: calling monclient init 2024-09-16T05:12:14.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.141+0000 7effac0ad640 10 monclient: init 2024-09-16T05:12:14.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:14.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:12:14.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effa4071da0 0x7effa4098490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:14.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient(hunting): picked mon.noname-a con 0x7effa4071da0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:14.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effa4099cc0 0x7effa40989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:14.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient(hunting): picked mon.noname-c con 0x7effa4099cc0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:14.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7effa40a1ab0 0x7effa41cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:14.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient(hunting): picked mon.noname-b con 0x7effa40a1ab0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:14.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:14.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7effa4142b60 con 0x7effa4071da0 2024-09-16T05:12:14.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:14.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7effa41429e0 con 0x7effa40a1ab0 2024-09-16T05:12:14.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:14.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7effa4142ce0 con 0x7effa4099cc0 2024-09-16T05:12:14.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:14.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient(hunting): authenticate will time out at 1218.776294s 2024-09-16T05:12:14.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9e49640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effa4071da0 msgr2=0x7effa4098490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:12:14.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9e49640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effa4071da0 0x7effa4098490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:12:14.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effa4099cc0 0x7effa40989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:14.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effa4099cc0 0x7effa40989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:59294/0 (socket says 172.21.15.114:59294) 2024-09-16T05:12:14.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 1 -- 172.21.15.114:0/1762199439 learned_addr learned my addr 172.21.15.114:0/1762199439 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:14.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 10 monclient(hunting): get_auth_request con 0x7effa4099cc0 auth_method 0 2024-09-16T05:12:14.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:14.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:14.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:14.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:14.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:14.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:14.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 10 monclient(hunting): handle_auth_done global_id 54506 payload 824 2024-09-16T05:12:14.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 1 -- 172.21.15.114:0/1762199439 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7effa40a1ab0 msgr2=0x7effa41cefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:14.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 1 --2- 172.21.15.114:0/1762199439 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7effa40a1ab0 0x7effa41cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:14.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 1 -- 172.21.15.114:0/1762199439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effa4071da0 msgr2=0x7effa4098490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:12:14.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 1 --2- 172.21.15.114:0/1762199439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effa4071da0 0x7effa4098490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:14.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 10 monclient: _finish_hunting 0 2024-09-16T05:12:14.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 1 monclient: found mon.noname-c 2024-09-16T05:12:14.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:14.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:12:14.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 1 -- 172.21.15.114:0/1762199439 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effa413e5c0 con 0x7effa4099cc0 2024-09-16T05:12:14.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9648640 1 --2- 172.21.15.114:0/1762199439 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effa4099cc0 0x7effa40989d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7eff9800c460 tx=0x7eff9800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:14.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 10 client.?.objecter ms_handle_connect 0x7effa4099cc0 2024-09-16T05:12:14.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:12:14.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 1 -- 172.21.15.114:0/1762199439 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff980052d0 con 0x7effa4099cc0 2024-09-16T05:12:14.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:14.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:12:14.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 10 monclient: dump: 2024-09-16T05:12:14.164 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:14.164 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:14.164 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:14.164 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:14.164 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:14.164 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:14.164 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:14.164 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:14.164 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:14.164 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:14.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 10 monclient: _finish_auth 0 2024-09-16T05:12:14.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:12:14.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:14.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 5 monclient: authenticate success, global_id 54506 2024-09-16T05:12:14.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 1 -- 172.21.15.114:0/1762199439 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff98005490 con 0x7effa4099cc0 2024-09-16T05:12:14.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient: _renew_subs 2024-09-16T05:12:14.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:14.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 1 -- 172.21.15.114:0/1762199439 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effa40a4570 con 0x7effa4099cc0 2024-09-16T05:12:14.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:14.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 1 -- 172.21.15.114:0/1762199439 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff980038b0 con 0x7effa4099cc0 2024-09-16T05:12:14.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:14.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:14.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 10 monclient: dump: 2024-09-16T05:12:14.166 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:14.166 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:14.166 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:14.166 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:14.166 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:14.166 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:14.166 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:14.166 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:14.166 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:14.166 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:14.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 client.54506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:12:14.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient: _renew_subs 2024-09-16T05:12:14.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa3fff640 4 set_mon_vals no callback set 2024-09-16T05:12:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 1 -- 172.21.15.114:0/1762199439 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7effa413e8a0 con 0x7effa4099cc0 2024-09-16T05:12:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 1 librados: init done 2024-09-16T05:12:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 1 -- 172.21.15.114:0/1762199439 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7eff98003a50 con 0x7effa4099cc0 2024-09-16T05:12:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effac0ad640 1 -- 172.21.15.114:0/1762199439 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effa4072180 con 0x7effa4099cc0 2024-09-16T05:12:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 1 --2- 172.21.15.114:0/1762199439 >> v2:172.21.15.149:6812/1185531135 conn(0x7eff900377c0 0x7eff90039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 1 -- 172.21.15.114:0/1762199439 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(180..180 src has 1..180) ==== 6023+0+0 (secure 0 0 0) 0x7eff98048c60 con 0x7effa4099cc0 2024-09-16T05:12:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 10 client.54506.objecter ms_dispatch 0x7effa4000ba0 osd_map(180..180 src has 1..180) 2024-09-16T05:12:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 3 client.54506.objecter handle_osd_map got epochs [180,180] > 0 2024-09-16T05:12:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 3 client.54506.objecter handle_osd_map decoding full epoch 180 2024-09-16T05:12:14.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 20 client.54506.objecter dump_active .. 0 homeless 2024-09-16T05:12:14.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9e49640 1 --2- 172.21.15.114:0/1762199439 >> v2:172.21.15.149:6812/1185531135 conn(0x7eff900377c0 0x7eff90039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:14.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9e49640 10 monclient: get_auth_request con 0x7eff900377c0 auth_method 0 2024-09-16T05:12:14.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa27fc640 10 client.54506.objecter ms_handle_connect 0x7eff900377c0 2024-09-16T05:12:14.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.145+0000 7effa9e49640 1 --2- 172.21.15.114:0/1762199439 >> v2:172.21.15.149:6812/1185531135 conn(0x7eff900377c0 0x7eff90039c80 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0ae000 tx=0x7eff9c0a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:14.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.149+0000 7effa27fc640 1 -- 172.21.15.114:0/1762199439 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7eff980182b0 con 0x7effa4099cc0 2024-09-16T05:12:14.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.149+0000 7effa27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:14.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.149+0000 7effa27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:12:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:12:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:12:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 1 -- 172.21.15.114:0/1762199439 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7effa413c9f0 con 0x7effa4099cc0 2024-09-16T05:12:14.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effa27fc640 1 -- 172.21.15.114:0/1762199439 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7eff98012630 con 0x7effa4099cc0 2024-09-16T05:12:14.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effa27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:12:14.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effa27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:12:14.366 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:12:14.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 10 librados: watch_flush enter 2024-09-16T05:12:14.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 10 librados: watch_flush exit 2024-09-16T05:12:14.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 20 client.54506.objecter shutdown clearing up homeless session... 2024-09-16T05:12:14.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 10 client.54506.objecter successfully canceled tick 2024-09-16T05:12:14.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 1 -- 172.21.15.114:0/1762199439 >> v2:172.21.15.149:6812/1185531135 conn(0x7eff900377c0 msgr2=0x7eff90039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:14.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 1 --2- 172.21.15.114:0/1762199439 >> v2:172.21.15.149:6812/1185531135 conn(0x7eff900377c0 0x7eff90039c80 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0ae000 tx=0x7eff9c0a7070 comp rx=0 tx=0).stop 2024-09-16T05:12:14.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 10 monclient: shutdown 2024-09-16T05:12:14.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 1 -- 172.21.15.114:0/1762199439 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effa4099cc0 msgr2=0x7effa40989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 1 --2- 172.21.15.114:0/1762199439 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effa4099cc0 0x7effa40989d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7eff9800c460 tx=0x7eff9800c920 comp rx=0 tx=0).stop 2024-09-16T05:12:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 1 -- 172.21.15.114:0/1762199439 shutdown_connections 2024-09-16T05:12:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 1 --2- 172.21.15.114:0/1762199439 >> v2:172.21.15.149:6812/1185531135 conn(0x7eff900377c0 0x7eff90039c80 unknown :-1 s=CLOSED pgs=3462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 1 --2- 172.21.15.114:0/1762199439 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7effa40a1ab0 0x7effa41cefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 1 --2- 172.21.15.114:0/1762199439 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effa4099cc0 0x7effa40989d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 1 --2- 172.21.15.114:0/1762199439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effa4071da0 0x7effa4098490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.361+0000 7effac0ad640 1 -- 172.21.15.114:0/1762199439 >> 172.21.15.114:0/1762199439 conn(0x7effa40a1fb0 msgr2=0x7effa4134820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.365+0000 7effac0ad640 1 -- 172.21.15.114:0/1762199439 shutdown_connections 2024-09-16T05:12:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.365+0000 7effac0ad640 1 -- 172.21.15.114:0/1762199439 wait complete. 2024-09-16T05:12:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:14.365+0000 7effac0ad640 1 librados: shutdown 2024-09-16T05:12:17.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:12:17.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 10 monclient: get_monmap_and_config 2024-09-16T05:12:17.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 10 monclient: build_initial_monmap 2024-09-16T05:12:17.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 10 monclient: monmap: 2024-09-16T05:12:17.514 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:17.514 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:17.514 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:12:17.514117+0000 2024-09-16T05:12:17.514 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:12:17.514117+0000 2024-09-16T05:12:17.514 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:17.514 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:17.514 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:12:17.514 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:12:17.514 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:12:17.515 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:12:17.515 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:12:17.515 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:17.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 1 Processor -- start 2024-09-16T05:12:17.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 1 -- start start 2024-09-16T05:12:17.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 10 monclient: init 2024-09-16T05:12:17.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:17.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-16T05:12:17.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f0009b6c0 addr v1:172.21.15.114:6789/0 2024-09-16T05:12:17.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f000a15d0 0x7f0f0013a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:17.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 10 monclient(hunting): picked mon.noname-b con 0x7f0f000a15d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:17.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f000a19e0 addr v1:172.21.15.149:6789/0 2024-09-16T05:12:17.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f0009d4e0 con 0x7f0f0009b6c0 2024-09-16T05:12:17.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f0009f430 con 0x7f0f000a19e0 2024-09-16T05:12:17.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:17.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f0f0009e390 con 0x7f0f000a15d0 2024-09-16T05:12:17.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:17.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0f05b66640 10 monclient(hunting): authenticate will time out at 1222.143271s 2024-09-16T05:12:17.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0eff7fe640 1 -- >> v1:172.21.15.114:6789/0 conn(0x7f0f0009b6c0 legacy=0x7f0f000a0ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T05:12:17.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f000a15d0 0x7f0f0013a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:17.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f000a15d0 0x7f0f0013a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:44952/0 (socket says 172.21.15.114:44952) 2024-09-16T05:12:17.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 1 -- 172.21.15.114:0/444934215 learned_addr learned my addr 172.21.15.114:0/444934215 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:17.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 10 monclient(hunting): get_auth_request con 0x7f0f000a15d0 auth_method 0 2024-09-16T05:12:17.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:17.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:17.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:17.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:17.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:17.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:17.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 10 monclient(hunting): handle_auth_done global_id 54509 payload 274 2024-09-16T05:12:17.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 1 -- 172.21.15.114:0/444934215 >> v1:172.21.15.149:6789/0 conn(0x7f0f000a19e0 legacy=0x7f0f0013cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:17.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 1 -- 172.21.15.114:0/444934215 >> v1:172.21.15.114:6789/0 conn(0x7f0f0009b6c0 legacy=0x7f0f000a0ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:12:17.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 10 monclient: _finish_hunting 0 2024-09-16T05:12:17.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 1 monclient: found mon.noname-b 2024-09-16T05:12:17.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:17.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:12:17.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.509+0000 7f0efeffd640 1 -- 172.21.15.114:0/444934215 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0009c2f0 con 0x7f0f000a15d0 2024-09-16T05:12:17.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0efeffd640 1 --2- 172.21.15.114:0/444934215 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f000a15d0 0x7f0f0013a650 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009870 tx=0x7f0ef001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4613ab7bd27112ad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:17.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0efe7fc640 1 -- 172.21.15.114:0/444934215 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef00b0070 con 0x7f0f000a15d0 2024-09-16T05:12:17.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0efe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:17.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0efe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:12:17.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0efe7fc640 10 monclient: dump: 2024-09-16T05:12:17.525 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:17.525 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:17.525 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:17.525 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:17.525 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:17.525 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:17.525 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:17.525 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:17.526 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:17.526 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:17.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0efe7fc640 10 monclient: _finish_auth 0 2024-09-16T05:12:17.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0efe7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:12:17.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0efe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:17.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0efe7fc640 1 -- 172.21.15.114:0/444934215 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef00ab600 con 0x7f0f000a15d0 2024-09-16T05:12:17.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 5 monclient: authenticate success, global_id 54509 2024-09-16T05:12:17.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:12:17.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0efe7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:17.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 10 monclient: get_monmap_and_config success 2024-09-16T05:12:17.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0efe7fc640 1 -- 172.21.15.114:0/444934215 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef00ab9d0 con 0x7f0f000a15d0 2024-09-16T05:12:17.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0efe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:17.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 4 set_mon_vals no callback set 2024-09-16T05:12:17.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0efe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:17.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0efe7fc640 10 monclient: dump: 2024-09-16T05:12:17.527 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:17.527 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:17.527 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:17.527 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:17.527 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:17.527 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:17.527 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:17.527 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:17.527 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:17.527 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:17.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 10 monclient: shutdown 2024-09-16T05:12:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 1 -- 172.21.15.114:0/444934215 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f000a15d0 msgr2=0x7f0f0013a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 1 --2- 172.21.15.114:0/444934215 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f000a15d0 0x7f0f0013a650 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009870 tx=0x7f0ef001c3c0 comp rx=0 tx=0).stop 2024-09-16T05:12:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 1 -- 172.21.15.114:0/444934215 shutdown_connections 2024-09-16T05:12:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 1 --2- 172.21.15.114:0/444934215 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f000a15d0 0x7f0f0013a650 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 1 -- 172.21.15.114:0/444934215 >> 172.21.15.114:0/444934215 conn(0x7f0f000a1fb0 msgr2=0x7f0f000a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 1 -- 172.21.15.114:0/444934215 shutdown_connections 2024-09-16T05:12:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 1 -- 172.21.15.114:0/444934215 wait complete. 2024-09-16T05:12:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 10 monclient: build_initial_monmap 2024-09-16T05:12:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 10 monclient: monmap: 2024-09-16T05:12:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:12:17.528 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:12:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:12:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:12:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:12:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:12:17.529 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 1 librados: starting msgr at 2024-09-16T05:12:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 1 librados: starting objecter 2024-09-16T05:12:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 1 Processor -- start 2024-09-16T05:12:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 1 -- start start 2024-09-16T05:12:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 1 librados: setting wanted keys 2024-09-16T05:12:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 1 librados: calling monclient init 2024-09-16T05:12:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.513+0000 7f0f05b66640 10 monclient: init 2024-09-16T05:12:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:12:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f0009b6c0 0x7f0f0013dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient(hunting): picked mon.noname-c con 0x7f0f0009b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0f000a15d0 0x7f0f000994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient(hunting): picked mon.noname-b con 0x7f0f000a15d0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f000a19e0 0x7f0f00099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient(hunting): picked mon.noname-a con 0x7f0f000a19e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0f00140c60 con 0x7f0f000a19e0 2024-09-16T05:12:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f0f00140ae0 con 0x7f0f000a15d0 2024-09-16T05:12:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f0f00140de0 con 0x7f0f0009b6c0 2024-09-16T05:12:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient(hunting): authenticate will time out at 1222.148469s 2024-09-16T05:12:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0efffff640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f000a19e0 msgr2=0x7f0f00099a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:12:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0efffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f000a19e0 0x7f0f00099a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:12:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f0009b6c0 0x7f0f0013dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f0009b6c0 0x7f0f0013dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:44962/0 (socket says 172.21.15.114:44962) 2024-09-16T05:12:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 1 -- 172.21.15.114:0/124124554 learned_addr learned my addr 172.21.15.114:0/124124554 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 10 monclient(hunting): get_auth_request con 0x7f0f0009b6c0 auth_method 0 2024-09-16T05:12:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 10 monclient(hunting): handle_auth_done global_id 54512 payload 824 2024-09-16T05:12:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 1 -- 172.21.15.114:0/124124554 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0f000a15d0 msgr2=0x7f0f000994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 1 --2- 172.21.15.114:0/124124554 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0f000a15d0 0x7f0f000994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 1 -- 172.21.15.114:0/124124554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f000a19e0 msgr2=0x7f0f00099a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:12:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 1 --2- 172.21.15.114:0/124124554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f000a19e0 0x7f0f00099a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:12:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 1 monclient: found mon.noname-c 2024-09-16T05:12:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:12:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 1 -- 172.21.15.114:0/124124554 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0009b180 con 0x7f0f0009b6c0 2024-09-16T05:12:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0eff7fe640 1 --2- 172.21.15.114:0/124124554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f0009b6c0 0x7f0f0013dc00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800b550 tx=0x7f0ee800ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 10 client.?.objecter ms_handle_connect 0x7f0f0009b6c0 2024-09-16T05:12:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:12:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 1 -- 172.21.15.114:0/124124554 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8003dd0 con 0x7f0f0009b6c0 2024-09-16T05:12:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:12:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 10 monclient: dump: 2024-09-16T05:12:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:17.534 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 10 monclient: _finish_auth 0 2024-09-16T05:12:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:12:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 1 -- 172.21.15.114:0/124124554 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ee8015070 con 0x7f0f0009b6c0 2024-09-16T05:12:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 1 -- 172.21.15.114:0/124124554 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8005330 con 0x7f0f0009b6c0 2024-09-16T05:12:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 10 monclient: dump: 2024-09-16T05:12:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:17.535 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0efdffb640 4 set_mon_vals no callback set 2024-09-16T05:12:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 5 monclient: authenticate success, global_id 54512 2024-09-16T05:12:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient: _renew_subs 2024-09-16T05:12:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 1 -- 172.21.15.114:0/124124554 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f00097900 con 0x7f0f0009b6c0 2024-09-16T05:12:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 client.54512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:12:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient: _renew_subs 2024-09-16T05:12:17.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 1 -- 172.21.15.114:0/124124554 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f00096450 con 0x7f0f0009b6c0 2024-09-16T05:12:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 1 librados: init done 2024-09-16T05:12:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0f05b66640 1 -- 172.21.15.114:0/124124554 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f000a0ec0 con 0x7f0f0009b6c0 2024-09-16T05:12:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 1 -- 172.21.15.114:0/124124554 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0ee80054d0 con 0x7f0f0009b6c0 2024-09-16T05:12:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 1 --2- 172.21.15.114:0/124124554 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0ee00377c0 0x7f0ee0039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 1 -- 172.21.15.114:0/124124554 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(180..180 src has 1..180) ==== 6023+0+0 (secure 0 0 0) 0x7f0ee8048e90 con 0x7f0f0009b6c0 2024-09-16T05:12:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 10 client.54512.objecter ms_dispatch 0x7f0f00000ba0 osd_map(180..180 src has 1..180) 2024-09-16T05:12:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 3 client.54512.objecter handle_osd_map got epochs [180,180] > 0 2024-09-16T05:12:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 3 client.54512.objecter handle_osd_map decoding full epoch 180 2024-09-16T05:12:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0ef7fff640 20 client.54512.objecter dump_active .. 0 homeless 2024-09-16T05:12:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0efeffd640 1 --2- 172.21.15.114:0/124124554 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0ee00377c0 0x7f0ee0039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.517+0000 7f0efeffd640 10 monclient: get_auth_request con 0x7f0ee00377c0 auth_method 0 2024-09-16T05:12:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.521+0000 7f0ef7fff640 10 client.54512.objecter ms_handle_connect 0x7f0ee00377c0 2024-09-16T05:12:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.521+0000 7f0efeffd640 1 --2- 172.21.15.114:0/124124554 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0ee00377c0 0x7f0ee0039c80 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7f0ef00ad000 tx=0x7f0ef0004740 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.521+0000 7f0ef7fff640 1 -- 172.21.15.114:0/124124554 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f0ee800e340 con 0x7f0f0009b6c0 2024-09-16T05:12:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.521+0000 7f0ef7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.521+0000 7f0ef7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:12:17.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.737+0000 7f0f05b66640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:12:17.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.737+0000 7f0f05b66640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:12:17.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.737+0000 7f0f05b66640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:17.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.737+0000 7f0f05b66640 1 -- 172.21.15.114:0/124124554 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0f0009c6a0 con 0x7f0f0009b6c0 2024-09-16T05:12:17.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.737+0000 7f0ef7fff640 1 -- 172.21.15.114:0/124124554 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f0ee8010480 con 0x7f0f0009b6c0 2024-09-16T05:12:17.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.737+0000 7f0ef7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:12:17.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.737+0000 7f0ef7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:12:17.745 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:12:17.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 10 librados: watch_flush enter 2024-09-16T05:12:17.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 10 librados: watch_flush exit 2024-09-16T05:12:17.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 20 client.54512.objecter shutdown clearing up homeless session... 2024-09-16T05:12:17.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 10 client.54512.objecter successfully canceled tick 2024-09-16T05:12:17.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 1 -- 172.21.15.114:0/124124554 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0ee00377c0 msgr2=0x7f0ee0039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:17.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 1 --2- 172.21.15.114:0/124124554 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0ee00377c0 0x7f0ee0039c80 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7f0ef00ad000 tx=0x7f0ef0004740 comp rx=0 tx=0).stop 2024-09-16T05:12:17.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 10 monclient: shutdown 2024-09-16T05:12:17.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:17.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 1 -- 172.21.15.114:0/124124554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f0009b6c0 msgr2=0x7f0f0013dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:17.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 1 --2- 172.21.15.114:0/124124554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f0009b6c0 0x7f0f0013dc00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800b550 tx=0x7f0ee800ba10 comp rx=0 tx=0).stop 2024-09-16T05:12:17.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 1 -- 172.21.15.114:0/124124554 shutdown_connections 2024-09-16T05:12:17.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 1 --2- 172.21.15.114:0/124124554 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0ee00377c0 0x7f0ee0039c80 unknown :-1 s=CLOSED pgs=3463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:17.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 1 --2- 172.21.15.114:0/124124554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f000a19e0 0x7f0f00099a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:17.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 1 --2- 172.21.15.114:0/124124554 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0f000a15d0 0x7f0f000994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:17.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 1 --2- 172.21.15.114:0/124124554 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f0009b6c0 0x7f0f0013dc00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:17.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 1 -- 172.21.15.114:0/124124554 >> 172.21.15.114:0/124124554 conn(0x7f0f000a1fb0 msgr2=0x7f0f00071330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:17.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 1 -- 172.21.15.114:0/124124554 shutdown_connections 2024-09-16T05:12:17.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 1 -- 172.21.15.114:0/124124554 wait complete. 2024-09-16T05:12:17.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:17.741+0000 7f0f05b66640 1 librados: shutdown 2024-09-16T05:12:20.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:12:20.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 10 monclient: get_monmap_and_config 2024-09-16T05:12:20.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 10 monclient: build_initial_monmap 2024-09-16T05:12:20.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 10 monclient: monmap: 2024-09-16T05:12:20.889 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:20.889 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:20.889 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:12:20.889218+0000 2024-09-16T05:12:20.889 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:12:20.889218+0000 2024-09-16T05:12:20.889 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:20.889 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:20.889 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:12:20.889 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:12:20.889 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:12:20.890 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:12:20.890 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:12:20.890 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:20.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 1 Processor -- start 2024-09-16T05:12:20.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 1 -- start start 2024-09-16T05:12:20.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 10 monclient: init 2024-09-16T05:12:20.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:20.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-16T05:12:20.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59b809b6c0 addr v1:172.21.15.114:6789/0 2024-09-16T05:12:20.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f59b80a15d0 0x7f59b813a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:20.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 10 monclient(hunting): picked mon.noname-c con 0x7f59b80a15d0 addr v2:172.21.15.149:3300/0 2024-09-16T05:12:20.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59b80a19e0 addr v1:172.21.15.149:6789/0 2024-09-16T05:12:20.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59b809d4e0 con 0x7f59b809b6c0 2024-09-16T05:12:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59b809f430 con 0x7f59b80a19e0 2024-09-16T05:12:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f59b809e390 con 0x7f59b80a15d0 2024-09-16T05:12:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bedf5640 10 monclient(hunting): authenticate will time out at 1225.518393s 2024-09-16T05:12:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bcb91640 1 -- >> v1:172.21.15.114:6789/0 conn(0x7f59b809b6c0 legacy=0x7f59b80a0ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T05:12:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f59b80a15d0 0x7f59b813a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f59b80a15d0 0x7f59b813a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.114:41624/0 (socket says 172.21.15.114:41624) 2024-09-16T05:12:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59bd392640 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f59b80a19e0 0x7f59b813cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.114:57788/0 (socket says 172.21.15.114:57788) 2024-09-16T05:12:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 1 -- 172.21.15.114:0/1716699535 learned_addr learned my addr 172.21.15.114:0/1716699535 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 10 monclient(hunting): get_auth_request con 0x7f59b80a15d0 auth_method 0 2024-09-16T05:12:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:20.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 10 monclient(hunting): handle_auth_done global_id 64150 payload 274 2024-09-16T05:12:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 1 -- 172.21.15.114:0/1716699535 >> v1:172.21.15.149:6789/0 conn(0x7f59b80a19e0 legacy=0x7f59b813cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 1 -- 172.21.15.114:0/1716699535 >> v1:172.21.15.114:6789/0 conn(0x7f59b809b6c0 legacy=0x7f59b80a0ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:12:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:12:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 1 monclient: found mon.noname-c 2024-09-16T05:12:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:12:20.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.885+0000 7f59b7fff640 1 -- 172.21.15.114:0/1716699535 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b809c2f0 con 0x7f59b80a15d0 2024-09-16T05:12:20.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59b7fff640 1 --2- 172.21.15.114:0/1716699535 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f59b80a15d0 0x7f59b813a650 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f59ac006fd0 tx=0x7f59ac01c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd1efe6f472c0e09 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:20.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59b77fe640 1 -- 172.21.15.114:0/1716699535 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac0b0070 con 0x7f59b80a15d0 2024-09-16T05:12:20.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:20.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:12:20.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59b77fe640 10 monclient: dump: 2024-09-16T05:12:20.896 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:20.896 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:20.896 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:20.896 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:20.896 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:20.896 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:20.896 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:20.896 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:20.896 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:20.896 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:20.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59b77fe640 10 monclient: _finish_auth 0 2024-09-16T05:12:20.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59b77fe640 10 monclient: _check_auth_tickets 2024-09-16T05:12:20.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:20.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59b77fe640 1 -- 172.21.15.114:0/1716699535 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59ac0ab600 con 0x7f59b80a15d0 2024-09-16T05:12:20.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59bedf5640 5 monclient: authenticate success, global_id 64150 2024-09-16T05:12:20.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59bedf5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:12:20.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59b77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:20.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59bedf5640 10 monclient: get_monmap_and_config success 2024-09-16T05:12:20.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59b77fe640 1 -- 172.21.15.114:0/1716699535 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac0ab9d0 con 0x7f59b80a15d0 2024-09-16T05:12:20.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:20.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59bedf5640 4 set_mon_vals no callback set 2024-09-16T05:12:20.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:12:20.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59b77fe640 10 monclient: dump: 2024-09-16T05:12:20.897 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:20.898 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:20.898 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:20.898 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:20.898 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:20.898 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:20.898 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:20.898 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:20.898 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:20.898 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:20.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59bedf5640 10 monclient: shutdown 2024-09-16T05:12:20.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59bedf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:20.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59bedf5640 1 -- 172.21.15.114:0/1716699535 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f59b80a15d0 msgr2=0x7f59b813a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:20.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59bedf5640 1 --2- 172.21.15.114:0/1716699535 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f59b80a15d0 0x7f59b813a650 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f59ac006fd0 tx=0x7f59ac01c450 comp rx=0 tx=0).stop 2024-09-16T05:12:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59bedf5640 1 -- 172.21.15.114:0/1716699535 shutdown_connections 2024-09-16T05:12:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59bedf5640 1 --2- 172.21.15.114:0/1716699535 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f59b80a15d0 0x7f59b813a650 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59bedf5640 1 -- 172.21.15.114:0/1716699535 >> 172.21.15.114:0/1716699535 conn(0x7f59b80a1fb0 msgr2=0x7f59b80a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59bedf5640 1 -- 172.21.15.114:0/1716699535 shutdown_connections 2024-09-16T05:12:20.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.889+0000 7f59bedf5640 1 -- 172.21.15.114:0/1716699535 wait complete. 2024-09-16T05:12:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient: build_initial_monmap 2024-09-16T05:12:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient: monmap: 2024-09-16T05:12:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:12:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:12:20.903 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:12:20.904 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:20.904 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:20.904 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:12:20.904 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:12:20.904 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:12:20.904 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:20.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 librados: starting msgr at 2024-09-16T05:12:20.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 librados: starting objecter 2024-09-16T05:12:20.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 Processor -- start 2024-09-16T05:12:20.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 -- start start 2024-09-16T05:12:20.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 librados: setting wanted keys 2024-09-16T05:12:20.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 librados: calling monclient init 2024-09-16T05:12:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient: init 2024-09-16T05:12:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:12:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59b809b6c0 0x7f59b80983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient(hunting): picked mon.noname-a con 0x7f59b809b6c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59b80a15d0 0x7f59b8093bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient(hunting): picked mon.noname-c con 0x7f59b80a15d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f59b80a19e0 0x7f59b8094100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient(hunting): picked mon.noname-b con 0x7f59b80a19e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f59b8140c60 con 0x7f59b809b6c0 2024-09-16T05:12:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:20.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f59b8140ae0 con 0x7f59b80a19e0 2024-09-16T05:12:20.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:20.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f59b8140de0 con 0x7f59b80a15d0 2024-09-16T05:12:20.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:20.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient(hunting): authenticate will time out at 1225.525317s 2024-09-16T05:12:20.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bcb91640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59b809b6c0 msgr2=0x7f59b80983c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:12:20.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bcb91640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59b809b6c0 0x7f59b80983c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:12:20.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59b80a15d0 0x7f59b8093bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:20.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59b80a15d0 0x7f59b8093bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:44978/0 (socket says 172.21.15.114:44978) 2024-09-16T05:12:20.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 1 -- 172.21.15.114:0/3650046819 learned_addr learned my addr 172.21.15.114:0/3650046819 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:20.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 10 monclient(hunting): get_auth_request con 0x7f59b80a15d0 auth_method 0 2024-09-16T05:12:20.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:20.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:20.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:20.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:20.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:20.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:20.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bd392640 1 --2- 172.21.15.114:0/3650046819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f59b80a19e0 0x7f59b8094100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:20.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bd392640 10 monclient(hunting): get_auth_request con 0x7f59b80a19e0 auth_method 0 2024-09-16T05:12:20.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bd392640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:20.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bd392640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:20.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bd392640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:20.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 10 monclient(hunting): handle_auth_done global_id 54515 payload 824 2024-09-16T05:12:20.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 1 -- 172.21.15.114:0/3650046819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f59b80a19e0 msgr2=0x7f59b8094100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:20.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 1 --2- 172.21.15.114:0/3650046819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f59b80a19e0 0x7f59b8094100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:20.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 1 -- 172.21.15.114:0/3650046819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59b809b6c0 msgr2=0x7f59b80983c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:12:20.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 1 --2- 172.21.15.114:0/3650046819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59b809b6c0 0x7f59b80983c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:20.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:12:20.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 1 monclient: found mon.noname-c 2024-09-16T05:12:20.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:20.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:12:20.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 1 -- 172.21.15.114:0/3650046819 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b80963c0 con 0x7f59b80a15d0 2024-09-16T05:12:20.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b7fff640 1 --2- 172.21.15.114:0/3650046819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59b80a15d0 0x7f59b8093bc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f59ac097d20 tx=0x7f59ac0191f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:20.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 10 client.?.objecter ms_handle_connect 0x7f59b80a15d0 2024-09-16T05:12:20.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:12:20.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 1 -- 172.21.15.114:0/3650046819 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac0b9030 con 0x7f59b80a15d0 2024-09-16T05:12:20.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:20.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:12:20.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 10 monclient: dump: 2024-09-16T05:12:20.908 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:20.908 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:20.909 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:20.909 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:20.909 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:20.909 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:20.909 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:20.909 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:20.909 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:20.909 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:20.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 10 monclient: _finish_auth 0 2024-09-16T05:12:20.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 10 monclient: _check_auth_tickets 2024-09-16T05:12:20.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:20.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 1 -- 172.21.15.114:0/3650046819 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59ac004020 con 0x7f59b80a15d0 2024-09-16T05:12:20.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 5 monclient: authenticate success, global_id 54515 2024-09-16T05:12:20.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient: _renew_subs 2024-09-16T05:12:20.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:20.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 -- 172.21.15.114:0/3650046819 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59b8097870 con 0x7f59b80a15d0 2024-09-16T05:12:20.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:20.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 1 -- 172.21.15.114:0/3650046819 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac0b0040 con 0x7f59b80a15d0 2024-09-16T05:12:20.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:20.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:20.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 10 monclient: dump: 2024-09-16T05:12:20.913 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:20.913 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:20.913 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:20.913 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:20.913 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:20.913 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:20.913 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:20.913 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:20.913 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:20.913 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:20.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 client.54515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:12:20.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient: _renew_subs 2024-09-16T05:12:20.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:20.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 -- 172.21.15.114:0/3650046819 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f59b8097590 con 0x7f59b80a15d0 2024-09-16T05:12:20.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 librados: init done 2024-09-16T05:12:20.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b6ffd640 4 set_mon_vals no callback set 2024-09-16T05:12:20.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:20.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:20.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:20.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bedf5640 1 -- 172.21.15.114:0/3650046819 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59b80a0ec0 con 0x7f59b80a15d0 2024-09-16T05:12:20.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 1 -- 172.21.15.114:0/3650046819 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f59ac0bc070 con 0x7f59b80a15d0 2024-09-16T05:12:20.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 1 --2- 172.21.15.114:0/3650046819 >> v2:172.21.15.149:6812/1185531135 conn(0x7f59a40377c0 0x7f59a4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:20.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 1 -- 172.21.15.114:0/3650046819 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(180..180 src has 1..180) ==== 6023+0+0 (secure 0 0 0) 0x7f59ac0e2d90 con 0x7f59b80a15d0 2024-09-16T05:12:20.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 10 client.54515.objecter ms_dispatch 0x7f59b8000ba0 osd_map(180..180 src has 1..180) 2024-09-16T05:12:20.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 3 client.54515.objecter handle_osd_map got epochs [180,180] > 0 2024-09-16T05:12:20.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 3 client.54515.objecter handle_osd_map decoding full epoch 180 2024-09-16T05:12:20.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59b57fa640 20 client.54515.objecter dump_active .. 0 homeless 2024-09-16T05:12:20.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bcb91640 1 --2- 172.21.15.114:0/3650046819 >> v2:172.21.15.149:6812/1185531135 conn(0x7f59a40377c0 0x7f59a4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:20.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.893+0000 7f59bcb91640 10 monclient: get_auth_request con 0x7f59a40377c0 auth_method 0 2024-09-16T05:12:20.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.897+0000 7f59b57fa640 10 client.54515.objecter ms_handle_connect 0x7f59a40377c0 2024-09-16T05:12:20.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.897+0000 7f59bcb91640 1 --2- 172.21.15.114:0/3650046819 >> v2:172.21.15.149:6812/1185531135 conn(0x7f59a40377c0 0x7f59a4039c80 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f59b0003d10 tx=0x7f59b0009270 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:20.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.901+0000 7f59b57fa640 1 -- 172.21.15.114:0/3650046819 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f59ac0160d0 con 0x7f59b80a15d0 2024-09-16T05:12:20.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.901+0000 7f59b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:20.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:20.901+0000 7f59b57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:12:21.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.113+0000 7f59bedf5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:12:21.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.113+0000 7f59bedf5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:12:21.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.113+0000 7f59bedf5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:21.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.113+0000 7f59bedf5640 1 -- 172.21.15.114:0/3650046819 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f59b809c6a0 con 0x7f59b80a15d0 2024-09-16T05:12:21.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.113+0000 7f59b57fa640 1 -- 172.21.15.114:0/3650046819 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f59ac003620 con 0x7f59b80a15d0 2024-09-16T05:12:21.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.113+0000 7f59b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:12:21.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.113+0000 7f59b57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:12:21.119 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:12:21.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 10 librados: watch_flush enter 2024-09-16T05:12:21.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 10 librados: watch_flush exit 2024-09-16T05:12:21.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 20 client.54515.objecter shutdown clearing up homeless session... 2024-09-16T05:12:21.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 10 client.54515.objecter successfully canceled tick 2024-09-16T05:12:21.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 1 -- 172.21.15.114:0/3650046819 >> v2:172.21.15.149:6812/1185531135 conn(0x7f59a40377c0 msgr2=0x7f59a4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:21.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 1 --2- 172.21.15.114:0/3650046819 >> v2:172.21.15.149:6812/1185531135 conn(0x7f59a40377c0 0x7f59a4039c80 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f59b0003d10 tx=0x7f59b0009270 comp rx=0 tx=0).stop 2024-09-16T05:12:21.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 10 monclient: shutdown 2024-09-16T05:12:21.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:21.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 1 -- 172.21.15.114:0/3650046819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59b80a15d0 msgr2=0x7f59b8093bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:21.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 1 --2- 172.21.15.114:0/3650046819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59b80a15d0 0x7f59b8093bc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f59ac097d20 tx=0x7f59ac0191f0 comp rx=0 tx=0).stop 2024-09-16T05:12:21.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 1 -- 172.21.15.114:0/3650046819 shutdown_connections 2024-09-16T05:12:21.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 1 --2- 172.21.15.114:0/3650046819 >> v2:172.21.15.149:6812/1185531135 conn(0x7f59a40377c0 0x7f59a4039c80 unknown :-1 s=CLOSED pgs=3464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:21.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 1 --2- 172.21.15.114:0/3650046819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f59b80a19e0 0x7f59b8094100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:21.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 1 --2- 172.21.15.114:0/3650046819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f59b80a15d0 0x7f59b8093bc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:21.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 1 --2- 172.21.15.114:0/3650046819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59b809b6c0 0x7f59b80983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:21.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 1 -- 172.21.15.114:0/3650046819 >> 172.21.15.114:0/3650046819 conn(0x7f59b80a1fb0 msgr2=0x7f59b80994b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:21.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 1 -- 172.21.15.114:0/3650046819 shutdown_connections 2024-09-16T05:12:21.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 1 -- 172.21.15.114:0/3650046819 wait complete. 2024-09-16T05:12:21.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:21.117+0000 7f59bedf5640 1 librados: shutdown 2024-09-16T05:12:24.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:12:24.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 10 monclient: get_monmap_and_config 2024-09-16T05:12:24.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 10 monclient: build_initial_monmap 2024-09-16T05:12:24.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 10 monclient: monmap: 2024-09-16T05:12:24.269 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:24.269 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:24.270 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:12:24.269746+0000 2024-09-16T05:12:24.270 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:12:24.269746+0000 2024-09-16T05:12:24.270 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:24.270 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:24.270 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:12:24.270 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:12:24.270 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:12:24.270 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:12:24.270 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:12:24.270 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:24.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 1 Processor -- start 2024-09-16T05:12:24.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 1 -- start start 2024-09-16T05:12:24.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 10 monclient: init 2024-09-16T05:12:24.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:24.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-16T05:12:24.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe1b0137730 addr v1:172.21.15.149:6789/0 2024-09-16T05:12:24.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe1b00a5d00 0x7fe1b013c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:24.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 10 monclient(hunting): picked mon.noname-a con 0x7fe1b00a5d00 addr v2:172.21.15.114:3300/0 2024-09-16T05:12:24.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe1b013edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:24.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 10 monclient(hunting): picked mon.noname-c con 0x7fe1b013ca00 addr v2:172.21.15.149:3300/0 2024-09-16T05:12:24.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b00a55e0 con 0x7fe1b0137730 2024-09-16T05:12:24.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:24.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe1b009c450 con 0x7fe1b00a5d00 2024-09-16T05:12:24.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:24.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fe1b00a3710 con 0x7fe1b013ca00 2024-09-16T05:12:24.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:24.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b790b640 10 monclient(hunting): authenticate will time out at 1228.899692s 2024-09-16T05:12:24.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b4ea6640 1 -- >> v2:172.21.15.114:3300/0 conn(0x7fe1b00a5d00 msgr2=0x7fe1b013c4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:12:24.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.265+0000 7fe1b4ea6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe1b00a5d00 0x7fe1b013c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:12:24.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.269+0000 7fe1b5ea8640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe1b013edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:24.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.269+0000 7fe1b5ea8640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe1b013edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.114:41642/0 (socket says 172.21.15.114:41642) 2024-09-16T05:12:24.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.269+0000 7fe1b5ea8640 1 -- 172.21.15.114:0/3588784909 learned_addr learned my addr 172.21.15.114:0/3588784909 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:24.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.269+0000 7fe1b5ea8640 10 monclient(hunting): get_auth_request con 0x7fe1b013ca00 auth_method 0 2024-09-16T05:12:24.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.269+0000 7fe1b5ea8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:24.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.269+0000 7fe1b5ea8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:24.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.269+0000 7fe1b5ea8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:24.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.269+0000 7fe1b5ea8640 1 -- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 msgr2=0x7fe1b013edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:12:24.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.269+0000 7fe1b5ea8640 1 -- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 msgr2=0x7fe1b013edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:12:24.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.269+0000 7fe1b5ea8640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe1b013edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:12:24.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.269+0000 7fe1b5ea8640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe1b013edf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:12:24.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.469+0000 7fe1b4ea6640 1 -- 172.21.15.114:0/3588784909 >> v2:172.21.15.114:3300/0 conn(0x7fe1b00a5d00 msgr2=0x7fe1b013c4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:12:24.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.469+0000 7fe1b4ea6640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.114:3300/0 conn(0x7fe1b00a5d00 0x7fe1b013c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:12:24.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.473+0000 7fe1b5ea8640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe1b013edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:24.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.473+0000 7fe1b5ea8640 10 monclient(hunting): get_auth_request con 0x7fe1b013ca00 auth_method 0 2024-09-16T05:12:24.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.473+0000 7fe1b5ea8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:24.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.473+0000 7fe1b5ea8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:24.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.473+0000 7fe1b5ea8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:12:24.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.473+0000 7fe1b5ea8640 1 -- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 msgr2=0x7fe1b013edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:12:24.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.473+0000 7fe1b5ea8640 1 -- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 msgr2=0x7fe1b013edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:12:24.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.473+0000 7fe1b5ea8640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe1b013edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:12:24.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.473+0000 7fe1b5ea8640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe1b013edf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:12:24.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.873+0000 7fe1b4ea6640 1 -- 172.21.15.114:0/3588784909 >> v2:172.21.15.114:3300/0 conn(0x7fe1b00a5d00 msgr2=0x7fe1b013c4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:12:24.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.873+0000 7fe1b4ea6640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.114:3300/0 conn(0x7fe1b00a5d00 0x7fe1b013c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:12:24.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.877+0000 7fe1b5ea8640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe1b013edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:24.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.877+0000 7fe1b5ea8640 10 monclient(hunting): get_auth_request con 0x7fe1b013ca00 auth_method 0 2024-09-16T05:12:24.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.877+0000 7fe1b5ea8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:24.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.877+0000 7fe1b5ea8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:24.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.877+0000 7fe1b5ea8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:12:24.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.877+0000 7fe1b5ea8640 1 -- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 msgr2=0x7fe1b013edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:12:24.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.877+0000 7fe1b5ea8640 1 -- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 msgr2=0x7fe1b013edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:12:24.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.877+0000 7fe1b5ea8640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe1b013edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:12:24.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:24.877+0000 7fe1b5ea8640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe1b013edf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:12:25.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:25.677+0000 7fe1b4ea6640 1 -- 172.21.15.114:0/3588784909 >> v2:172.21.15.114:3300/0 conn(0x7fe1b00a5d00 msgr2=0x7fe1b013c4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:12:25.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:25.677+0000 7fe1b4ea6640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.114:3300/0 conn(0x7fe1b00a5d00 0x7fe1b013c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:12:25.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:25.681+0000 7fe1b5ea8640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe1b013edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:25.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:25.681+0000 7fe1b5ea8640 10 monclient(hunting): get_auth_request con 0x7fe1b013ca00 auth_method 0 2024-09-16T05:12:25.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:25.681+0000 7fe1b5ea8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:25.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:25.681+0000 7fe1b5ea8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:25.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:25.681+0000 7fe1b5ea8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:12:25.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:25.681+0000 7fe1b5ea8640 1 -- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 msgr2=0x7fe1b013edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:12:25.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:25.681+0000 7fe1b5ea8640 1 -- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 msgr2=0x7fe1b013edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:12:25.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:25.681+0000 7fe1b5ea8640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe1b013edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:12:25.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:25.681+0000 7fe1b5ea8640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe1b013edf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:12:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 10 monclient(hunting): tick 2024-09-16T05:12:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:12:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 1 monclient(hunting): continuing hunt 2024-09-16T05:12:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:12:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 1 -- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 msgr2=0x7fe1b013edf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:12:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe1b013edf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 1 -- 172.21.15.114:0/3588784909 >> v2:172.21.15.114:3300/0 conn(0x7fe1b00a5d00 msgr2=0x7fe1b013c4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:12:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.114:3300/0 conn(0x7fe1b00a5d00 0x7fe1b013c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 1 -- 172.21.15.114:0/3588784909 >> v1:172.21.15.149:6789/0 conn(0x7fe1b0137730 legacy=0x7fe1b0137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 10 monclient: _add_conns ranks=[2,0] 2024-09-16T05:12:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 10 monclient(hunting): picked mon.noname-b con 0x7fe1940036c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe194006220 addr v1:172.21.15.114:6789/0 2024-09-16T05:12:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 1 -- 172.21.15.114:0/3588784909 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b009be00 con 0x7fe194006220 2024-09-16T05:12:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.265+0000 7fe1a6ffd640 1 -- 172.21.15.114:0/3588784909 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe1b009c450 con 0x7fe1940036c0 2024-09-16T05:12:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.269+0000 7fe1a6ffd640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.269+0000 7fe1b4ea6640 1 -- 172.21.15.114:0/3588784909 >> v1:172.21.15.114:6789/0 conn(0x7fe194006220 legacy=0x7fe1940086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T05:12:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.269+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.269+0000 7fe1b56a7640 10 monclient(hunting): get_auth_request con 0x7fe1940036c0 auth_method 0 2024-09-16T05:12:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.269+0000 7fe1b56a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.269+0000 7fe1b56a7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.269+0000 7fe1b56a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:27.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.269+0000 7fe1b56a7640 1 -- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 msgr2=0x7fe194005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:12:27.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.269+0000 7fe1b56a7640 1 -- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 msgr2=0x7fe194005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:12:27.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.269+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:12:27.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.269+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:12:27.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.473+0000 7fe1b4ea6640 1 -- 172.21.15.114:0/3588784909 >> v1:172.21.15.114:6789/0 conn(0x7fe194006220 legacy=0x7fe1940086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T05:12:27.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.473+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:27.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.473+0000 7fe1b56a7640 10 monclient(hunting): get_auth_request con 0x7fe1940036c0 auth_method 0 2024-09-16T05:12:27.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.473+0000 7fe1b56a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:27.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.473+0000 7fe1b56a7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:27.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.473+0000 7fe1b56a7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:12:27.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.473+0000 7fe1b56a7640 1 -- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 msgr2=0x7fe194005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:12:27.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.473+0000 7fe1b56a7640 1 -- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 msgr2=0x7fe194005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:12:27.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.473+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:12:27.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.473+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:12:27.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.877+0000 7fe1b4ea6640 1 -- 172.21.15.114:0/3588784909 >> v1:172.21.15.114:6789/0 conn(0x7fe194006220 legacy=0x7fe1940086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T05:12:27.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.877+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:27.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.877+0000 7fe1b56a7640 10 monclient(hunting): get_auth_request con 0x7fe1940036c0 auth_method 0 2024-09-16T05:12:27.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.877+0000 7fe1b56a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:27.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.877+0000 7fe1b56a7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:27.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.877+0000 7fe1b56a7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:12:27.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.877+0000 7fe1b56a7640 1 -- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 msgr2=0x7fe194005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:12:27.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.877+0000 7fe1b56a7640 1 -- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 msgr2=0x7fe194005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:12:27.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.877+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:12:27.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:27.877+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:12:28.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:28.681+0000 7fe1b4ea6640 1 -- 172.21.15.114:0/3588784909 >> v1:172.21.15.114:6789/0 conn(0x7fe194006220 legacy=0x7fe1940086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T05:12:28.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:28.681+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:28.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:28.681+0000 7fe1b56a7640 10 monclient(hunting): get_auth_request con 0x7fe1940036c0 auth_method 0 2024-09-16T05:12:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:28.681+0000 7fe1b56a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:28.681+0000 7fe1b56a7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:28.681+0000 7fe1b56a7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:12:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:28.681+0000 7fe1b56a7640 1 -- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 msgr2=0x7fe194005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:12:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:28.681+0000 7fe1b56a7640 1 -- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 msgr2=0x7fe194005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:12:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:28.681+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:12:28.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:28.681+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:12:30.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 10 monclient(hunting): tick 2024-09-16T05:12:30.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:12:30.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 1 monclient(hunting): continuing hunt 2024-09-16T05:12:30.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:12:30.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 1 -- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 msgr2=0x7fe194005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:12:30.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 1 -- 172.21.15.114:0/3588784909 >> v1:172.21.15.114:6789/0 conn(0x7fe194006220 legacy=0x7fe1940086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:12:30.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-16T05:12:30.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b5ea8640 1 -- 172.21.15.114:0/3588784909 reap_dead start 2024-09-16T05:12:30.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe194009f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 10 monclient(hunting): picked mon.noname-c con 0x7fe1b013ca00 addr v2:172.21.15.149:3300/0 2024-09-16T05:12:30.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 10 monclient(hunting): picked mon.noname-b con 0x7fe1940036c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:30.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe194006690 addr v1:172.21.15.149:6789/0 2024-09-16T05:12:30.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 1 -- 172.21.15.114:0/3588784909 --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe19400b080 con 0x7fe194006690 2024-09-16T05:12:30.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 1 -- 172.21.15.114:0/3588784909 --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fe1b009c450 con 0x7fe1b013ca00 2024-09-16T05:12:30.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 1 -- 172.21.15.114:0/3588784909 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe1b00a3710 con 0x7fe1940036c0 2024-09-16T05:12:30.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1a6ffd640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:30.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 10 monclient(hunting): get_auth_request con 0x7fe1940036c0 auth_method 0 2024-09-16T05:12:30.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:30.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:30.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe194009f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b56a7640 10 monclient(hunting): get_auth_request con 0x7fe1b013ca00 auth_method 0 2024-09-16T05:12:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b56a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b56a7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b56a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b56a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b56a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b56a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 10 monclient(hunting): handle_auth_done global_id 54518 payload 274 2024-09-16T05:12:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 1 -- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 msgr2=0x7fe194009f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:12:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe194009f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 1 -- 172.21.15.114:0/3588784909 >> v1:172.21.15.149:6789/0 conn(0x7fe194006690 legacy=0x7fe19400a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 10 monclient: _finish_hunting 0 2024-09-16T05:12:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 1 monclient: found mon.noname-b 2024-09-16T05:12:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:12:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b4ea6640 1 -- 172.21.15.114:0/3588784909 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b009be00 con 0x7fe1940036c0 2024-09-16T05:12:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.269+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe194009f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:12:30.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1b4ea6640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00a7740 tx=0x7fe1a001cb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ee5f1db393badb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:30.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1a7fff640 1 -- 172.21.15.114:0/3588784909 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a00a59e0 con 0x7fe1940036c0 2024-09-16T05:12:30.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1a7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:12:30.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1a7fff640 10 monclient: dump: 2024-09-16T05:12:30.280 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.280 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.280 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.280 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.280 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.280 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.280 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.280 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.280 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.280 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1a7fff640 10 monclient: _finish_auth 0 2024-09-16T05:12:30.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1a7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:12:30.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1b790b640 5 monclient: authenticate success, global_id 54518 2024-09-16T05:12:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1b790b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:12:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1a7fff640 1 -- 172.21.15.114:0/3588784909 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a00a9070 con 0x7fe1940036c0 2024-09-16T05:12:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1a7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1b790b640 10 monclient: get_monmap_and_config success 2024-09-16T05:12:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1a7fff640 1 -- 172.21.15.114:0/3588784909 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a00a5dc0 con 0x7fe1940036c0 2024-09-16T05:12:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1b790b640 4 set_mon_vals no callback set 2024-09-16T05:12:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.273+0000 7fe1a7fff640 10 monclient: dump: 2024-09-16T05:12:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.281 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.282 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 10 monclient: shutdown 2024-09-16T05:12:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 1 -- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 msgr2=0x7fe194005af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00a7740 tx=0x7fe1a001cb10 comp rx=0 tx=0).stop 2024-09-16T05:12:30.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 1 -- 172.21.15.114:0/3588784909 shutdown_connections 2024-09-16T05:12:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 1 --2- 172.21.15.114:0/3588784909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1940036c0 0x7fe194005af0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 1 --2- 172.21.15.114:0/3588784909 >> v2:172.21.15.149:3300/0 conn(0x7fe1b013ca00 0x7fe194009f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 1 -- 172.21.15.114:0/3588784909 >> 172.21.15.114:0/3588784909 conn(0x7fe1b00a9c70 msgr2=0x7fe1b00aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 1 -- 172.21.15.114:0/3588784909 shutdown_connections 2024-09-16T05:12:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 1 -- 172.21.15.114:0/3588784909 wait complete. 2024-09-16T05:12:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 10 monclient: build_initial_monmap 2024-09-16T05:12:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 10 monclient: monmap: 2024-09-16T05:12:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:12:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:12:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:12:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:30.284 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:12:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:12:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:12:30.285 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 1 librados: starting msgr at 2024-09-16T05:12:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 1 librados: starting objecter 2024-09-16T05:12:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 1 Processor -- start 2024-09-16T05:12:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 1 -- start start 2024-09-16T05:12:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 1 librados: setting wanted keys 2024-09-16T05:12:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 1 librados: calling monclient init 2024-09-16T05:12:30.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.277+0000 7fe1b790b640 10 monclient: init 2024-09-16T05:12:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:12:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe1940036c0 0x7fe1b0136310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient(hunting): picked mon.noname-b con 0x7fe1940036c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe194006690 0x7fe1b0136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient(hunting): picked mon.noname-a con 0x7fe194006690 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b013ca00 0x7fe1b011fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient(hunting): picked mon.noname-c con 0x7fe1b013ca00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b01380a0 con 0x7fe194006690 2024-09-16T05:12:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b0137f20 con 0x7fe1940036c0 2024-09-16T05:12:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe1b00a4a80 con 0x7fe1b013ca00 2024-09-16T05:12:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient(hunting): authenticate will time out at 1234.912802s 2024-09-16T05:12:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b4ea6640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe194006690 msgr2=0x7fe1b0136850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:12:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b4ea6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe194006690 0x7fe1b0136850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:12:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b013ca00 0x7fe1b011fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b013ca00 0x7fe1b011fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:49444/0 (socket says 172.21.15.114:49444) 2024-09-16T05:12:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 1 -- 172.21.15.114:0/353004971 learned_addr learned my addr 172.21.15.114:0/353004971 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 10 monclient(hunting): get_auth_request con 0x7fe1b013ca00 auth_method 0 2024-09-16T05:12:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/353004971 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe1940036c0 0x7fe1b0136310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b56a7640 10 monclient(hunting): get_auth_request con 0x7fe1940036c0 auth_method 0 2024-09-16T05:12:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b56a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b56a7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b56a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b56a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b56a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b56a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:30.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 10 monclient(hunting): handle_auth_done global_id 54521 payload 824 2024-09-16T05:12:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 1 -- 172.21.15.114:0/353004971 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe1940036c0 msgr2=0x7fe1b0136310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 1 --2- 172.21.15.114:0/353004971 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe1940036c0 0x7fe1b0136310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 1 -- 172.21.15.114:0/353004971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe194006690 msgr2=0x7fe1b0136850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:12:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 1 --2- 172.21.15.114:0/353004971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe194006690 0x7fe1b0136850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 10 monclient: _finish_hunting 0 2024-09-16T05:12:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 1 monclient: found mon.noname-c 2024-09-16T05:12:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:12:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 1 -- 172.21.15.114:0/353004971 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b0123cd0 con 0x7fe1b013ca00 2024-09-16T05:12:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b5ea8640 1 --2- 172.21.15.114:0/353004971 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b013ca00 0x7fe1b011fec0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe194009f20 tx=0x7fe1a8014a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 10 client.?.objecter ms_handle_connect 0x7fe1b013ca00 2024-09-16T05:12:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:12:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 1 -- 172.21.15.114:0/353004971 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a801d020 con 0x7fe1b013ca00 2024-09-16T05:12:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:12:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 10 monclient: dump: 2024-09-16T05:12:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.293 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:12:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:12:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 1 -- 172.21.15.114:0/353004971 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a8004440 con 0x7fe1b013ca00 2024-09-16T05:12:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 1 -- 172.21.15.114:0/353004971 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a800f040 con 0x7fe1b013ca00 2024-09-16T05:12:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 10 monclient: dump: 2024-09-16T05:12:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.294 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.295 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.295 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.295 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.295 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.295 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 5 monclient: authenticate success, global_id 54521 2024-09-16T05:12:30.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient: _renew_subs 2024-09-16T05:12:30.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a77fe640 4 set_mon_vals no callback set 2024-09-16T05:12:30.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:30.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 1 -- 172.21.15.114:0/353004971 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b013ae50 con 0x7fe1b013ca00 2024-09-16T05:12:30.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 client.54521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:12:30.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient: _renew_subs 2024-09-16T05:12:30.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:30.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 1 -- 172.21.15.114:0/353004971 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1b0123710 con 0x7fe1b013ca00 2024-09-16T05:12:30.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 1 librados: init done 2024-09-16T05:12:30.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:30.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:30.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:30.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1b790b640 1 -- 172.21.15.114:0/353004971 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1b00a4db0 con 0x7fe1b013ca00 2024-09-16T05:12:30.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 1 -- 172.21.15.114:0/353004971 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe1a8020020 con 0x7fe1b013ca00 2024-09-16T05:12:30.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.281+0000 7fe1a5ffb640 1 --2- 172.21.15.114:0/353004971 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe1940389e0 0x7fe19403aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.285+0000 7fe1a5ffb640 1 -- 172.21.15.114:0/353004971 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(182..182 src has 1..182) ==== 7060+0+0 (secure 0 0 0) 0x7fe1a8049740 con 0x7fe1b013ca00 2024-09-16T05:12:30.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.285+0000 7fe1a5ffb640 10 client.54521.objecter ms_dispatch 0x7fe1b0000ba0 osd_map(182..182 src has 1..182) 2024-09-16T05:12:30.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.285+0000 7fe1a5ffb640 3 client.54521.objecter handle_osd_map got epochs [182,182] > 0 2024-09-16T05:12:30.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.285+0000 7fe1a5ffb640 3 client.54521.objecter handle_osd_map decoding full epoch 182 2024-09-16T05:12:30.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.285+0000 7fe1a5ffb640 20 client.54521.objecter dump_active .. 0 homeless 2024-09-16T05:12:30.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.285+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/353004971 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe1940389e0 0x7fe19403aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.285+0000 7fe1b56a7640 10 monclient: get_auth_request con 0x7fe1940389e0 auth_method 0 2024-09-16T05:12:30.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.285+0000 7fe1b56a7640 1 --2- 172.21.15.114:0/353004971 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe1940389e0 0x7fe19403aea0 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac000f90 tx=0x7fe1ac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:30.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.285+0000 7fe1a5ffb640 10 client.54521.objecter ms_handle_connect 0x7fe1940389e0 2024-09-16T05:12:30.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.289+0000 7fe1a5ffb640 1 -- 172.21.15.114:0/353004971 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fe1a8015360 con 0x7fe1b013ca00 2024-09-16T05:12:30.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.289+0000 7fe1a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:30.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.289+0000 7fe1a5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:12:30.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.513+0000 7fe1b790b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:12:30.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.513+0000 7fe1b790b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:12:30.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.513+0000 7fe1b790b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:30.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.513+0000 7fe1b790b640 1 -- 172.21.15.114:0/353004971 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe1b0068650 con 0x7fe1b013ca00 2024-09-16T05:12:30.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.513+0000 7fe1a5ffb640 1 -- 172.21.15.114:0/353004971 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fe1a8003d30 con 0x7fe1b013ca00 2024-09-16T05:12:30.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.513+0000 7fe1a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:12:30.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.513+0000 7fe1a5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:12:30.520 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:12:30.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 10 librados: watch_flush enter 2024-09-16T05:12:30.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 10 librados: watch_flush exit 2024-09-16T05:12:30.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 20 client.54521.objecter shutdown clearing up homeless session... 2024-09-16T05:12:30.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 10 client.54521.objecter successfully canceled tick 2024-09-16T05:12:30.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 1 -- 172.21.15.114:0/353004971 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe1940389e0 msgr2=0x7fe19403aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 1 --2- 172.21.15.114:0/353004971 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe1940389e0 0x7fe19403aea0 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac000f90 tx=0x7fe1ac009040 comp rx=0 tx=0).stop 2024-09-16T05:12:30.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 10 monclient: shutdown 2024-09-16T05:12:30.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:30.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 1 -- 172.21.15.114:0/353004971 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b013ca00 msgr2=0x7fe1b011fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 1 --2- 172.21.15.114:0/353004971 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b013ca00 0x7fe1b011fec0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe194009f20 tx=0x7fe1a8014a30 comp rx=0 tx=0).stop 2024-09-16T05:12:30.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 1 -- 172.21.15.114:0/353004971 shutdown_connections 2024-09-16T05:12:30.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 1 --2- 172.21.15.114:0/353004971 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe1940389e0 0x7fe19403aea0 unknown :-1 s=CLOSED pgs=3465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 1 --2- 172.21.15.114:0/353004971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe194006690 0x7fe1b0136850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 1 --2- 172.21.15.114:0/353004971 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1b013ca00 0x7fe1b011fec0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 1 --2- 172.21.15.114:0/353004971 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe1940036c0 0x7fe1b0136310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 1 -- 172.21.15.114:0/353004971 >> 172.21.15.114:0/353004971 conn(0x7fe1b00a9c70 msgr2=0x7fe1b01c1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:30.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 1 -- 172.21.15.114:0/353004971 shutdown_connections 2024-09-16T05:12:30.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 1 -- 172.21.15.114:0/353004971 wait complete. 2024-09-16T05:12:30.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.517+0000 7fe1b790b640 1 librados: shutdown 2024-09-16T05:12:30.539 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:12:30.539 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T05:12:30.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 10 monclient: get_monmap_and_config 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 10 monclient: build_initial_monmap 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 10 monclient: monmap: 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:12:30.656603+0000 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:12:30.656603+0000 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 1 Processor -- start 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 1 -- start start 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 10 monclient: init 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a5809b6c0 addr v1:172.21.15.114:6789/0 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a580a15d0 addr v1:172.21.15.149:6789/0 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f3a580a19e0 0x7f3a5813dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 10 monclient(hunting): picked mon.noname-c con 0x7f3a580a19e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a5809d4e0 con 0x7f3a5809b6c0 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a5809f430 con 0x7f3a580a15d0 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f3a5809e390 con 0x7f3a580a19e0 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a5d994640 10 monclient(hunting): authenticate will time out at 1235.285808s 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a56ffd640 1 -- >> v1:172.21.15.114:6789/0 conn(0x7f3a5809b6c0 legacy=0x7f3a580a0ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f3a580a19e0 0x7f3a5813dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f3a580a19e0 0x7f3a5813dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.114:37424/0 (socket says 172.21.15.114:37424) 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 1 -- 172.21.15.114:0/1031849843 learned_addr learned my addr 172.21.15.114:0/1031849843 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 10 monclient(hunting): get_auth_request con 0x7f3a580a19e0 auth_method 0 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 10 monclient(hunting): handle_auth_done global_id 64165 payload 274 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 1 -- 172.21.15.114:0/1031849843 >> v1:172.21.15.149:6789/0 conn(0x7f3a580a15d0 legacy=0x7f3a5813a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 1 -- 172.21.15.114:0/1031849843 >> v1:172.21.15.114:6789/0 conn(0x7f3a5809b6c0 legacy=0x7f3a580a0ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 10 monclient: _finish_hunting 0 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 1 monclient: found mon.noname-c 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.653+0000 7f3a577fe640 1 -- 172.21.15.114:0/1031849843 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5809c2f0 con 0x7f3a580a19e0 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a577fe640 1 --2- 172.21.15.114:0/1031849843 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a580a19e0 0x7f3a5813dd80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3a48009870 tx=0x7f3a4801c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ced389758880411b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a55ffb640 1 -- 172.21.15.114:0/1031849843 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a480b0070 con 0x7f3a580a19e0 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a55ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a55ffb640 10 monclient: dump: 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a55ffb640 10 monclient: _finish_auth 0 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a55ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a55ffb640 1 -- 172.21.15.114:0/1031849843 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a480ab600 con 0x7f3a580a19e0 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a55ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a55ffb640 1 -- 172.21.15.114:0/1031849843 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a480ab9d0 con 0x7f3a580a19e0 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a55ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a55ffb640 10 monclient: dump: 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a5d994640 5 monclient: authenticate success, global_id 64165 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a5d994640 10 monclient: get_monmap_and_config success 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.657+0000 7f3a5d994640 4 set_mon_vals no callback set 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient: shutdown 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 -- 172.21.15.114:0/1031849843 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a580a19e0 msgr2=0x7f3a5813dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 --2- 172.21.15.114:0/1031849843 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a580a19e0 0x7f3a5813dd80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3a48009870 tx=0x7f3a4801c470 comp rx=0 tx=0).stop 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 -- 172.21.15.114:0/1031849843 shutdown_connections 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 --2- 172.21.15.114:0/1031849843 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a580a19e0 0x7f3a5813dd80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 -- 172.21.15.114:0/1031849843 >> 172.21.15.114:0/1031849843 conn(0x7f3a580a1fb0 msgr2=0x7f3a580a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 -- 172.21.15.114:0/1031849843 shutdown_connections 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 -- 172.21.15.114:0/1031849843 wait complete. 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient: build_initial_monmap 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient: monmap: 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 librados: starting msgr at 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 librados: starting objecter 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 Processor -- start 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 -- start start 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 librados: setting wanted keys 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 librados: calling monclient init 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient: init 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a5809b6c0 0x7f3a5813dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient(hunting): picked mon.noname-a con 0x7f3a5809b6c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a580a15d0 0x7f3a58098860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient(hunting): picked mon.noname-c con 0x7f3a580a15d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a580a19e0 0x7f3a580994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient(hunting): picked mon.noname-b con 0x7f3a580a19e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3a58140c60 con 0x7f3a5809b6c0 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f3a58140ae0 con 0x7f3a580a19e0 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f3a58140de0 con 0x7f3a580a15d0 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient(hunting): authenticate will time out at 1235.294011s 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a56ffd640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a5809b6c0 msgr2=0x7f3a5813dd30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a56ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a5809b6c0 0x7f3a5813dd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a580a15d0 0x7f3a58098860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a580a15d0 0x7f3a58098860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:49454/0 (socket says 172.21.15.114:49454) 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 1 -- 172.21.15.114:0/3187693380 learned_addr learned my addr 172.21.15.114:0/3187693380 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 10 monclient(hunting): get_auth_request con 0x7f3a580a15d0 auth_method 0 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a577fe640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a580a19e0 0x7f3a580994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 10 monclient(hunting): handle_auth_done global_id 54524 payload 824 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 1 -- 172.21.15.114:0/3187693380 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a580a19e0 msgr2=0x7f3a580994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a580a19e0 0x7f3a580994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 1 -- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a5809b6c0 msgr2=0x7f3a5813dd30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:12:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a5809b6c0 0x7f3a5813dd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 10 monclient: _finish_hunting 0 2024-09-16T05:12:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 1 monclient: found mon.noname-c 2024-09-16T05:12:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:12:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 1 -- 172.21.15.114:0/3187693380 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a58097d50 con 0x7f3a580a15d0 2024-09-16T05:12:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a577fe640 10 monclient: get_auth_request con 0x7f3a580a19e0 auth_method 0 2024-09-16T05:12:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a577fe640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a580a19e0 0x7f3a580994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:12:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a567fc640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a580a15d0 0x7f3a58098860 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000c430 tx=0x7f3a4000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 10 client.?.objecter ms_handle_connect 0x7f3a580a15d0 2024-09-16T05:12:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:12:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 1 -- 172.21.15.114:0/3187693380 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a40017070 con 0x7f3a580a15d0 2024-09-16T05:12:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 10 monclient: dump: 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 10 monclient: _finish_auth 0 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 5 monclient: authenticate success, global_id 54524 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient: _renew_subs 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 1 -- 172.21.15.114:0/3187693380 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a400052f0 con 0x7f3a580a15d0 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 -- 172.21.15.114:0/3187693380 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a5809b180 con 0x7f3a580a15d0 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 1 -- 172.21.15.114:0/3187693380 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a40003670 con 0x7f3a580a15d0 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 10 monclient: dump: 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a557fa640 4 set_mon_vals no callback set 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 client.54524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient: _renew_subs 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 -- 172.21.15.114:0/3187693380 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a580965c0 con 0x7f3a580a15d0 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 librados: init done 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a58099a20 0x7f3a58099e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a5d994640 1 -- 172.21.15.114:0/3187693380 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f3a5809a340 con 0x7f3a58099a20 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 1 -- 172.21.15.114:0/3187693380 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3a4001a070 con 0x7f3a580a15d0 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a56ffd640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a58099a20 0x7f3a58099e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a56ffd640 10 monclient: get_auth_request con 0x7f3a58099a20 auth_method 0 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.661+0000 7f3a4ffff640 1 --2- 172.21.15.114:0/3187693380 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3a3c0377c0 0x7f3a3c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 1 -- 172.21.15.114:0/3187693380 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(182..182 src has 1..182) ==== 7060+0+0 (secure 0 0 0) 0x7f3a40048890 con 0x7f3a580a15d0 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 10 client.54524.objecter ms_dispatch 0x7f3a58000ba0 osd_map(182..182 src has 1..182) 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 3 client.54524.objecter handle_osd_map got epochs [182,182] > 0 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 3 client.54524.objecter handle_osd_map decoding full epoch 182 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a56ffd640 10 monclient(hunting): handle_auth_done global_id 54527 payload 824 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 20 client.54524.objecter dump_active .. 0 homeless 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a56ffd640 1 -- 172.21.15.114:0/3187693380 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a5806ded0 con 0x7f3a58099a20 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 10 client.54524.objecter ms_handle_connect 0x7f3a58099a20 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 10 client.54524.objecter resend_mon_ops 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a56ffd640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a58099a20 0x7f3a58099e00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3a5000bcc0 tx=0x7f3a50009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a577fe640 1 --2- 172.21.15.114:0/3187693380 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3a3c0377c0 0x7f3a3c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a577fe640 10 monclient: get_auth_request con 0x7f3a3c0377c0 auth_method 0 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 1 -- 172.21.15.114:0/3187693380 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a50014630 con 0x7f3a58099a20 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 10 monclient: dump: 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 1 -- 172.21.15.114:0/3187693380 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3a50014630 con 0x7f3a58099a20 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 1 -- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a58099a20 msgr2=0x7f3a58099e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a58099a20 0x7f3a58099e00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3a5000bcc0 tx=0x7f3a50009f30 comp rx=0 tx=0).stop 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 1 -- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a58099a20 msgr2=0x7f3a58099e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a58099a20 0x7f3a58099e00 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3a5000bcc0 tx=0x7f3a50009f30 comp rx=0 tx=0).stop 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a4ffff640 10 client.54524.objecter ms_handle_connect 0x7f3a3c0377c0 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.665+0000 7f3a577fe640 1 --2- 172.21.15.114:0/3187693380 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3a3c0377c0 0x7f3a3c039c80 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7f3a480ad000 tx=0x7f3a48008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a5d994640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a5d994640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a5808d000 0x7f3a5808d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a5d994640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a5d994640 1 -- 172.21.15.114:0/3187693380 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f3a580a0ec0 con 0x7f3a5808d000 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a56ffd640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a5808d000 0x7f3a5808d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a56ffd640 10 monclient: get_auth_request con 0x7f3a5808d000 auth_method 0 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a56ffd640 10 monclient(hunting): handle_auth_done global_id 54530 payload 824 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a56ffd640 1 -- 172.21.15.114:0/3187693380 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3a580686f0 con 0x7f3a5808d000 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a56ffd640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a5808d000 0x7f3a5808d410 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009fd0 tx=0x7f3a50005db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a4ffff640 10 client.54524.objecter ms_handle_connect 0x7f3a5808d000 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a4ffff640 10 client.54524.objecter resend_mon_ops 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a4ffff640 1 -- 172.21.15.114:0/3187693380 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a50019040 con 0x7f3a5808d000 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a4ffff640 10 monclient: dump: 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a4ffff640 1 -- 172.21.15.114:0/3187693380 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f3a5001b960 con 0x7f3a5808d000 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a4ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a4ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a4ffff640 1 -- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a5808d000 msgr2=0x7f3a5808d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a4ffff640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a5808d000 0x7f3a5808d410 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009fd0 tx=0x7f3a50005db0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a4ffff640 1 -- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a5808d000 msgr2=0x7f3a5808d410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.669+0000 7f3a4ffff640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a5808d000 0x7f3a5808d410 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009fd0 tx=0x7f3a50005db0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 22, 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 164805, 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 1, 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 6 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 7 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 10 librados: watch_flush enter 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 10 librados: watch_flush exit 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 20 client.54524.objecter shutdown clearing up homeless session... 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 10 client.54524.objecter successfully canceled tick 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 1 -- 172.21.15.114:0/3187693380 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3a3c0377c0 msgr2=0x7f3a3c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 1 --2- 172.21.15.114:0/3187693380 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3a3c0377c0 0x7f3a3c039c80 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7f3a480ad000 tx=0x7f3a48008040 comp rx=0 tx=0).stop 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 10 monclient: shutdown 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 1 -- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a580a15d0 msgr2=0x7f3a58098860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a577fe640 1 -- 172.21.15.114:0/3187693380 reap_dead start 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a580a15d0 0x7f3a58098860 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000c430 tx=0x7f3a4000c8f0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 1 -- 172.21.15.114:0/3187693380 shutdown_connections 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 1 --2- 172.21.15.114:0/3187693380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a580a15d0 0x7f3a58098860 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 1 -- 172.21.15.114:0/3187693380 >> 172.21.15.114:0/3187693380 conn(0x7f3a580a1fb0 msgr2=0x7f3a580722b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 1 -- 172.21.15.114:0/3187693380 shutdown_connections 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 1 -- 172.21.15.114:0/3187693380 wait complete. 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.673+0000 7f3a5d994640 1 librados: shutdown 2024-09-16T05:12:30.706 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:12:30.706 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T05:12:30.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:12:30.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.809+0000 7fee37673640 10 monclient: get_monmap_and_config 2024-09-16T05:12:30.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.809+0000 7fee37673640 10 monclient: build_initial_monmap 2024-09-16T05:12:30.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.809+0000 7fee37673640 10 monclient: monmap: 2024-09-16T05:12:30.816 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:30.816 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.816 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:12:30.815832+0000 2024-09-16T05:12:30.816 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:12:30.815832+0000 2024-09-16T05:12:30.816 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:30.816 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.816 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:12:30.816 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:12:30.816 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:12:30.816 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:12:30.816 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:12:30.816 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 1 Processor -- start 2024-09-16T05:12:30.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 1 -- start start 2024-09-16T05:12:30.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 10 monclient: init 2024-09-16T05:12:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-16T05:12:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fee300a0ec0 0x7fee3009dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 10 monclient(hunting): picked mon.noname-a con 0x7fee300a0ec0 addr v2:172.21.15.114:3300/0 2024-09-16T05:12:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee3009e000 addr v1:172.21.15.114:6789/0 2024-09-16T05:12:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fee30139640 0x7fee3013ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 10 monclient(hunting): picked mon.noname-c con 0x7fee30139640 addr v2:172.21.15.149:3300/0 2024-09-16T05:12:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee3009a760 con 0x7fee3009e000 2024-09-16T05:12:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fee300994b0 con 0x7fee300a0ec0 2024-09-16T05:12:30.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fee300a1870 con 0x7fee30139640 2024-09-16T05:12:30.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:30.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee37673640 10 monclient(hunting): authenticate will time out at 1235.444631s 2024-09-16T05:12:30.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee34c0e640 1 -- >> v1:172.21.15.114:6789/0 conn(0x7fee3009e000 legacy=0x7fee3009e410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T05:12:30.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee3540f640 1 -- >> v2:172.21.15.114:3300/0 conn(0x7fee300a0ec0 msgr2=0x7fee3009dac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:12:30.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee3540f640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fee300a0ec0 0x7fee3009dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:12:30.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fee30139640 0x7fee3013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fee30139640 0x7fee3013ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.114:37430/0 (socket says 172.21.15.114:37430) 2024-09-16T05:12:30.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 1 -- 172.21.15.114:0/1310023080 learned_addr learned my addr 172.21.15.114:0/1310023080 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:30.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 10 monclient(hunting): get_auth_request con 0x7fee30139640 auth_method 0 2024-09-16T05:12:30.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:30.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:30.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:30.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 10 monclient(hunting): handle_auth_done global_id 64171 payload 274 2024-09-16T05:12:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 1 -- 172.21.15.114:0/1310023080 >> v2:172.21.15.114:3300/0 conn(0x7fee300a0ec0 msgr2=0x7fee3009dac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:12:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 1 --2- 172.21.15.114:0/1310023080 >> v2:172.21.15.114:3300/0 conn(0x7fee300a0ec0 0x7fee3009dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 1 -- 172.21.15.114:0/1310023080 >> v1:172.21.15.114:6789/0 conn(0x7fee3009e000 legacy=0x7fee3009e410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:12:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 10 monclient: _finish_hunting 0 2024-09-16T05:12:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 1 monclient: found mon.noname-c 2024-09-16T05:12:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:12:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 1 -- 172.21.15.114:0/1310023080 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee30099790 con 0x7fee30139640 2024-09-16T05:12:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.813+0000 7fee35c10640 1 --2- 172.21.15.114:0/1310023080 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee30139640 0x7fee3013ba30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fee300a2fd0 tx=0x7fee2801c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b09e7c34a7618708 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:30.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee27fff640 1 -- 172.21.15.114:0/1310023080 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee280a56f0 con 0x7fee30139640 2024-09-16T05:12:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee27fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:12:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee27fff640 10 monclient: dump: 2024-09-16T05:12:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.829 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee27fff640 10 monclient: _finish_auth 0 2024-09-16T05:12:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee27fff640 10 monclient: _check_auth_tickets 2024-09-16T05:12:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee27fff640 1 -- 172.21.15.114:0/1310023080 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee280a5890 con 0x7fee30139640 2024-09-16T05:12:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee27fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee27fff640 1 -- 172.21.15.114:0/1310023080 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee280a5c60 con 0x7fee30139640 2024-09-16T05:12:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:12:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee27fff640 10 monclient: dump: 2024-09-16T05:12:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.831 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 5 monclient: authenticate success, global_id 64171 2024-09-16T05:12:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient: get_monmap_and_config success 2024-09-16T05:12:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 4 set_mon_vals no callback set 2024-09-16T05:12:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient: shutdown 2024-09-16T05:12:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 -- 172.21.15.114:0/1310023080 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee30139640 msgr2=0x7fee3013ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 --2- 172.21.15.114:0/1310023080 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee30139640 0x7fee3013ba30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fee300a2fd0 tx=0x7fee2801c920 comp rx=0 tx=0).stop 2024-09-16T05:12:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 -- 172.21.15.114:0/1310023080 shutdown_connections 2024-09-16T05:12:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 --2- 172.21.15.114:0/1310023080 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee30139640 0x7fee3013ba30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 --2- 172.21.15.114:0/1310023080 >> v2:172.21.15.114:3300/0 conn(0x7fee300a0ec0 0x7fee3009dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 -- 172.21.15.114:0/1310023080 >> 172.21.15.114:0/1310023080 conn(0x7fee300a1fb0 msgr2=0x7fee300a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:30.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 -- 172.21.15.114:0/1310023080 shutdown_connections 2024-09-16T05:12:30.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 -- 172.21.15.114:0/1310023080 wait complete. 2024-09-16T05:12:30.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient: build_initial_monmap 2024-09-16T05:12:30.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient: monmap: 2024-09-16T05:12:30.832 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:12:30.832 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:12:30.832 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:12:30.832 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:12:30.832 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:12:30.832 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.832 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:12:30.832 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:12:30.833 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:12:30.833 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 librados: starting msgr at 2024-09-16T05:12:30.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 librados: starting objecter 2024-09-16T05:12:30.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 Processor -- start 2024-09-16T05:12:30.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 -- start start 2024-09-16T05:12:30.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 librados: setting wanted keys 2024-09-16T05:12:30.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 librados: calling monclient init 2024-09-16T05:12:30.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient: init 2024-09-16T05:12:30.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient: _reopen_session rank -1 2024-09-16T05:12:30.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:12:30.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee3009e000 0x7fee30097f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient(hunting): picked mon.noname-b con 0x7fee3009e000 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:12:30.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee300a0ec0 0x7fee300984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient(hunting): picked mon.noname-c con 0x7fee300a0ec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:12:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee30139640 0x7fee300716d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient(hunting): picked mon.noname-a con 0x7fee30139640 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:12:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fee3009cb30 con 0x7fee30139640 2024-09-16T05:12:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fee3009c9b0 con 0x7fee3009e000 2024-09-16T05:12:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fee3009ccb0 con 0x7fee300a0ec0 2024-09-16T05:12:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient(hunting): _renew_subs 2024-09-16T05:12:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient(hunting): authenticate will time out at 1235.450359s 2024-09-16T05:12:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee35c10640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee30139640 msgr2=0x7fee300716d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:12:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee35c10640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee30139640 0x7fee300716d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:12:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee300a0ec0 0x7fee300984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee300a0ec0 0x7fee300984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:49486/0 (socket says 172.21.15.114:49486) 2024-09-16T05:12:30.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 1 -- 172.21.15.114:0/4155636730 learned_addr learned my addr 172.21.15.114:0/4155636730 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:12:30.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 10 monclient(hunting): get_auth_request con 0x7fee300a0ec0 auth_method 0 2024-09-16T05:12:30.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:30.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:30.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:30.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:30.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:30.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:30.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee3540f640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee3009e000 0x7fee30097f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 10 monclient(hunting): handle_auth_done global_id 54533 payload 824 2024-09-16T05:12:30.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 1 -- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee3009e000 msgr2=0x7fee30097f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee3009e000 0x7fee30097f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 1 -- 172.21.15.114:0/4155636730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee30139640 msgr2=0x7fee300716d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:12:30.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee30139640 0x7fee300716d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 10 monclient: _finish_hunting 0 2024-09-16T05:12:30.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 1 monclient: found mon.noname-c 2024-09-16T05:12:30.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:12:30.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:12:30.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 1 -- 172.21.15.114:0/4155636730 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee3009c6a0 con 0x7fee300a0ec0 2024-09-16T05:12:30.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee3540f640 10 monclient: get_auth_request con 0x7fee3009e000 auth_method 0 2024-09-16T05:12:30.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee3540f640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee3009e000 0x7fee30097f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:12:30.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee34c0e640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee300a0ec0 0x7fee300984a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00b580 tx=0x7fee2c00ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:30.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee25ffb640 10 client.?.objecter ms_handle_connect 0x7fee300a0ec0 2024-09-16T05:12:30.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee25ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:12:30.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee25ffb640 1 -- 172.21.15.114:0/4155636730 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c013020 con 0x7fee300a0ec0 2024-09-16T05:12:30.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee25ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:12:30.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee25ffb640 10 monclient: dump: 2024-09-16T05:12:30.837 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.837 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.837 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.837 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.837 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.837 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.837 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.837 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.837 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.837 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee25ffb640 10 monclient: _finish_auth 0 2024-09-16T05:12:30.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee25ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:12:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:12:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 5 monclient: authenticate success, global_id 54533 2024-09-16T05:12:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient: _renew_subs 2024-09-16T05:12:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 -- 172.21.15.114:0/4155636730 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee3008e3f0 con 0x7fee300a0ec0 2024-09-16T05:12:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee25ffb640 1 -- 172.21.15.114:0/4155636730 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee2c003e40 con 0x7fee300a0ec0 2024-09-16T05:12:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee25ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:12:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee25ffb640 1 -- 172.21.15.114:0/4155636730 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c0052b0 con 0x7fee300a0ec0 2024-09-16T05:12:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 client.54533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:12:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee25ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:12:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee25ffb640 10 monclient: dump: 2024-09-16T05:12:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.839 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient: _renew_subs 2024-09-16T05:12:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:12:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 -- 172.21.15.114:0/4155636730 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fee3013eac0 con 0x7fee300a0ec0 2024-09-16T05:12:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee26ffd640 4 set_mon_vals no callback set 2024-09-16T05:12:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 librados: init done 2024-09-16T05:12:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:30.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee30071c10 0x7fee30071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.817+0000 7fee37673640 1 -- 172.21.15.114:0/4155636730 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fee30069390 con 0x7fee30071c10 2024-09-16T05:12:30.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 1 -- 172.21.15.114:0/4155636730 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fee2c016020 con 0x7fee300a0ec0 2024-09-16T05:12:30.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 1 --2- 172.21.15.114:0/4155636730 >> v2:172.21.15.149:6812/1185531135 conn(0x7fee140377c0 0x7fee14039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 1 -- 172.21.15.114:0/4155636730 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(182..182 src has 1..182) ==== 7060+0+0 (secure 0 0 0) 0x7fee2c049280 con 0x7fee300a0ec0 2024-09-16T05:12:30.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 10 client.54533.objecter ms_dispatch 0x7fee30000ba0 osd_map(182..182 src has 1..182) 2024-09-16T05:12:30.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 3 client.54533.objecter handle_osd_map got epochs [182,182] > 0 2024-09-16T05:12:30.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 3 client.54533.objecter handle_osd_map decoding full epoch 182 2024-09-16T05:12:30.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee3540f640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee30071c10 0x7fee30071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee3540f640 10 monclient: get_auth_request con 0x7fee30071c10 auth_method 0 2024-09-16T05:12:30.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee3540f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:30.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee3540f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:30.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee3540f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:30.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 20 client.54533.objecter dump_active .. 0 homeless 2024-09-16T05:12:30.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee3540f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:30.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee3540f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:30.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee3540f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:30.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee35c10640 1 --2- 172.21.15.114:0/4155636730 >> v2:172.21.15.149:6812/1185531135 conn(0x7fee140377c0 0x7fee14039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee35c10640 10 monclient: get_auth_request con 0x7fee140377c0 auth_method 0 2024-09-16T05:12:30.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee3540f640 10 monclient(hunting): handle_auth_done global_id 64174 payload 824 2024-09-16T05:12:30.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee3540f640 1 -- 172.21.15.114:0/4155636730 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee30072530 con 0x7fee30071c10 2024-09-16T05:12:30.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 10 client.54533.objecter ms_handle_connect 0x7fee30071c10 2024-09-16T05:12:30.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 10 client.54533.objecter resend_mon_ops 2024-09-16T05:12:30.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee3540f640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee30071c10 0x7fee30071ff0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fee2000c6f0 tx=0x7fee2000cbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:30.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 10 client.54533.objecter ms_handle_connect 0x7fee140377c0 2024-09-16T05:12:30.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee35c10640 1 --2- 172.21.15.114:0/4155636730 >> v2:172.21.15.149:6812/1185531135 conn(0x7fee140377c0 0x7fee14039c80 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7fee280ae000 tx=0x7fee280a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:30.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 1 -- 172.21.15.114:0/4155636730 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20005c50 con 0x7fee30071c10 2024-09-16T05:12:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:12:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 10 monclient: dump: 2024-09-16T05:12:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.842 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 1 -- 172.21.15.114:0/4155636730 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fee20005c50 con 0x7fee30071c10 2024-09-16T05:12:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:12:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:12:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 1 -- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee30071c10 msgr2=0x7fee30071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee30071c10 0x7fee30071ff0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fee2000c6f0 tx=0x7fee2000cbb0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 1 -- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee30071c10 msgr2=0x7fee30071ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.821+0000 7fee25ffb640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee30071c10 0x7fee30071ff0 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fee2000c6f0 tx=0x7fee2000cbb0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:12:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee3006d990 0x7fee3006dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:12:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 10 monclient(hunting): start opening mon connection 2024-09-16T05:12:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 1 -- 172.21.15.114:0/4155636730 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fee300a3490 con 0x7fee3006d990 2024-09-16T05:12:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee3540f640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee3006d990 0x7fee3006dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:12:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee3540f640 10 monclient: get_auth_request con 0x7fee3006d990 auth_method 0 2024-09-16T05:12:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee3540f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:12:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee3540f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:12:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee3540f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:12:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee3540f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:12:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee3540f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:12:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee3540f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:12:30.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee3540f640 10 monclient(hunting): handle_auth_done global_id 64177 payload 824 2024-09-16T05:12:30.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee3540f640 1 -- 172.21.15.114:0/4155636730 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fee3009e410 con 0x7fee3006d990 2024-09-16T05:12:30.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee3540f640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee3006d990 0x7fee3006dda0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fee20000c10 tx=0x7fee200059e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:12:30.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee25ffb640 10 client.54533.objecter ms_handle_connect 0x7fee3006d990 2024-09-16T05:12:30.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee25ffb640 10 client.54533.objecter resend_mon_ops 2024-09-16T05:12:30.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee25ffb640 1 -- 172.21.15.114:0/4155636730 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20005e00 con 0x7fee3006d990 2024-09-16T05:12:30.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:12:30.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee25ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:12:30.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee25ffb640 10 monclient: dump: 2024-09-16T05:12:30.846 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:12:30.846 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:12:30.846 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.846 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:12:30.846 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:12:30.847 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:12:30.847 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:12:30.847 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:12:30.847 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:12:30.847 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:12:30.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee25ffb640 1 -- 172.21.15.114:0/4155636730 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7fee20005e00 con 0x7fee3006d990 2024-09-16T05:12:30.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee25ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:12:30.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee25ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:12:30.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee25ffb640 1 -- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee3006d990 msgr2=0x7fee3006dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee25ffb640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee3006d990 0x7fee3006dda0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fee20000c10 tx=0x7fee200059e0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee25ffb640 1 -- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee3006d990 msgr2=0x7fee3006dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:12:30.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee25ffb640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee3006d990 0x7fee3006dda0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fee20000c10 tx=0x7fee200059e0 comp rx=0 tx=0).stop 2024-09-16T05:12:30.848 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:12:30.848 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:12:30.848 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:12:30.848 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:12:30.848 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 22, 2024-09-16T05:12:30.848 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 34963, 2024-09-16T05:12:30.848 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:12:30.848 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:12:30.848 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:12:30.848 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:30.848 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 1, 2024-09-16T05:12:30.848 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:30.849 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:12:30.849 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:12:30.849 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:30.849 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:30.849 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:30.849 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:30.849 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:30.849 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:30.849 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:30.849 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:30.849 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:30.849 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:30.849 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:30.849 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:30.850 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:12:30.850 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:12:30.850 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:30.850 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:30.850 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:30.850 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:30.850 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:30.850 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:30.850 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:30.850 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:30.850 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:30.850 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:30.850 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:30.850 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:30.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.851 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:12:30.851 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:12:30.851 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:12:30.851 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:12:30.851 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:12:30.851 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:12:30.851 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:30.851 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:12:30.851 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:12:30.851 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:12:30.851 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:12:30.851 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:12:30.852 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:12:30.852 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:12:30.852 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:12:30.852 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:12:30.852 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:12:30.852 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:12:30.852 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:12:30.852 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:12:30.852 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:12:30.852 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:12:30.852 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:12:30.852 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:12:30.852 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:12:30.852 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:12:30.853 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:12:30.853 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:12:30.853 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:30.853 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:12:30.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.853 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:12:30.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.853 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:12:30.853 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:12:30.853 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:30.853 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:30.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.853 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:30.853 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:12:30.854 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:30.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.854 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.854 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:30.854 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:12:30.854 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:30.854 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:30.854 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:30.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.854 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:12:30.854 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:12:30.854 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:30.854 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:30.855 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:30.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.855 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.855 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:12:30.855 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:12:30.855 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:30.855 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:30.855 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.855 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:30.855 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:12:30.855 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:30.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.855 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.855 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:30.856 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:12:30.856 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:30.856 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:30.856 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:30.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.856 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:12:30.856 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:12:30.856 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:30.856 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:30.856 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:30.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.856 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.856 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:12:30.856 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:12:30.857 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:12:30.857 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:12:30.857 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.857 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:12:30.857 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:12:30.857 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:30.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.857 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.857 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:12:30.857 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:12:30.857 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:12:30.857 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:30.857 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:30.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.858 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:12:30.858 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:12:30.858 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:12:30.858 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:12:30.858 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:12:30.858 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:30.858 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:30.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.858 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:12:30.858 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:12:30.858 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.858 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:30.858 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:30.858 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:12:30.859 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:30.859 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:12:30.859 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:12:30.859 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:12:30.859 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:12:30.859 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:12:30.859 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:12:30.859 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:12:30.859 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:12:30.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:12:30.859 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:12:30.859 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:12:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 10 librados: watch_flush enter 2024-09-16T05:12:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 10 librados: watch_flush exit 2024-09-16T05:12:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 20 client.54533.objecter shutdown clearing up homeless session... 2024-09-16T05:12:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 10 client.54533.objecter successfully canceled tick 2024-09-16T05:12:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 1 -- 172.21.15.114:0/4155636730 >> v2:172.21.15.149:6812/1185531135 conn(0x7fee140377c0 msgr2=0x7fee14039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 1 --2- 172.21.15.114:0/4155636730 >> v2:172.21.15.149:6812/1185531135 conn(0x7fee140377c0 0x7fee14039c80 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7fee280ae000 tx=0x7fee280a7070 comp rx=0 tx=0).stop 2024-09-16T05:12:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 10 monclient: shutdown 2024-09-16T05:12:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:12:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 1 -- 172.21.15.114:0/4155636730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee300a0ec0 msgr2=0x7fee300984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:12:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 1 --2- 172.21.15.114:0/4155636730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fee300a0ec0 0x7fee300984a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00b580 tx=0x7fee2c00ba40 comp rx=0 tx=0).stop 2024-09-16T05:12:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee35c10640 1 -- 172.21.15.114:0/4155636730 reap_dead start 2024-09-16T05:12:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee35c10640 1 -- 172.21.15.114:0/4155636730 reap_dead start 2024-09-16T05:12:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 1 -- 172.21.15.114:0/4155636730 shutdown_connections 2024-09-16T05:12:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 1 -- 172.21.15.114:0/4155636730 >> 172.21.15.114:0/4155636730 conn(0x7fee300a1fb0 msgr2=0x7fee3009fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:12:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 1 -- 172.21.15.114:0/4155636730 shutdown_connections 2024-09-16T05:12:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 1 -- 172.21.15.114:0/4155636730 wait complete. 2024-09-16T05:12:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:12:30.829+0000 7fee37673640 1 librados: shutdown 2024-09-16T05:12:30.862 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:12:30.862 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:13:00.909 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-09-16T05:13:00.909 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-09-16T05:13:00.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-09-16T05:13:00.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: seed 29623 2024-09-16T05:13:00.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-09-16T05:13:00.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (273 ms) 2024-09-16T05:13:00.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-09-16T05:13:00.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10405 ms) 2024-09-16T05:13:00.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-09-16T05:13:00.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (110 ms) 2024-09-16T05:13:00.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (10788 ms total) 2024-09-16T05:13:00.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: 2024-09-16T05:13:00.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-09-16T05:13:00.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-09-16T05:13:00.910 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (6958 ms) 2024-09-16T05:13:00.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-09-16T05:13:00.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8039 ms) 2024-09-16T05:13:00.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-09-16T05:13:00.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10007 ms) 2024-09-16T05:13:00.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-09-16T05:13:00.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: my_snaps [3] 2024-09-16T05:13:00.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: my_snaps [4,3] 2024-09-16T05:13:00.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: my_snaps [5,4,3] 2024-09-16T05:13:00.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-09-16T05:13:00.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: promoting some heads 2024-09-16T05:13:00.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: promoting from clones for snap 6 2024-09-16T05:13:00.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: promoting from clones for snap 5 2024-09-16T05:13:00.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: promoting from clones for snap 4 2024-09-16T05:13:00.911 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: promoting from clones for snap 3 2024-09-16T05:13:00.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: waiting for scrubs... 2024-09-16T05:13:00.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: done waiting 2024-09-16T05:13:00.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (61022 ms) 2024-09-16T05:13:00.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-09-16T05:13:00.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10045 ms) 2024-09-16T05:13:00.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-09-16T05:13:00.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7954 ms) 2024-09-16T05:13:00.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-09-16T05:13:00.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8056 ms) 2024-09-16T05:13:00.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-09-16T05:13:00.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8018 ms) 2024-09-16T05:13:00.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-09-16T05:13:00.912 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (9923 ms) 2024-09-16T05:13:00.913 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-09-16T05:13:00.913 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9008 ms) 2024-09-16T05:13:00.913 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-09-16T05:13:00.913 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9996 ms) 2024-09-16T05:13:00.913 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-09-16T05:13:00.913 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13087 ms) 2024-09-16T05:13:00.913 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-09-16T05:13:00.913 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7942 ms) 2024-09-16T05:13:00.913 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-09-16T05:13:00.913 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (36747 ms) 2024-09-16T05:13:00.913 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-09-16T05:13:00.913 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13026 ms) 2024-09-16T05:13:00.913 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-09-16T05:13:00.913 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7976 ms) 2024-09-16T05:13:00.914 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-09-16T05:13:00.914 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8001 ms) 2024-09-16T05:13:01.962 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSe 2024-09-16T05:13:01.962 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6943 ms) 2024-09-16T05:13:01.963 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-09-16T05:13:01.963 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7026 ms) 2024-09-16T05:13:01.963 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-16T05:13:01.963 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6976 ms) 2024-09-16T05:13:01.963 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-16T05:13:01.963 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7124 ms) 2024-09-16T05:13:01.963 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-16T05:13:01.963 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6892 ms) 2024-09-16T05:13:01.963 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-16T05:13:01.963 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (6992 ms) 2024-09-16T05:13:01.963 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-16T05:13:01.963 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7064 ms) 2024-09-16T05:13:01.963 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-16T05:13:01.963 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6962 ms) 2024-09-16T05:13:01.964 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-16T05:13:01.964 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6995 ms) 2024-09-16T05:13:01.964 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-16T05:13:01.964 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7042 ms) 2024-09-16T05:13:01.964 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-16T05:13:01.964 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (20941 ms) 2024-09-16T05:13:01.964 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-16T05:13:01.964 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (21781 ms) 2024-09-16T05:13:01.964 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-16T05:13:01.964 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6985 ms) 2024-09-16T05:13:01.964 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-16T05:13:01.964 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7050 ms) 2024-09-16T05:13:01.964 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-16T05:13:01.965 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6983 ms) 2024-09-16T05:13:01.965 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-16T05:13:01.965 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6992 ms) 2024-09-16T05:13:01.965 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (140750 ms total) 2024-09-16T05:13:01.965 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: 2024-09-16T05:13:01.965 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [----------] Global test environment tear-down 2024-09-16T05:13:01.965 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (252635 ms total) 2024-09-16T05:13:01.965 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio: [ PASSED ] 42 tests. 2024-09-16T05:13:16.675 INFO:tasks.workunit.client.0.smithi149.stdout:etRead 2024-09-16T05:13:16.675 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-16T05:13:16.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: ok, hit_set contains 268:602f83fe:::foo:head 2024-09-16T05:13:16.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9036 ms) 2024-09-16T05:13:16.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-09-16T05:13:16.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg_num = 32 2024-09-16T05:13:16.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 0 ls 1726463596,0 2024-09-16T05:13:16.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 1 ls 1726463596,0 2024-09-16T05:13:16.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 2 ls 1726463596,0 2024-09-16T05:13:16.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 3 ls 1726463596,0 2024-09-16T05:13:16.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 4 ls 1726463596,0 2024-09-16T05:13:16.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 5 ls 1726463596,0 2024-09-16T05:13:16.676 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 6 ls 1726463596,0 2024-09-16T05:13:16.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 7 ls 1726463596,0 2024-09-16T05:13:16.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 8 ls 1726463596,0 2024-09-16T05:13:16.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 9 ls 1726463596,0 2024-09-16T05:13:16.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 10 ls 1726463596,0 2024-09-16T05:13:16.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 11 ls 1726463596,0 2024-09-16T05:13:16.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 12 ls 1726463596,0 2024-09-16T05:13:16.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 13 ls 1726463596,0 2024-09-16T05:13:16.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 14 ls 1726463596,0 2024-09-16T05:13:16.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 15 ls 1726463596,0 2024-09-16T05:13:16.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 16 ls 1726463596,0 2024-09-16T05:13:16.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 17 ls 1726463596,0 2024-09-16T05:13:16.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 18 ls 1726463596,0 2024-09-16T05:13:16.677 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 19 ls 1726463596,0 2024-09-16T05:13:16.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 20 ls 1726463596,0 2024-09-16T05:13:16.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 21 ls 1726463596,0 2024-09-16T05:13:16.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 22 ls 1726463596,0 2024-09-16T05:13:16.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 23 ls 1726463596,0 2024-09-16T05:13:16.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 24 ls 1726463596,0 2024-09-16T05:13:16.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 25 ls 1726463596,0 2024-09-16T05:13:16.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 26 ls 1726463596,0 2024-09-16T05:13:16.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 27 ls 1726463596,0 2024-09-16T05:13:16.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 28 ls 1726463596,0 2024-09-16T05:13:16.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 29 ls 1726463596,0 2024-09-16T05:13:16.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 30 ls 1726463596,0 2024-09-16T05:13:16.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg 31 ls 1726463596,0 2024-09-16T05:13:16.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: pg_num = 32 2024-09-16T05:13:16.678 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:6cac518f:::0:head 2024-09-16T05:13:16.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:02547ec2:::1:head 2024-09-16T05:13:16.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:f905c69b:::2:head 2024-09-16T05:13:16.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:cfc208b3:::3:head 2024-09-16T05:13:16.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:d83876eb:::4:head 2024-09-16T05:13:16.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:b29083e3:::5:head 2024-09-16T05:13:16.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:c4fdafeb:::6:head 2024-09-16T05:13:16.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:5c6b0b28:::7:head 2024-09-16T05:13:16.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:bd63b0f1:::8:head 2024-09-16T05:13:16.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:e960b815:::9:head 2024-09-16T05:13:16.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:52ea6a34:::10:head 2024-09-16T05:13:16.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:89d3ae78:::11:head 2024-09-16T05:13:16.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:de5d7c5f:::12:head 2024-09-16T05:13:16.679 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:566253c9:::13:head 2024-09-16T05:13:16.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:62a1935d:::14:head 2024-09-16T05:13:16.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:863748b0:::15:head 2024-09-16T05:13:16.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:3958e169:::16:head 2024-09-16T05:13:16.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:4d4dabf9:::17:head 2024-09-16T05:13:16.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:8391935d:::18:head 2024-09-16T05:13:16.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:28883081:::19:head 2024-09-16T05:13:16.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:69259c59:::20:head 2024-09-16T05:13:16.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:4bdb80b7:::21:head 2024-09-16T05:13:16.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:a11c5d71:::22:head 2024-09-16T05:13:16.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:271af37b:::23:head 2024-09-16T05:13:16.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:95b121be:::24:head 2024-09-16T05:13:16.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:58d1031b:::25:head 2024-09-16T05:13:16.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:0a050783:::26:head 2024-09-16T05:13:16.680 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:c709704c:::27:head 2024-09-16T05:13:16.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:cbe56eaf:::28:head 2024-09-16T05:13:16.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:86b4b162:::29:head 2024-09-16T05:13:16.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:70d89383:::30:head 2024-09-16T05:13:16.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:dd450c7c:::31:head 2024-09-16T05:13:16.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:6d5729b1:::32:head 2024-09-16T05:13:16.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:c388f3fb:::33:head 2024-09-16T05:13:16.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:56cfea31:::34:head 2024-09-16T05:13:16.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:9dbc1bf7:::35:head 2024-09-16T05:13:16.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:40b74ccd:::36:head 2024-09-16T05:13:16.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:4d5aaf42:::37:head 2024-09-16T05:13:16.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:920f362c:::38:head 2024-09-16T05:13:16.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:6cc53222:::39:head 2024-09-16T05:13:16.681 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:9cad833f:::40:head 2024-09-16T05:13:16.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:1ea84d41:::41:head 2024-09-16T05:13:16.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:c4480ef6:::42:head 2024-09-16T05:13:16.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:a694361e:::43:head 2024-09-16T05:13:16.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:d1bd33e9:::44:head 2024-09-16T05:13:16.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:ddc2cd5d:::45:head 2024-09-16T05:13:16.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:2b782207:::46:head 2024-09-16T05:13:16.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:7b187fca:::47:head 2024-09-16T05:13:16.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:90ecdf6f:::48:head 2024-09-16T05:13:16.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:a5ed95fe:::49:head 2024-09-16T05:13:16.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:ea0eaa55:::50:head 2024-09-16T05:13:16.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:f33ef17b:::51:head 2024-09-16T05:13:16.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:a0d1b2f6:::52:head 2024-09-16T05:13:16.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:60c5229e:::53:head 2024-09-16T05:13:16.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:edcbc575:::54:head 2024-09-16T05:13:16.682 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:102cf253:::55:head 2024-09-16T05:13:16.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:efb7fb0b:::56:head 2024-09-16T05:13:16.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:50d0a326:::57:head 2024-09-16T05:13:16.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:d4dc5daf:::58:head 2024-09-16T05:13:16.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:3a130462:::59:head 2024-09-16T05:13:16.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:ec87ed71:::60:head 2024-09-16T05:13:16.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:d5bc9454:::61:head 2024-09-16T05:13:16.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:3ddfe313:::62:head 2024-09-16T05:13:16.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:7c2816b9:::63:head 2024-09-16T05:13:16.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:47e00e4d:::64:head 2024-09-16T05:13:16.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:c6410c18:::65:head 2024-09-16T05:13:16.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:b48ed237:::66:head 2024-09-16T05:13:16.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:cd63ad31:::67:head 2024-09-16T05:13:16.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:b179e92b:::68:head 2024-09-16T05:13:16.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:0d9f741a:::69:head 2024-09-16T05:13:16.683 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:6d3352ae:::70:head 2024-09-16T05:13:16.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:c6d5c19e:::71:head 2024-09-16T05:13:16.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:bc4729c3:::72:head 2024-09-16T05:13:16.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:77e930b9:::73:head 2024-09-16T05:13:16.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:0abeecfd:::74:head 2024-09-16T05:13:16.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:b7c37e15:::75:head 2024-09-16T05:13:16.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:b6378398:::76:head 2024-09-16T05:13:16.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:02bd68de:::77:head 2024-09-16T05:13:16.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:cc795d2d:::78:head 2024-09-16T05:13:16.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:630d4fea:::79:head 2024-09-16T05:13:16.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:e0d29ef5:::80:head 2024-09-16T05:13:16.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:fd6f13d2:::81:head 2024-09-16T05:13:16.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:606461d5:::82:head 2024-09-16T05:13:16.684 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:eadbdc43:::83:head 2024-09-16T05:13:16.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:8761d0bb:::84:head 2024-09-16T05:13:16.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:9ef0186f:::85:head 2024-09-16T05:13:16.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:e0d41294:::86:head 2024-09-16T05:13:16.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:961de695:::87:head 2024-09-16T05:13:16.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:1423148f:::88:head 2024-09-16T05:13:16.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:633a8fa2:::89:head 2024-09-16T05:13:16.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:a8653809:::90:head 2024-09-16T05:13:16.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:3dac8b33:::91:head 2024-09-16T05:13:16.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:35aad435:::92:head 2024-09-16T05:13:16.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:f6dcc343:::93:head 2024-09-16T05:13:16.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:dbbdad87:::94:head 2024-09-16T05:13:16.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:1cb48ce0:::95:head 2024-09-16T05:13:16.685 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:03cd461c:::96:head 2024-09-16T05:13:16.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:17a4ea99:::97:head 2024-09-16T05:13:16.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:9993c9a7:::98:head 2024-09-16T05:13:16.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:6394211c:::99:head 2024-09-16T05:13:16.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:94c7ae57:::100:head 2024-09-16T05:13:16.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:6fdee5bb:::101:head 2024-09-16T05:13:16.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:9a477fd1:::102:head 2024-09-16T05:13:16.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:eb850916:::103:head 2024-09-16T05:13:16.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:affc56b9:::104:head 2024-09-16T05:13:16.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:b42dc814:::105:head 2024-09-16T05:13:16.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:f319f8f0:::106:head 2024-09-16T05:13:16.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:9a40b9de:::107:head 2024-09-16T05:13:16.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:8b524f28:::108:head 2024-09-16T05:13:16.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:e3de589f:::109:head 2024-09-16T05:13:16.686 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:90f90a5b:::110:head 2024-09-16T05:13:16.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:a7b4f1d7:::111:head 2024-09-16T05:13:16.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:af51766e:::112:head 2024-09-16T05:13:16.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:b6f90bd1:::113:head 2024-09-16T05:13:16.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:e0261208:::114:head 2024-09-16T05:13:16.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:c9569ef7:::115:head 2024-09-16T05:13:16.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:61bebe50:::116:head 2024-09-16T05:13:16.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:fe93412b:::117:head 2024-09-16T05:13:16.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:d3d38bee:::118:head 2024-09-16T05:13:16.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:3100ba0c:::119:head 2024-09-16T05:13:16.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:d0560ada:::120:head 2024-09-16T05:13:16.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:f0ea8b35:::121:head 2024-09-16T05:13:16.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:766f231a:::122:head 2024-09-16T05:13:16.687 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:a07a2582:::123:head 2024-09-16T05:13:16.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:bd7c6b3a:::124:head 2024-09-16T05:13:16.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:fb2ddaff:::125:head 2024-09-16T05:13:16.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:4408e1fe:::126:head 2024-09-16T05:13:16.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:ee1df7a7:::127:head 2024-09-16T05:13:16.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:c3002909:::128:head 2024-09-16T05:13:16.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:4f48ffa9:::129:head 2024-09-16T05:13:16.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:edf38733:::130:head 2024-09-16T05:13:16.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:c08425c0:::131:head 2024-09-16T05:13:16.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:5f902d98:::132:head 2024-09-16T05:13:16.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:41ea2c93:::133:head 2024-09-16T05:13:16.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:813cee13:::134:head 2024-09-16T05:13:16.688 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:0131818d:::135:head 2024-09-16T05:13:16.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:26ba5a85:::136:head 2024-09-16T05:13:16.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:381b8a5a:::137:head 2024-09-16T05:13:16.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:28797e47:::138:head 2024-09-16T05:13:16.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:bfca7f22:::139:head 2024-09-16T05:13:16.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:36807075:::140:head 2024-09-16T05:13:16.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:80b03975:::141:head 2024-09-16T05:13:16.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:5c15709b:::142:head 2024-09-16T05:13:16.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:f39ea15e:::143:head 2024-09-16T05:13:16.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:ea992956:::144:head 2024-09-16T05:13:16.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:48887b1c:::145:head 2024-09-16T05:13:16.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:9f24a9dd:::146:head 2024-09-16T05:13:16.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:987f100b:::147:head 2024-09-16T05:13:16.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:d2dd3581:::148:head 2024-09-16T05:13:16.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:7fed1808:::149:head 2024-09-16T05:13:16.689 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:c80b70e9:::150:head 2024-09-16T05:13:16.690 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:85ed90f9:::151:head 2024-09-16T05:13:16.690 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:36428b24:::152:head 2024-09-16T05:13:16.690 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:d044c34a:::153:head 2024-09-16T05:13:16.690 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:7c18bf58:::154:head 2024-09-16T05:13:16.690 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:d1c21232:::155:head 2024-09-16T05:13:16.690 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:a7a3c575:::156:head 2024-09-16T05:13:16.690 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:87da0633:::157:head 2024-09-16T05:13:16.690 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:d5ac3822:::158:head 2024-09-16T05:13:16.690 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:3f20522d:::159:head 2024-09-16T05:13:16.690 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:6ca26563:::160:head 2024-09-16T05:13:16.690 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:532ce135:::161:head 2024-09-16T05:13:16.690 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:c78863e6:::162:head 2024-09-16T05:13:53.954 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:7462ddf6::io.RoundTripAppendPP (2950 ms) 2024-09-16T05:13:53.954 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-16T05:13:53.954 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3094 ms) 2024-09-16T05:13:53.955 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-16T05:13:53.955 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3029 ms) 2024-09-16T05:13:53.955 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-16T05:13:53.955 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3060 ms) 2024-09-16T05:13:53.955 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-16T05:13:53.955 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: setting pool EIO 2024-09-16T05:13:53.955 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: max_success 99, min_failed 100 2024-09-16T05:13:53.955 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (3825 ms) 2024-09-16T05:13:53.955 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-16T05:13:53.955 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3062 ms) 2024-09-16T05:13:53.955 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (148898 ms total) 2024-09-16T05:13:53.955 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: 2024-09-16T05:13:53.956 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-16T05:13:53.956 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-16T05:13:53.956 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (2975 ms) 2024-09-16T05:13:53.956 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-16T05:13:53.956 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9026 ms) 2024-09-16T05:13:53.956 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-16T05:13:53.956 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14955 ms) 2024-09-16T05:13:53.956 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-16T05:13:53.956 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3036 ms) 2024-09-16T05:13:53.956 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29992 ms total) 2024-09-16T05:13:53.956 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: 2024-09-16T05:13:53.956 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-16T05:13:53.957 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-16T05:13:53.957 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2999 ms) 2024-09-16T05:13:53.957 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2999 ms total) 2024-09-16T05:13:53.957 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: 2024-09-16T05:13:53.957 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-16T05:13:53.957 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-16T05:13:53.957 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (42733 ms) 2024-09-16T05:13:53.957 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-16T05:13:53.957 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6985 ms) 2024-09-16T05:13:53.957 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-16T05:13:53.957 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7049 ms) 2024-09-16T05:13:53.957 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-16T05:13:53.958 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6982 ms) 2024-09-16T05:13:53.958 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-16T05:13:53.958 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3011 ms) 2024-09-16T05:13:53.958 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-16T05:13:53.958 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6989 ms) 2024-09-16T05:13:53.958 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-16T05:13:53.958 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6984 ms) 2024-09-16T05:13:53.958 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-16T05:13:53.958 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7028 ms) 2024-09-16T05:13:53.958 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-16T05:13:53.958 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6985 ms) 2024-09-16T05:13:53.958 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-16T05:13:53.958 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7002 ms) 2024-09-16T05:13:53.959 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-16T05:13:53.959 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7051 ms) 2024-09-16T05:13:53.959 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-16T05:13:53.959 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6977 ms) 2024-09-16T05:13:53.959 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-16T05:13:53.959 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6993 ms) 2024-09-16T05:14:00.863 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T05:14:00.863 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T05:14:00.863 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T05:14:00.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T05:14:00.867 INFO:tasks.ceph.mon.a:Started 2024-09-16T05:14:00.867 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:14:00.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:14:00.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 10 monclient: get_monmap_and_config 2024-09-16T05:14:00.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 10 monclient: build_initial_monmap 2024-09-16T05:14:00.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 10 monclient: monmap: 2024-09-16T05:14:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:00.998736+0000 2024-09-16T05:14:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:00.998736+0000 2024-09-16T05:14:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:00.998 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:00.998 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:00.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 1 Processor -- start 2024-09-16T05:14:00.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 1 -- start start 2024-09-16T05:14:00.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 10 monclient: init 2024-09-16T05:14:01.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:01.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T05:14:01.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb41c1376e0 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:01.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb41c1370a0 0x7fb41c137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:01.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 10 monclient(hunting): picked mon.noname-c con 0x7fb41c1370a0 addr v2:172.21.15.149:3300/0 2024-09-16T05:14:01.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb41c0a5d00 addr v1:172.21.15.149:6789/0 2024-09-16T05:14:01.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb41c0a5090 con 0x7fb41c1376e0 2024-09-16T05:14:01.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb41c09bd20 con 0x7fb41c0a5d00 2024-09-16T05:14:01.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:01.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fb41c138000 con 0x7fb41c1370a0 2024-09-16T05:14:01.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:01.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb421817640 10 monclient(hunting): authenticate will time out at 1325.627623s 2024-09-16T05:14:01.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb41affd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb41c1376e0 0x7fb41c0a4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33008/0 (socket says 172.21.15.114:33008) 2024-09-16T05:14:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.992+0000 7fb41affd640 1 -- 172.21.15.114:0/897868817 learned_addr learned my addr 172.21.15.114:0/897868817 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:14:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 1 --2- 172.21.15.114:0/897868817 >> v2:172.21.15.149:3300/0 conn(0x7fb41c1370a0 0x7fb41c137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 10 monclient(hunting): get_auth_request con 0x7fb41c1370a0 auth_method 0 2024-09-16T05:14:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 10 monclient(hunting): handle_auth_done global_id 64291 payload 274 2024-09-16T05:14:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 1 -- 172.21.15.114:0/897868817 >> v1:172.21.15.149:6789/0 conn(0x7fb41c0a5d00 legacy=0x7fb41c0a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 1 -- 172.21.15.114:0/897868817 >> v1:172.21.15.114:6789/0 conn(0x7fb41c1376e0 legacy=0x7fb41c0a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:14:01.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 1 monclient: found mon.noname-c 2024-09-16T05:14:01.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:01.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:14:01.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 1 -- 172.21.15.114:0/897868817 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb41c09c000 con 0x7fb41c1370a0 2024-09-16T05:14:01.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb41a7fc640 1 --2- 172.21.15.114:0/897868817 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb41c1370a0 0x7fb41c137ac0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb4040097c0 tx=0x7fb40401c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b39da7df9890698 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:01.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb419ffb640 1 -- 172.21.15.114:0/897868817 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4040b0070 con 0x7fb41c1370a0 2024-09-16T05:14:01.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb419ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:01.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb419ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:14:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb419ffb640 10 monclient: dump: 2024-09-16T05:14:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:01.011 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb419ffb640 10 monclient: _finish_auth 0 2024-09-16T05:14:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb419ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:14:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb419ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb421817640 5 monclient: authenticate success, global_id 64291 2024-09-16T05:14:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb421817640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb419ffb640 1 -- 172.21.15.114:0/897868817 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4040ab600 con 0x7fb41c1370a0 2024-09-16T05:14:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb419ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb421817640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb421817640 4 set_mon_vals no callback set 2024-09-16T05:14:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb419ffb640 1 -- 172.21.15.114:0/897868817 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4040ab9d0 con 0x7fb41c1370a0 2024-09-16T05:14:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb419ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb419ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:14:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb419ffb640 10 monclient: dump: 2024-09-16T05:14:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:01.012 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb421817640 10 monclient: shutdown 2024-09-16T05:14:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb421817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb421817640 1 -- 172.21.15.114:0/897868817 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb41c1370a0 msgr2=0x7fb41c137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb421817640 1 --2- 172.21.15.114:0/897868817 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb41c1370a0 0x7fb41c137ac0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb4040097c0 tx=0x7fb40401c360 comp rx=0 tx=0).stop 2024-09-16T05:14:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb421817640 1 -- 172.21.15.114:0/897868817 shutdown_connections 2024-09-16T05:14:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb421817640 1 --2- 172.21.15.114:0/897868817 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb41c1370a0 0x7fb41c137ac0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb421817640 1 -- 172.21.15.114:0/897868817 >> 172.21.15.114:0/897868817 conn(0x7fb41c0a9a40 msgr2=0x7fb41c0a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb421817640 1 -- 172.21.15.114:0/897868817 shutdown_connections 2024-09-16T05:14:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:00.996+0000 7fb421817640 1 -- 172.21.15.114:0/897868817 wait complete. 2024-09-16T05:14:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient: build_initial_monmap 2024-09-16T05:14:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient: monmap: 2024-09-16T05:14:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:01.014 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 librados: starting msgr at 2024-09-16T05:14:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 librados: starting objecter 2024-09-16T05:14:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 Processor -- start 2024-09-16T05:14:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 -- start start 2024-09-16T05:14:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 librados: setting wanted keys 2024-09-16T05:14:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 librados: calling monclient init 2024-09-16T05:14:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient: init 2024-09-16T05:14:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:14:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb41c0a5d00 0x7fb41c135c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient(hunting): picked mon.noname-c con 0x7fb41c0a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb41c1370a0 0x7fb41c1361c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient(hunting): picked mon.noname-a con 0x7fb41c1370a0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb41c1376e0 0x7fb41c13bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient(hunting): picked mon.noname-b con 0x7fb41c1376e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c142fa0 con 0x7fb41c1370a0 2024-09-16T05:14:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c142e20 con 0x7fb41c1376e0 2024-09-16T05:14:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb41c143120 con 0x7fb41c0a5d00 2024-09-16T05:14:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient(hunting): authenticate will time out at 1325.633410s 2024-09-16T05:14:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb41c0a5d00 0x7fb41c135c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb41c1370a0 0x7fb41c1361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb41c1370a0 0x7fb41c1361c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54524/0 (socket says 172.21.15.114:54524) 2024-09-16T05:14:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41a7fc640 1 -- 172.21.15.114:0/482166764 learned_addr learned my addr 172.21.15.114:0/482166764 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41a7fc640 10 monclient(hunting): get_auth_request con 0x7fb41c1370a0 auth_method 0 2024-09-16T05:14:01.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb41c0a5d00 0x7fb41c135c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:49084/0 (socket says 172.21.15.114:49084) 2024-09-16T05:14:01.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:01.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:01.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:01.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 10 monclient(hunting): get_auth_request con 0x7fb41c0a5d00 auth_method 0 2024-09-16T05:14:01.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:01.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:01.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:01.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41a7fc640 1 -- 172.21.15.114:0/482166764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb41c1370a0 msgr2=0x7fb41c1361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:14:01.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41a7fc640 1 -- 172.21.15.114:0/482166764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb41c1370a0 msgr2=0x7fb41c1361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:14:01.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41a7fc640 1 --2- 172.21.15.114:0/482166764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb41c1370a0 0x7fb41c1361c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:01.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:01.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41a7fc640 1 --2- 172.21.15.114:0/482166764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb41c1370a0 0x7fb41c1361c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 10 monclient(hunting): handle_auth_done global_id 54599 payload 824 2024-09-16T05:14:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 1 -- 172.21.15.114:0/482166764 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb41c1376e0 msgr2=0x7fb41c13bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 1 --2- 172.21.15.114:0/482166764 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb41c1376e0 0x7fb41c13bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 1 -- 172.21.15.114:0/482166764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb41c1370a0 msgr2=0x7fb41c1361c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:14:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 1 --2- 172.21.15.114:0/482166764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb41c1370a0 0x7fb41c1361c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 10 monclient: _finish_hunting 0 2024-09-16T05:14:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 1 monclient: found mon.noname-c 2024-09-16T05:14:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:14:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 1 -- 172.21.15.114:0/482166764 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb41c13e550 con 0x7fb41c0a5d00 2024-09-16T05:14:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41affd640 1 --2- 172.21.15.114:0/482166764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb41c0a5d00 0x7fb41c135c80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb41400c430 tx=0x7fb41400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 10 client.?.objecter ms_handle_connect 0x7fb41c0a5d00 2024-09-16T05:14:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 1 -- 172.21.15.114:0/482166764 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414017070 con 0x7fb41c0a5d00 2024-09-16T05:14:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:14:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 10 monclient: dump: 2024-09-16T05:14:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:01.020 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 10 monclient: _finish_auth 0 2024-09-16T05:14:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 10 monclient: _check_auth_tickets 2024-09-16T05:14:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 1 -- 172.21.15.114:0/482166764 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb414005cf0 con 0x7fb41c0a5d00 2024-09-16T05:14:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 5 monclient: authenticate success, global_id 54599 2024-09-16T05:14:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient: _renew_subs 2024-09-16T05:14:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 -- 172.21.15.114:0/482166764 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb41c13e2d0 con 0x7fb41c0a5d00 2024-09-16T05:14:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 1 -- 172.21.15.114:0/482166764 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4140072d0 con 0x7fb41c0a5d00 2024-09-16T05:14:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 10 monclient: dump: 2024-09-16T05:14:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:01.021 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb418ff9640 4 set_mon_vals no callback set 2024-09-16T05:14:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 client.54599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient: _renew_subs 2024-09-16T05:14:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 -- 172.21.15.114:0/482166764 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb41c13f440 con 0x7fb41c0a5d00 2024-09-16T05:14:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 librados: init done 2024-09-16T05:14:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb421817640 1 -- 172.21.15.114:0/482166764 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb41c137ac0 con 0x7fb41c0a5d00 2024-09-16T05:14:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 1 -- 172.21.15.114:0/482166764 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb41401a070 con 0x7fb41c0a5d00 2024-09-16T05:14:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 1 --2- 172.21.15.114:0/482166764 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb3fc037810 0x7fb3fc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 1 -- 172.21.15.114:0/482166764 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(273..273 src has 1..273) ==== 6245+0+0 (secure 0 0 0) 0x7fb414048500 con 0x7fb41c0a5d00 2024-09-16T05:14:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 10 client.54599.objecter ms_dispatch 0x7fb41c000ba0 osd_map(273..273 src has 1..273) 2024-09-16T05:14:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 3 client.54599.objecter handle_osd_map got epochs [273,273] > 0 2024-09-16T05:14:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 3 client.54599.objecter handle_osd_map decoding full epoch 273 2024-09-16T05:14:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb413fff640 20 client.54599.objecter dump_active .. 0 homeless 2024-09-16T05:14:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41a7fc640 1 --2- 172.21.15.114:0/482166764 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb3fc037810 0x7fb3fc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.000+0000 7fb41a7fc640 10 monclient: get_auth_request con 0x7fb3fc037810 auth_method 0 2024-09-16T05:14:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.004+0000 7fb413fff640 10 client.54599.objecter ms_handle_connect 0x7fb3fc037810 2024-09-16T05:14:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.004+0000 7fb41a7fc640 1 --2- 172.21.15.114:0/482166764 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb3fc037810 0x7fb3fc039cd0 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 crypto rx=0x7fb4040097c0 tx=0x7fb4040a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.008+0000 7fb413fff640 1 -- 172.21.15.114:0/482166764 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb4140106d0 con 0x7fb41c0a5d00 2024-09-16T05:14:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.008+0000 7fb413fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.008+0000 7fb413fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:01.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.236+0000 7fb421817640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:14:01.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.236+0000 7fb421817640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:01.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.236+0000 7fb421817640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:01.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.236+0000 7fb421817640 1 -- 172.21.15.114:0/482166764 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb41c068820 con 0x7fb41c0a5d00 2024-09-16T05:14:01.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb413fff640 1 -- 172.21.15.114:0/482166764 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fb41404a6e0 con 0x7fb41c0a5d00 2024-09-16T05:14:01.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb413fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:01.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb413fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:01.243 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":92,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:14:01.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 10 librados: watch_flush enter 2024-09-16T05:14:01.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 10 librados: watch_flush exit 2024-09-16T05:14:01.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 20 client.54599.objecter shutdown clearing up homeless session... 2024-09-16T05:14:01.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 10 client.54599.objecter successfully canceled tick 2024-09-16T05:14:01.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 1 -- 172.21.15.114:0/482166764 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb3fc037810 msgr2=0x7fb3fc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:01.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 1 --2- 172.21.15.114:0/482166764 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb3fc037810 0x7fb3fc039cd0 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 crypto rx=0x7fb4040097c0 tx=0x7fb4040a6000 comp rx=0 tx=0).stop 2024-09-16T05:14:01.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 10 monclient: shutdown 2024-09-16T05:14:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 1 -- 172.21.15.114:0/482166764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb41c0a5d00 msgr2=0x7fb41c135c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 1 --2- 172.21.15.114:0/482166764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb41c0a5d00 0x7fb41c135c80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb41400c430 tx=0x7fb41400c8f0 comp rx=0 tx=0).stop 2024-09-16T05:14:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 1 -- 172.21.15.114:0/482166764 shutdown_connections 2024-09-16T05:14:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 1 --2- 172.21.15.114:0/482166764 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb3fc037810 0x7fb3fc039cd0 unknown :-1 s=CLOSED pgs=3487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 1 --2- 172.21.15.114:0/482166764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb41c1370a0 0x7fb41c1361c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 1 --2- 172.21.15.114:0/482166764 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb41c1376e0 0x7fb41c13bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 1 --2- 172.21.15.114:0/482166764 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb41c0a5d00 0x7fb41c135c80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 1 -- 172.21.15.114:0/482166764 >> 172.21.15.114:0/482166764 conn(0x7fb41c0a9a40 msgr2=0x7fb41c0a8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 1 -- 172.21.15.114:0/482166764 shutdown_connections 2024-09-16T05:14:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 1 -- 172.21.15.114:0/482166764 wait complete. 2024-09-16T05:14:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:01.240+0000 7fb421817640 1 librados: shutdown 2024-09-16T05:14:04.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:14:04.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.384+0000 7f1b569f2640 10 monclient: get_monmap_and_config 2024-09-16T05:14:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.384+0000 7f1b569f2640 10 monclient: build_initial_monmap 2024-09-16T05:14:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.384+0000 7f1b569f2640 10 monclient: monmap: 2024-09-16T05:14:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:04.390230+0000 2024-09-16T05:14:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:04.390230+0000 2024-09-16T05:14:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:04.389 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.384+0000 7f1b569f2640 1 Processor -- start 2024-09-16T05:14:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.384+0000 7f1b569f2640 1 -- start start 2024-09-16T05:14:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.384+0000 7f1b569f2640 10 monclient: init 2024-09-16T05:14:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b569f2640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b569f2640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T05:14:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b569f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b501376e0 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b569f2640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1b501370a0 0x7f1b50137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b569f2640 10 monclient(hunting): picked mon.noname-a con 0x7f1b501370a0 addr v2:172.21.15.114:3300/0 2024-09-16T05:14:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b569f2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b500a5d00 addr v1:172.21.15.149:6789/0 2024-09-16T05:14:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b569f2640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b500a5090 con 0x7f1b501376e0 2024-09-16T05:14:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b569f2640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b5009bd20 con 0x7f1b500a5d00 2024-09-16T05:14:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b569f2640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b569f2640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f1b50138000 con 0x7f1b501370a0 2024-09-16T05:14:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b569f2640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b569f2640 10 monclient(hunting): authenticate will time out at 1329.020226s 2024-09-16T05:14:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1b501370a0 0x7f1b50137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f1b501376e0 0x7f1b500a4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33020/0 (socket says 172.21.15.114:33020) 2024-09-16T05:14:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 1 -- 172.21.15.114:0/4194356552 learned_addr learned my addr 172.21.15.114:0/4194356552 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:14:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1b501370a0 0x7f1b50137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54538/0 (socket says 172.21.15.114:54538) 2024-09-16T05:14:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 10 monclient(hunting): get_auth_request con 0x7f1b501370a0 auth_method 0 2024-09-16T05:14:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 1 -- 172.21.15.114:0/4194356552 >> v2:172.21.15.114:3300/0 conn(0x7f1b501370a0 msgr2=0x7f1b50137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:14:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 1 -- 172.21.15.114:0/4194356552 >> v2:172.21.15.114:3300/0 conn(0x7f1b501370a0 msgr2=0x7f1b50137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 1 --2- 172.21.15.114:0/4194356552 >> v2:172.21.15.114:3300/0 conn(0x7f1b501370a0 0x7f1b50137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 1 --2- 172.21.15.114:0/4194356552 >> v2:172.21.15.114:3300/0 conn(0x7f1b501370a0 0x7f1b50137ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 -- 172.21.15.114:0/4194356552 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1710309449 0 0) 0x7f1b5009bd20 con 0x7f1b500a5d00 2024-09-16T05:14:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient(hunting): _negotiate 2024-09-16T05:14:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient(hunting): my global_id is 64306 2024-09-16T05:14:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 -- 172.21.15.114:0/4194356552 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b38003530 con 0x7f1b500a5d00 2024-09-16T05:14:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 -- 172.21.15.114:0/4194356552 <== mon.1 v1:172.21.15.149:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2162671230 0 0) 0x7f1b38003530 con 0x7f1b500a5d00 2024-09-16T05:14:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 -- 172.21.15.114:0/4194356552 >> v2:172.21.15.114:3300/0 conn(0x7f1b501370a0 msgr2=0x7f1b50137ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:14:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 --2- 172.21.15.114:0/4194356552 >> v2:172.21.15.114:3300/0 conn(0x7f1b501370a0 0x7f1b50137ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 -- 172.21.15.114:0/4194356552 >> v1:172.21.15.114:6789/0 conn(0x7f1b501376e0 legacy=0x7f1b500a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient: _finish_hunting 0 2024-09-16T05:14:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 monclient: found mon.noname-c-legacy 2024-09-16T05:14:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.149:6789/0 2024-09-16T05:14:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 -- 172.21.15.114:0/4194356552 --> v1:172.21.15.149:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5009c000 con 0x7f1b500a5d00 2024-09-16T05:14:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient: _finish_auth 0 2024-09-16T05:14:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient: _check_auth_tickets 2024-09-16T05:14:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b569f2640 5 monclient: authenticate success, global_id 64306 2024-09-16T05:14:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 -- 172.21.15.114:0/4194356552 <== mon.1 v1:172.21.15.149:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f1b48001a00 con 0x7f1b500a5d00 2024-09-16T05:14:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b569f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T05:14:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient: dump: 2024-09-16T05:14:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:04.398 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] but i'm connected to v1:172.21.15.149:6789/0, reconnecting 2024-09-16T05:14:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 -- 172.21.15.114:0/4194356552 >> v1:172.21.15.149:6789/0 conn(0x7f1b500a5d00 legacy=0x7f1b500a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:14:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 --2- 172.21.15.114:0/4194356552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b380039c0 0x7f1b38003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient(hunting): picked mon.a con 0x7f1b380039c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 --2- 172.21.15.114:0/4194356552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b38004810 0x7f1b38004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient(hunting): picked mon.c con 0x7f1b38004810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 --2- 172.21.15.114:0/4194356552 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b380053d0 0x7f1b3800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient(hunting): picked mon.b con 0x7f1b380053d0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 -- 172.21.15.114:0/4194356552 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3800c230 con 0x7f1b380039c0 2024-09-16T05:14:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 -- 172.21.15.114:0/4194356552 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3800c3e0 con 0x7f1b380053d0 2024-09-16T05:14:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 1 -- 172.21.15.114:0/4194356552 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1b3800c5e0 con 0x7f1b38004810 2024-09-16T05:14:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:14:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b569f2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 1 --2- 172.21.15.114:0/4194356552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b380039c0 0x7f1b38003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 10 monclient(hunting): get_auth_request con 0x7f1b380039c0 auth_method 0 2024-09-16T05:14:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 1 --2- 172.21.15.114:0/4194356552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b38004810 0x7f1b38004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 10 monclient(hunting): get_auth_request con 0x7f1b38004810 auth_method 0 2024-09-16T05:14:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:14:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:14:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b54f8f640 1 --2- 172.21.15.114:0/4194356552 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b380053d0 0x7f1b3800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b54f8f640 10 monclient(hunting): get_auth_request con 0x7f1b380053d0 auth_method 0 2024-09-16T05:14:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b54f8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b54f8f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b54f8f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 10 monclient(hunting): handle_auth_done global_id 64306 payload 293 2024-09-16T05:14:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 1 -- 172.21.15.114:0/4194356552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b38004810 msgr2=0x7f1b38004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:04.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 1 --2- 172.21.15.114:0/4194356552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b38004810 0x7f1b38004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:04.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 1 -- 172.21.15.114:0/4194356552 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b380053d0 msgr2=0x7f1b3800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:04.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 1 --2- 172.21.15.114:0/4194356552 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b380053d0 0x7f1b3800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:04.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:14:04.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 1 monclient: found mon.a 2024-09-16T05:14:04.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:04.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:04.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 1 -- 172.21.15.114:0/4194356552 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b38003530 con 0x7f1b380039c0 2024-09-16T05:14:04.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 10 monclient: _finish_auth 0 2024-09-16T05:14:04.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:14:04.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:04.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.388+0000 7f1b54f8f640 1 -- 172.21.15.114:0/4194356552 reap_dead start 2024-09-16T05:14:04.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 1 --2- 172.21.15.114:0/4194356552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b380039c0 0x7f1b38003de0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1b40001850 tx=0x7f1b4001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a1ba5fb92163c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:04.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4effd640 1 -- 172.21.15.114:0/4194356552 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b40001470 con 0x7f1b380039c0 2024-09-16T05:14:04.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:04.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:04.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4effd640 10 monclient: dump: 2024-09-16T05:14:04.408 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:04.408 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:04.408 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:04.409 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:04.409 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:04.409 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:04.409 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:04.409 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:04.409 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:04.409 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:04.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4effd640 1 -- 172.21.15.114:0/4194356552 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b400a4e50 con 0x7f1b380039c0 2024-09-16T05:14:04.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:04.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:04.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:04.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4effd640 1 -- 172.21.15.114:0/4194356552 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b40004cf0 con 0x7f1b380039c0 2024-09-16T05:14:04.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:04.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 4 set_mon_vals no callback set 2024-09-16T05:14:04.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:04.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4effd640 10 monclient: dump: 2024-09-16T05:14:04.410 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:04.410 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:04.410 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:04.410 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:04.410 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:04.410 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:04.410 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:04.410 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:04.410 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:04.410 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:04.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient: shutdown 2024-09-16T05:14:04.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:04.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 -- 172.21.15.114:0/4194356552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b380039c0 msgr2=0x7f1b38003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:04.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 --2- 172.21.15.114:0/4194356552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b380039c0 0x7f1b38003de0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1b40001850 tx=0x7f1b4001c4a0 comp rx=0 tx=0).stop 2024-09-16T05:14:04.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 -- 172.21.15.114:0/4194356552 shutdown_connections 2024-09-16T05:14:04.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 --2- 172.21.15.114:0/4194356552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b380039c0 0x7f1b38003de0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:04.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 -- 172.21.15.114:0/4194356552 >> 172.21.15.114:0/4194356552 conn(0x7f1b500a9a40 msgr2=0x7f1b500a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:04.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 -- 172.21.15.114:0/4194356552 shutdown_connections 2024-09-16T05:14:04.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 -- 172.21.15.114:0/4194356552 wait complete. 2024-09-16T05:14:04.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient: build_initial_monmap 2024-09-16T05:14:04.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient: monmap: 2024-09-16T05:14:04.412 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:04.412 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:04.412 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:04.412 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:04.412 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:04.412 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:04.412 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:04.413 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:04.413 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:04.413 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:04.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 librados: starting msgr at 2024-09-16T05:14:04.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 librados: starting objecter 2024-09-16T05:14:04.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 Processor -- start 2024-09-16T05:14:04.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 -- start start 2024-09-16T05:14:04.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 librados: setting wanted keys 2024-09-16T05:14:04.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 librados: calling monclient init 2024-09-16T05:14:04.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient: init 2024-09-16T05:14:04.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:04.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:14:04.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b380039c0 0x7f1b501cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:04.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient(hunting): picked mon.noname-c con 0x7f1b380039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:04.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b501cfd00 0x7f1b501247f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:04.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient(hunting): picked mon.noname-a con 0x7f1b501cfd00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:04.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b501d0140 0x7f1b50124d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:04.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient(hunting): picked mon.noname-b con 0x7f1b501d0140 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:04.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:04.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1b500a4e10 con 0x7f1b501cfd00 2024-09-16T05:14:04.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:04.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f1b500a4c90 con 0x7f1b501d0140 2024-09-16T05:14:04.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:04.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1b5009c5c0 con 0x7f1b380039c0 2024-09-16T05:14:04.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:04.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient(hunting): authenticate will time out at 1329.026570s 2024-09-16T05:14:04.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b501cfd00 0x7f1b501247f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:04.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b501cfd00 0x7f1b501247f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54554/0 (socket says 172.21.15.114:54554) 2024-09-16T05:14:04.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4f7fe640 1 -- 172.21.15.114:0/3020211770 learned_addr learned my addr 172.21.15.114:0/3020211770 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:04.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4f7fe640 10 monclient(hunting): get_auth_request con 0x7f1b501cfd00 auth_method 0 2024-09-16T05:14:04.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:04.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:04.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:04.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 1 --2- 172.21.15.114:0/3020211770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b380039c0 0x7f1b501cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:04.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 10 monclient(hunting): get_auth_request con 0x7f1b380039c0 auth_method 0 2024-09-16T05:14:04.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:04.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:04.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:04.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4f7fe640 1 -- 172.21.15.114:0/3020211770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b501cfd00 msgr2=0x7f1b501247f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:14:04.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4f7fe640 1 -- 172.21.15.114:0/3020211770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b501cfd00 msgr2=0x7f1b501247f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:14:04.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4f7fe640 1 --2- 172.21.15.114:0/3020211770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b501cfd00 0x7f1b501247f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:04.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4f7fe640 1 --2- 172.21.15.114:0/3020211770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b501cfd00 0x7f1b501247f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:04.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:04.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:04.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:04.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b54f8f640 1 --2- 172.21.15.114:0/3020211770 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b501d0140 0x7f1b50124d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:04.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 10 monclient(hunting): handle_auth_done global_id 54605 payload 824 2024-09-16T05:14:04.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 1 -- 172.21.15.114:0/3020211770 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b501d0140 msgr2=0x7f1b50124d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:04.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 1 --2- 172.21.15.114:0/3020211770 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b501d0140 0x7f1b50124d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:04.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 1 -- 172.21.15.114:0/3020211770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b501cfd00 msgr2=0x7f1b501247f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:14:04.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 1 --2- 172.21.15.114:0/3020211770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b501cfd00 0x7f1b501247f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:04.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:14:04.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 1 monclient: found mon.noname-c 2024-09-16T05:14:04.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:04.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:14:04.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 1 -- 172.21.15.114:0/3020211770 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b50129110 con 0x7f1b380039c0 2024-09-16T05:14:04.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4ffff640 1 --2- 172.21.15.114:0/3020211770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b380039c0 0x7f1b501cf7c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1b40005000 tx=0x7f1b38005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:04.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4cff9640 10 client.?.objecter ms_handle_connect 0x7f1b380039c0 2024-09-16T05:14:04.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:04.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4cff9640 1 -- 172.21.15.114:0/3020211770 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b40004930 con 0x7f1b380039c0 2024-09-16T05:14:04.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:04.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:14:04.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4cff9640 10 monclient: dump: 2024-09-16T05:14:04.418 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:04.418 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:04.418 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:04.418 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:04.418 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:04.418 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:04.418 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:04.418 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:04.419 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:04.419 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:04.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4cff9640 10 monclient: _finish_auth 0 2024-09-16T05:14:04.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4cff9640 10 monclient: _check_auth_tickets 2024-09-16T05:14:04.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:04.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 5 monclient: authenticate success, global_id 54605 2024-09-16T05:14:04.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4cff9640 1 -- 172.21.15.114:0/3020211770 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b40004ad0 con 0x7f1b380039c0 2024-09-16T05:14:04.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient: _renew_subs 2024-09-16T05:14:04.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:04.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 -- 172.21.15.114:0/3020211770 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b50128450 con 0x7f1b380039c0 2024-09-16T05:14:04.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:04.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4cff9640 1 -- 172.21.15.114:0/3020211770 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b40007120 con 0x7f1b380039c0 2024-09-16T05:14:04.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:04.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:04.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4cff9640 10 monclient: dump: 2024-09-16T05:14:04.421 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:04.421 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:04.421 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:04.421 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:04.421 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:04.421 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:04.421 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:04.421 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:04.421 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:04.421 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:04.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b4e7fc640 4 set_mon_vals no callback set 2024-09-16T05:14:04.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 client.54605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:04.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient: _renew_subs 2024-09-16T05:14:04.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:04.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 -- 172.21.15.114:0/3020211770 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b500a1ec0 con 0x7f1b380039c0 2024-09-16T05:14:04.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.392+0000 7f1b569f2640 1 librados: init done 2024-09-16T05:14:04.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.396+0000 7f1b569f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:04.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.396+0000 7f1b569f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:04.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.396+0000 7f1b569f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:04.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.396+0000 7f1b569f2640 1 -- 172.21.15.114:0/3020211770 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b50073d20 con 0x7f1b380039c0 2024-09-16T05:14:04.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.396+0000 7f1b4cff9640 1 -- 172.21.15.114:0/3020211770 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1b400b5030 con 0x7f1b380039c0 2024-09-16T05:14:04.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.396+0000 7f1b4cff9640 1 --2- 172.21.15.114:0/3020211770 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1b3c037810 0x7f1b3c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:04.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.396+0000 7f1b4cff9640 1 -- 172.21.15.114:0/3020211770 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(276..276 src has 1..276) ==== 5398+0+0 (secure 0 0 0) 0x7f1b400e1ad0 con 0x7f1b380039c0 2024-09-16T05:14:04.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.396+0000 7f1b4cff9640 10 client.54605.objecter ms_dispatch 0x7f1b50000ba0 osd_map(276..276 src has 1..276) 2024-09-16T05:14:04.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.396+0000 7f1b4cff9640 3 client.54605.objecter handle_osd_map got epochs [276,276] > 0 2024-09-16T05:14:04.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.396+0000 7f1b4cff9640 3 client.54605.objecter handle_osd_map decoding full epoch 276 2024-09-16T05:14:04.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.396+0000 7f1b4cff9640 20 client.54605.objecter dump_active .. 0 homeless 2024-09-16T05:14:04.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.396+0000 7f1b4f7fe640 1 --2- 172.21.15.114:0/3020211770 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1b3c037810 0x7f1b3c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:04.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.396+0000 7f1b4f7fe640 10 monclient: get_auth_request con 0x7f1b3c037810 auth_method 0 2024-09-16T05:14:04.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.396+0000 7f1b4cff9640 10 client.54605.objecter ms_handle_connect 0x7f1b3c037810 2024-09-16T05:14:04.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.396+0000 7f1b4f7fe640 1 --2- 172.21.15.114:0/3020211770 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1b3c037810 0x7f1b3c039cd0 secure :-1 s=READY pgs=3489 cs=0 l=1 rev1=1 crypto rx=0x7f1b50126040 tx=0x7f1b44008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:04.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.400+0000 7f1b4cff9640 1 -- 172.21.15.114:0/3020211770 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f1b50129110 con 0x7f1b380039c0 2024-09-16T05:14:04.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.400+0000 7f1b4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:04.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.400+0000 7f1b4cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:04.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.620+0000 7f1b569f2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:14:04.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.620+0000 7f1b569f2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:04.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.620+0000 7f1b569f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:04.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.620+0000 7f1b569f2640 1 -- 172.21.15.114:0/3020211770 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1b5013a4d0 con 0x7f1b380039c0 2024-09-16T05:14:04.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.620+0000 7f1b4cff9640 1 -- 172.21.15.114:0/3020211770 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f1b400161b0 con 0x7f1b380039c0 2024-09-16T05:14:04.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.620+0000 7f1b4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:04.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.620+0000 7f1b4cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:04.626 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":95,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:14:04.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 10 librados: watch_flush enter 2024-09-16T05:14:04.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 10 librados: watch_flush exit 2024-09-16T05:14:04.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 20 client.54605.objecter shutdown clearing up homeless session... 2024-09-16T05:14:04.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 10 client.54605.objecter successfully canceled tick 2024-09-16T05:14:04.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 1 -- 172.21.15.114:0/3020211770 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1b3c037810 msgr2=0x7f1b3c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:04.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 1 --2- 172.21.15.114:0/3020211770 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1b3c037810 0x7f1b3c039cd0 secure :-1 s=READY pgs=3489 cs=0 l=1 rev1=1 crypto rx=0x7f1b50126040 tx=0x7f1b44008040 comp rx=0 tx=0).stop 2024-09-16T05:14:04.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 10 monclient: shutdown 2024-09-16T05:14:04.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:04.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 1 -- 172.21.15.114:0/3020211770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b380039c0 msgr2=0x7f1b501cf7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:04.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 1 --2- 172.21.15.114:0/3020211770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b380039c0 0x7f1b501cf7c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1b40005000 tx=0x7f1b38005840 comp rx=0 tx=0).stop 2024-09-16T05:14:04.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 1 -- 172.21.15.114:0/3020211770 shutdown_connections 2024-09-16T05:14:04.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 1 --2- 172.21.15.114:0/3020211770 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1b3c037810 0x7f1b3c039cd0 unknown :-1 s=CLOSED pgs=3489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:04.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 1 --2- 172.21.15.114:0/3020211770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b501cfd00 0x7f1b501247f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:04.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 1 --2- 172.21.15.114:0/3020211770 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b501d0140 0x7f1b50124d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:04.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 1 --2- 172.21.15.114:0/3020211770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b380039c0 0x7f1b501cf7c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:04.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 1 -- 172.21.15.114:0/3020211770 >> 172.21.15.114:0/3020211770 conn(0x7f1b500a9a40 msgr2=0x7f1b5009d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:04.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 1 -- 172.21.15.114:0/3020211770 shutdown_connections 2024-09-16T05:14:04.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 1 -- 172.21.15.114:0/3020211770 wait complete. 2024-09-16T05:14:04.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:04.624+0000 7f1b569f2640 1 librados: shutdown 2024-09-16T05:14:07.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:14:07.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 10 monclient: get_monmap_and_config 2024-09-16T05:14:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 10 monclient: build_initial_monmap 2024-09-16T05:14:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 10 monclient: monmap: 2024-09-16T05:14:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:07.777325+0000 2024-09-16T05:14:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:07.777325+0000 2024-09-16T05:14:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:07.777 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:07.777 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:07.777 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:07.777 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:07.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 1 Processor -- start 2024-09-16T05:14:07.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 1 -- start start 2024-09-16T05:14:07.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 10 monclient: init 2024-09-16T05:14:07.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:07.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T05:14:07.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb4640a0ec0 addr v1:172.21.15.149:6789/0 2024-09-16T05:14:07.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46409e1d0 0x7fb46409e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:07.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 10 monclient(hunting): picked mon.noname-b con 0x7fb46409e1d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:07.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb464139640 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:07.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb46409a760 con 0x7fb464139640 2024-09-16T05:14:07.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4640a1870 con 0x7fb4640a0ec0 2024-09-16T05:14:07.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:07.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb4640a12a0 con 0x7fb46409e1d0 2024-09-16T05:14:07.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:07.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46bb95640 10 monclient(hunting): authenticate will time out at 1332.407307s 2024-09-16T05:14:07.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46a132640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb464139640 0x7fb46413ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34508/0 (socket says 172.21.15.114:34508) 2024-09-16T05:14:07.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb46a132640 1 -- 172.21.15.114:0/910011322 learned_addr learned my addr 172.21.15.114:0/910011322 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:14:07.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 1 --2- 172.21.15.114:0/910011322 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46409e1d0 0x7fb46409e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:07.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 10 monclient(hunting): get_auth_request con 0x7fb46409e1d0 auth_method 0 2024-09-16T05:14:07.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:07.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:07.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:07.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:07.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:07.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:07.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 10 monclient(hunting): handle_auth_done global_id 54608 payload 274 2024-09-16T05:14:07.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 1 -- 172.21.15.114:0/910011322 >> v1:172.21.15.149:6789/0 conn(0x7fb4640a0ec0 legacy=0x7fb46409dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:07.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 1 -- 172.21.15.114:0/910011322 >> v1:172.21.15.114:6789/0 conn(0x7fb464139640 legacy=0x7fb46413ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:07.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 10 monclient: _finish_hunting 0 2024-09-16T05:14:07.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 1 monclient: found mon.noname-b 2024-09-16T05:14:07.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:07.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:14:07.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.772+0000 7fb469130640 1 -- 172.21.15.114:0/910011322 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb464099790 con 0x7fb46409e1d0 2024-09-16T05:14:07.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb469130640 1 --2- 172.21.15.114:0/910011322 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46409e1d0 0x7fb46409e5e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb4540097a0 tx=0x7fb45401c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc2a6a90d8ce733d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:07.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46892f640 1 -- 172.21.15.114:0/910011322 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4540b0070 con 0x7fb46409e1d0 2024-09-16T05:14:07.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46892f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46892f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:14:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46892f640 10 monclient: dump: 2024-09-16T05:14:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:07.791 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46892f640 10 monclient: _finish_auth 0 2024-09-16T05:14:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46892f640 10 monclient: _check_auth_tickets 2024-09-16T05:14:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46892f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46892f640 1 -- 172.21.15.114:0/910011322 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4540ab600 con 0x7fb46409e1d0 2024-09-16T05:14:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 5 monclient: authenticate success, global_id 54608 2024-09-16T05:14:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46892f640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 4 set_mon_vals no callback set 2024-09-16T05:14:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46892f640 1 -- 172.21.15.114:0/910011322 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4540ab9d0 con 0x7fb46409e1d0 2024-09-16T05:14:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46892f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46892f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46892f640 10 monclient: dump: 2024-09-16T05:14:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:07.793 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 10 monclient: shutdown 2024-09-16T05:14:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 1 -- 172.21.15.114:0/910011322 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46409e1d0 msgr2=0x7fb46409e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 1 --2- 172.21.15.114:0/910011322 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46409e1d0 0x7fb46409e5e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb4540097a0 tx=0x7fb45401c390 comp rx=0 tx=0).stop 2024-09-16T05:14:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 1 -- 172.21.15.114:0/910011322 shutdown_connections 2024-09-16T05:14:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 1 --2- 172.21.15.114:0/910011322 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46409e1d0 0x7fb46409e5e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 1 -- 172.21.15.114:0/910011322 >> 172.21.15.114:0/910011322 conn(0x7fb4640a1fb0 msgr2=0x7fb4640a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 1 -- 172.21.15.114:0/910011322 shutdown_connections 2024-09-16T05:14:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 1 -- 172.21.15.114:0/910011322 wait complete. 2024-09-16T05:14:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 10 monclient: build_initial_monmap 2024-09-16T05:14:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 10 monclient: monmap: 2024-09-16T05:14:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:07.795 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:07.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 1 librados: starting msgr at 2024-09-16T05:14:07.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 1 librados: starting objecter 2024-09-16T05:14:07.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 1 Processor -- start 2024-09-16T05:14:07.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 1 -- start start 2024-09-16T05:14:07.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 1 librados: setting wanted keys 2024-09-16T05:14:07.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 1 librados: calling monclient init 2024-09-16T05:14:07.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.776+0000 7fb46bb95640 10 monclient: init 2024-09-16T05:14:07.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:07.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:14:07.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46409e1d0 0x7fb46409c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:07.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient(hunting): picked mon.noname-c con 0x7fb46409e1d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:07.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb4640a0ec0 0x7fb46408d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:07.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient(hunting): picked mon.noname-b con 0x7fb4640a0ec0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:07.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb464139640 0x7fb46408d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:07.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient(hunting): picked mon.noname-a con 0x7fb464139640 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:07.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:07.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb46413faf0 con 0x7fb464139640 2024-09-16T05:14:07.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:07.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb46413f970 con 0x7fb4640a0ec0 2024-09-16T05:14:07.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:07.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb46413fc70 con 0x7fb46409e1d0 2024-09-16T05:14:07.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:07.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient(hunting): authenticate will time out at 1332.412502s 2024-09-16T05:14:07.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46409e1d0 0x7fb46409c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:07.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46409e1d0 0x7fb46409c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:33532/0 (socket says 172.21.15.114:33532) 2024-09-16T05:14:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 1 -- 172.21.15.114:0/1347129800 learned_addr learned my addr 172.21.15.114:0/1347129800 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 10 monclient(hunting): get_auth_request con 0x7fb46409e1d0 auth_method 0 2024-09-16T05:14:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46a132640 1 --2- 172.21.15.114:0/1347129800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb464139640 0x7fb46408d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46a132640 10 monclient(hunting): get_auth_request con 0x7fb464139640 auth_method 0 2024-09-16T05:14:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46a132640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46a132640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46a132640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:07.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46a132640 1 -- 172.21.15.114:0/1347129800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb464139640 msgr2=0x7fb46408d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T05:14:07.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46a132640 1 -- 172.21.15.114:0/1347129800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb464139640 msgr2=0x7fb46408d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:14:07.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46a132640 1 --2- 172.21.15.114:0/1347129800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb464139640 0x7fb46408d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:07.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469130640 1 --2- 172.21.15.114:0/1347129800 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb4640a0ec0 0x7fb46408d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:07.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46a132640 1 --2- 172.21.15.114:0/1347129800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb464139640 0x7fb46408d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:07.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469130640 10 monclient(hunting): get_auth_request con 0x7fb4640a0ec0 auth_method 0 2024-09-16T05:14:07.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469130640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:07.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469130640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:07.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469130640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:07.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 10 monclient(hunting): handle_auth_done global_id 54611 payload 824 2024-09-16T05:14:07.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 1 -- 172.21.15.114:0/1347129800 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb4640a0ec0 msgr2=0x7fb46408d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:07.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 1 --2- 172.21.15.114:0/1347129800 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb4640a0ec0 0x7fb46408d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:07.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 1 -- 172.21.15.114:0/1347129800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb464139640 msgr2=0x7fb46408d7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:14:07.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 1 --2- 172.21.15.114:0/1347129800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb464139640 0x7fb46408d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 10 monclient: _finish_hunting 0 2024-09-16T05:14:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 1 monclient: found mon.noname-c 2024-09-16T05:14:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:14:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 1 -- 172.21.15.114:0/1347129800 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb464093ba0 con 0x7fb46409e1d0 2024-09-16T05:14:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469931640 1 --2- 172.21.15.114:0/1347129800 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46409e1d0 0x7fb46409c9b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb46000c480 tx=0x7fb46000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 10 client.?.objecter ms_handle_connect 0x7fb46409e1d0 2024-09-16T05:14:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 1 -- 172.21.15.114:0/1347129800 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb460012830 con 0x7fb46409e1d0 2024-09-16T05:14:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:14:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 10 monclient: dump: 2024-09-16T05:14:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:07.800 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:14:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:14:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 1 -- 172.21.15.114:0/1347129800 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb460017070 con 0x7fb46409e1d0 2024-09-16T05:14:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 5 monclient: authenticate success, global_id 54611 2024-09-16T05:14:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient: _renew_subs 2024-09-16T05:14:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 1 -- 172.21.15.114:0/1347129800 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb464093e80 con 0x7fb46409e1d0 2024-09-16T05:14:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 1 -- 172.21.15.114:0/1347129800 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb460012c30 con 0x7fb46409e1d0 2024-09-16T05:14:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 10 monclient: dump: 2024-09-16T05:14:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:07.802 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:07.802 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:07.802 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:07.802 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:07.802 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:07.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45bfff640 4 set_mon_vals no callback set 2024-09-16T05:14:07.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 client.54611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:07.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient: _renew_subs 2024-09-16T05:14:07.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:07.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 1 -- 172.21.15.114:0/1347129800 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb46408e560 con 0x7fb46409e1d0 2024-09-16T05:14:07.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 1 librados: init done 2024-09-16T05:14:07.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 1 -- 172.21.15.114:0/1347129800 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb46001c460 con 0x7fb46409e1d0 2024-09-16T05:14:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb46bb95640 1 -- 172.21.15.114:0/1347129800 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb46409e5e0 con 0x7fb46409e1d0 2024-09-16T05:14:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 1 --2- 172.21.15.114:0/1347129800 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb448037810 0x7fb448039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 1 -- 172.21.15.114:0/1347129800 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(279..279 src has 1..279) ==== 6253+0+0 (secure 0 0 0) 0x7fb460048fc0 con 0x7fb46409e1d0 2024-09-16T05:14:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 10 client.54611.objecter ms_dispatch 0x7fb464000ba0 osd_map(279..279 src has 1..279) 2024-09-16T05:14:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 3 client.54611.objecter handle_osd_map got epochs [279,279] > 0 2024-09-16T05:14:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 3 client.54611.objecter handle_osd_map decoding full epoch 279 2024-09-16T05:14:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 20 client.54611.objecter dump_active .. 0 homeless 2024-09-16T05:14:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469130640 1 --2- 172.21.15.114:0/1347129800 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb448037810 0x7fb448039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469130640 10 monclient: get_auth_request con 0x7fb448037810 auth_method 0 2024-09-16T05:14:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb469130640 1 --2- 172.21.15.114:0/1347129800 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb448037810 0x7fb448039cd0 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 crypto rx=0x7fb4540ad000 tx=0x7fb4540a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.780+0000 7fb45a7fc640 10 client.54611.objecter ms_handle_connect 0x7fb448037810 2024-09-16T05:14:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.784+0000 7fb45a7fc640 1 -- 172.21.15.114:0/1347129800 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb46001a760 con 0x7fb46409e1d0 2024-09-16T05:14:07.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.784+0000 7fb45a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:07.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:07.784+0000 7fb45a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:08.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:14:08.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:08.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:08.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 1 -- 172.21.15.114:0/1347129800 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb464139380 con 0x7fb46409e1d0 2024-09-16T05:14:08.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb45a7fc640 1 -- 172.21.15.114:0/1347129800 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fb46001b3c0 con 0x7fb46409e1d0 2024-09-16T05:14:08.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb45a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:08.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb45a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:08.020 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":99,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:14:08.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 10 librados: watch_flush enter 2024-09-16T05:14:08.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 10 librados: watch_flush exit 2024-09-16T05:14:08.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 20 client.54611.objecter shutdown clearing up homeless session... 2024-09-16T05:14:08.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 10 client.54611.objecter successfully canceled tick 2024-09-16T05:14:08.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 1 -- 172.21.15.114:0/1347129800 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb448037810 msgr2=0x7fb448039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:08.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 1 --2- 172.21.15.114:0/1347129800 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb448037810 0x7fb448039cd0 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 crypto rx=0x7fb4540ad000 tx=0x7fb4540a6000 comp rx=0 tx=0).stop 2024-09-16T05:14:08.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 10 monclient: shutdown 2024-09-16T05:14:08.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:08.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 1 -- 172.21.15.114:0/1347129800 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46409e1d0 msgr2=0x7fb46409c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:08.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 1 --2- 172.21.15.114:0/1347129800 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46409e1d0 0x7fb46409c9b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb46000c480 tx=0x7fb46000c940 comp rx=0 tx=0).stop 2024-09-16T05:14:08.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 1 -- 172.21.15.114:0/1347129800 shutdown_connections 2024-09-16T05:14:08.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 1 --2- 172.21.15.114:0/1347129800 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb448037810 0x7fb448039cd0 unknown :-1 s=CLOSED pgs=3490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:08.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 1 --2- 172.21.15.114:0/1347129800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb464139640 0x7fb46408d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:08.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 1 --2- 172.21.15.114:0/1347129800 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb4640a0ec0 0x7fb46408d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:08.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 1 --2- 172.21.15.114:0/1347129800 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46409e1d0 0x7fb46409c9b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:08.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 1 -- 172.21.15.114:0/1347129800 >> 172.21.15.114:0/1347129800 conn(0x7fb4640a1fb0 msgr2=0x7fb4640a32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:08.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 1 -- 172.21.15.114:0/1347129800 shutdown_connections 2024-09-16T05:14:08.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 1 -- 172.21.15.114:0/1347129800 wait complete. 2024-09-16T05:14:08.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:08.016+0000 7fb46bb95640 1 librados: shutdown 2024-09-16T05:14:11.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:14:11.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 10 monclient: get_monmap_and_config 2024-09-16T05:14:11.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 10 monclient: build_initial_monmap 2024-09-16T05:14:11.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 10 monclient: monmap: 2024-09-16T05:14:11.168 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:11.168 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:11.168 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:11.169515+0000 2024-09-16T05:14:11.168 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:11.169515+0000 2024-09-16T05:14:11.168 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:11.168 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:11.168 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:11.169 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:11.169 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:11.169 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:11.169 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:11.169 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:11.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 1 Processor -- start 2024-09-16T05:14:11.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 1 -- start start 2024-09-16T05:14:11.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 10 monclient: init 2024-09-16T05:14:11.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:11.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-16T05:14:11.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97bc137480 addr v1:172.21.15.149:6789/0 2024-09-16T05:14:11.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97bc0a57b0 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:11.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f97bc13da70 0x7f97bc13fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:11.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 10 monclient(hunting): picked mon.noname-c con 0x7f97bc13da70 addr v2:172.21.15.149:3300/0 2024-09-16T05:14:11.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97bc09cfa0 con 0x7f97bc0a57b0 2024-09-16T05:14:11.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97bc0a52f0 con 0x7f97bc137480 2024-09-16T05:14:11.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:11.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f97bc137f70 con 0x7f97bc13da70 2024-09-16T05:14:11.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:11.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c254f640 10 monclient(hunting): authenticate will time out at 1335.798694s 2024-09-16T05:14:11.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97b37fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f97bc0a57b0 0x7f97bc13c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34512/0 (socket says 172.21.15.114:34512) 2024-09-16T05:14:11.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97b37fe640 1 -- 172.21.15.114:0/2652402006 learned_addr learned my addr 172.21.15.114:0/2652402006 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:14:11.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 1 --2- 172.21.15.114:0/2652402006 >> v2:172.21.15.149:3300/0 conn(0x7f97bc13da70 0x7f97bc13fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:11.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 10 monclient(hunting): get_auth_request con 0x7f97bc13da70 auth_method 0 2024-09-16T05:14:11.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:11.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:11.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:11.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:11.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:11.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:11.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 10 monclient(hunting): handle_auth_done global_id 64318 payload 274 2024-09-16T05:14:11.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 1 -- 172.21.15.114:0/2652402006 >> v1:172.21.15.149:6789/0 conn(0x7f97bc137480 legacy=0x7f97bc137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:11.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 1 -- 172.21.15.114:0/2652402006 >> v1:172.21.15.114:6789/0 conn(0x7f97bc0a57b0 legacy=0x7f97bc13c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:11.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 10 monclient: _finish_hunting 0 2024-09-16T05:14:11.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 1 monclient: found mon.noname-c 2024-09-16T05:14:11.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:11.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:14:11.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.164+0000 7f97c0aec640 1 -- 172.21.15.114:0/2652402006 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97bc0a5570 con 0x7f97bc13da70 2024-09-16T05:14:11.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97c0aec640 1 --2- 172.21.15.114:0/2652402006 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97bc13da70 0x7f97bc13fe60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009d90 tx=0x7f97b401c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b364f40d0413f7f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:11.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97bb7fe640 1 -- 172.21.15.114:0/2652402006 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b40b2070 con 0x7f97bc13da70 2024-09-16T05:14:11.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:11.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97bb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:14:11.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97bb7fe640 10 monclient: dump: 2024-09-16T05:14:11.179 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:11.179 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:11.179 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:11.179 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:11.180 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:11.180 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:11.180 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:11.180 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:11.180 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:11.180 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:11.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97bb7fe640 10 monclient: _finish_auth 0 2024-09-16T05:14:11.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97bb7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:14:11.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:11.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97bb7fe640 1 -- 172.21.15.114:0/2652402006 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97b40a58c0 con 0x7f97bc13da70 2024-09-16T05:14:11.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97bb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:11.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97bb7fe640 1 -- 172.21.15.114:0/2652402006 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b40a5c90 con 0x7f97bc13da70 2024-09-16T05:14:11.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:11.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97bb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:14:11.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97bb7fe640 10 monclient: dump: 2024-09-16T05:14:11.181 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:11.181 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:11.181 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:11.181 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:11.181 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:11.181 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:11.181 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:11.181 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:11.181 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:11.181 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:11.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97c254f640 5 monclient: authenticate success, global_id 64318 2024-09-16T05:14:11.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97c254f640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:11.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97c254f640 4 set_mon_vals no callback set 2024-09-16T05:14:11.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97c254f640 10 monclient: shutdown 2024-09-16T05:14:11.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97c254f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:11.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97c254f640 1 -- 172.21.15.114:0/2652402006 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97bc13da70 msgr2=0x7f97bc13fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:11.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97c254f640 1 --2- 172.21.15.114:0/2652402006 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97bc13da70 0x7f97bc13fe60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009d90 tx=0x7f97b401c800 comp rx=0 tx=0).stop 2024-09-16T05:14:11.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97c254f640 1 -- 172.21.15.114:0/2652402006 shutdown_connections 2024-09-16T05:14:11.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97c254f640 1 --2- 172.21.15.114:0/2652402006 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97bc13da70 0x7f97bc13fe60 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:11.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97c254f640 1 -- 172.21.15.114:0/2652402006 >> 172.21.15.114:0/2652402006 conn(0x7f97bc0a9c40 msgr2=0x7f97bc0aa050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:11.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97c254f640 1 -- 172.21.15.114:0/2652402006 shutdown_connections 2024-09-16T05:14:11.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97c254f640 1 -- 172.21.15.114:0/2652402006 wait complete. 2024-09-16T05:14:11.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97c254f640 10 monclient: build_initial_monmap 2024-09-16T05:14:11.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.168+0000 7f97c254f640 10 monclient: monmap: 2024-09-16T05:14:11.183 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:11.183 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:11.183 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:11.183 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:11.183 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:11.183 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:11.183 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:11.183 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:11.183 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:11.183 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:11.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 librados: starting msgr at 2024-09-16T05:14:11.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 librados: starting objecter 2024-09-16T05:14:11.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 Processor -- start 2024-09-16T05:14:11.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 -- start start 2024-09-16T05:14:11.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 librados: setting wanted keys 2024-09-16T05:14:11.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 librados: calling monclient init 2024-09-16T05:14:11.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient: init 2024-09-16T05:14:11.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:11.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:14:11.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97bc0a57b0 0x7f97bc1cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:11.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient(hunting): picked mon.noname-b con 0x7f97bc0a57b0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:11.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97bc137480 0x7f97bc1361d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:11.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient(hunting): picked mon.noname-a con 0x7f97bc137480 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:11.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97bc13da70 0x7f97bc09f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:11.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient(hunting): picked mon.noname-c con 0x7f97bc13da70 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:11.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:11.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc142bc0 con 0x7f97bc137480 2024-09-16T05:14:11.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:11.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc0a5e60 con 0x7f97bc0a57b0 2024-09-16T05:14:11.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:11.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f97bc142d40 con 0x7f97bc13da70 2024-09-16T05:14:11.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:11.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient(hunting): authenticate will time out at 1335.804909s 2024-09-16T05:14:11.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97bc137480 0x7f97bc1361d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:11.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97bc137480 0x7f97bc1361d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42110/0 (socket says 172.21.15.114:42110) 2024-09-16T05:14:11.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b37fe640 1 -- 172.21.15.114:0/2196282885 learned_addr learned my addr 172.21.15.114:0/2196282885 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:11.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b37fe640 10 monclient(hunting): get_auth_request con 0x7f97bc137480 auth_method 0 2024-09-16T05:14:11.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:11.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:11.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:11.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 1 --2- 172.21.15.114:0/2196282885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97bc13da70 0x7f97bc09f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:11.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 10 monclient(hunting): get_auth_request con 0x7f97bc13da70 auth_method 0 2024-09-16T05:14:11.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:11.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:11.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:11.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b37fe640 1 -- 172.21.15.114:0/2196282885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97bc137480 msgr2=0x7f97bc1361d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:14:11.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b37fe640 1 -- 172.21.15.114:0/2196282885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97bc137480 msgr2=0x7f97bc1361d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:14:11.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b37fe640 1 --2- 172.21.15.114:0/2196282885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97bc137480 0x7f97bc1361d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:11.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b37fe640 1 --2- 172.21.15.114:0/2196282885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97bc137480 0x7f97bc1361d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:11.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:11.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:11.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:11.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97bbfff640 1 --2- 172.21.15.114:0/2196282885 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97bc0a57b0 0x7f97bc1cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:11.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97bbfff640 10 monclient(hunting): get_auth_request con 0x7f97bc0a57b0 auth_method 0 2024-09-16T05:14:11.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:11.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:11.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:11.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:11.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:11.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:11.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 10 monclient(hunting): handle_auth_done global_id 54614 payload 824 2024-09-16T05:14:11.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 1 -- 172.21.15.114:0/2196282885 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97bc0a57b0 msgr2=0x7f97bc1cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:11.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 1 --2- 172.21.15.114:0/2196282885 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97bc0a57b0 0x7f97bc1cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:11.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 1 -- 172.21.15.114:0/2196282885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97bc137480 msgr2=0x7f97bc1361d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:14:11.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 1 --2- 172.21.15.114:0/2196282885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97bc137480 0x7f97bc1361d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:11.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 10 monclient: _finish_hunting 0 2024-09-16T05:14:11.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 1 monclient: found mon.noname-c 2024-09-16T05:14:11.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:11.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:14:11.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 1 -- 172.21.15.114:0/2196282885 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97bc13d660 con 0x7f97bc13da70 2024-09-16T05:14:11.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c0aec640 1 --2- 172.21.15.114:0/2196282885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97bc13da70 0x7f97bc09f010 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f97b40ac000 tx=0x7f97b40195a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:11.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 10 client.?.objecter ms_handle_connect 0x7f97bc13da70 2024-09-16T05:14:11.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:11.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 1 -- 172.21.15.114:0/2196282885 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b40b2050 con 0x7f97bc13da70 2024-09-16T05:14:11.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:11.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:14:11.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 10 monclient: dump: 2024-09-16T05:14:11.189 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:11.189 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:11.189 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:11.189 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:11.189 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:11.189 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:11.189 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:11.189 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:11.189 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:11.189 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:11.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 10 monclient: _finish_auth 0 2024-09-16T05:14:11.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 10 monclient: _check_auth_tickets 2024-09-16T05:14:11.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:11.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 5 monclient: authenticate success, global_id 54614 2024-09-16T05:14:11.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 1 -- 172.21.15.114:0/2196282885 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97b40b3040 con 0x7f97bc13da70 2024-09-16T05:14:11.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient: _renew_subs 2024-09-16T05:14:11.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:11.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 -- 172.21.15.114:0/2196282885 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97bc0a29a0 con 0x7f97bc13da70 2024-09-16T05:14:11.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:11.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 1 -- 172.21.15.114:0/2196282885 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b40ad850 con 0x7f97bc13da70 2024-09-16T05:14:11.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:11.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:11.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 10 monclient: dump: 2024-09-16T05:14:11.191 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:11.191 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:11.191 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:11.191 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:11.191 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:11.191 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:11.191 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:11.191 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:11.191 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:11.191 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:11.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 client.54614.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:11.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient: _renew_subs 2024-09-16T05:14:11.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:11.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 -- 172.21.15.114:0/2196282885 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f97bc0a26c0 con 0x7f97bc13da70 2024-09-16T05:14:11.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 librados: init done 2024-09-16T05:14:11.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97baffd640 4 set_mon_vals no callback set 2024-09-16T05:14:11.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 1 -- 172.21.15.114:0/2196282885 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f97b40c0020 con 0x7f97bc13da70 2024-09-16T05:14:11.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:11.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:11.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:11.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97c254f640 1 -- 172.21.15.114:0/2196282885 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97bc137860 con 0x7f97bc13da70 2024-09-16T05:14:11.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 1 --2- 172.21.15.114:0/2196282885 >> v2:172.21.15.149:6812/1185531135 conn(0x7f979c037840 0x7f979c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:11.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 1 -- 172.21.15.114:0/2196282885 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(283..283 src has 1..283) ==== 5390+0+0 (secure 0 0 0) 0x7f97b40e3840 con 0x7f97bc13da70 2024-09-16T05:14:11.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 10 client.54614.objecter ms_dispatch 0x7f97bc000ba0 osd_map(283..283 src has 1..283) 2024-09-16T05:14:11.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 3 client.54614.objecter handle_osd_map got epochs [283,283] > 0 2024-09-16T05:14:11.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 3 client.54614.objecter handle_osd_map decoding full epoch 283 2024-09-16T05:14:11.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97b97fa640 20 client.54614.objecter dump_active .. 0 homeless 2024-09-16T05:14:11.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97bbfff640 1 --2- 172.21.15.114:0/2196282885 >> v2:172.21.15.149:6812/1185531135 conn(0x7f979c037840 0x7f979c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:11.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.172+0000 7f97bbfff640 10 monclient: get_auth_request con 0x7f979c037840 auth_method 0 2024-09-16T05:14:11.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.176+0000 7f97b97fa640 10 client.54614.objecter ms_handle_connect 0x7f979c037840 2024-09-16T05:14:11.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.176+0000 7f97bbfff640 1 --2- 172.21.15.114:0/2196282885 >> v2:172.21.15.149:6812/1185531135 conn(0x7f979c037840 0x7f979c039d00 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 crypto rx=0x7f97ac001810 tx=0x7f97ac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:11.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.176+0000 7f97b97fa640 1 -- 172.21.15.114:0/2196282885 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f97bc13d660 con 0x7f97bc13da70 2024-09-16T05:14:11.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.176+0000 7f97b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:11.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.176+0000 7f97b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:11.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.400+0000 7f97c254f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:14:11.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.400+0000 7f97c254f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:11.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.400+0000 7f97c254f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:11.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.400+0000 7f97c254f640 1 -- 172.21.15.114:0/2196282885 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f97bc0a8d10 con 0x7f97bc13da70 2024-09-16T05:14:11.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97b97fa640 1 -- 172.21.15.114:0/2196282885 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f97b4016540 con 0x7f97bc13da70 2024-09-16T05:14:11.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:11.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97b97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:11.407 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":102,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:14:11.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 10 librados: watch_flush enter 2024-09-16T05:14:11.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 10 librados: watch_flush exit 2024-09-16T05:14:11.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 20 client.54614.objecter shutdown clearing up homeless session... 2024-09-16T05:14:11.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 10 client.54614.objecter successfully canceled tick 2024-09-16T05:14:11.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 1 -- 172.21.15.114:0/2196282885 >> v2:172.21.15.149:6812/1185531135 conn(0x7f979c037840 msgr2=0x7f979c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:11.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 1 --2- 172.21.15.114:0/2196282885 >> v2:172.21.15.149:6812/1185531135 conn(0x7f979c037840 0x7f979c039d00 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 crypto rx=0x7f97ac001810 tx=0x7f97ac009040 comp rx=0 tx=0).stop 2024-09-16T05:14:11.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 10 monclient: shutdown 2024-09-16T05:14:11.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:11.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 1 -- 172.21.15.114:0/2196282885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97bc13da70 msgr2=0x7f97bc09f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:11.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 1 --2- 172.21.15.114:0/2196282885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97bc13da70 0x7f97bc09f010 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f97b40ac000 tx=0x7f97b40195a0 comp rx=0 tx=0).stop 2024-09-16T05:14:11.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 1 -- 172.21.15.114:0/2196282885 shutdown_connections 2024-09-16T05:14:11.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 1 --2- 172.21.15.114:0/2196282885 >> v2:172.21.15.149:6812/1185531135 conn(0x7f979c037840 0x7f979c039d00 unknown :-1 s=CLOSED pgs=3492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:11.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 1 --2- 172.21.15.114:0/2196282885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97bc137480 0x7f97bc1361d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:11.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 1 --2- 172.21.15.114:0/2196282885 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97bc13da70 0x7f97bc09f010 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:11.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 1 --2- 172.21.15.114:0/2196282885 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97bc0a57b0 0x7f97bc1cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:11.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 1 -- 172.21.15.114:0/2196282885 >> 172.21.15.114:0/2196282885 conn(0x7f97bc0a9c40 msgr2=0x7f97bc0a8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:11.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 1 -- 172.21.15.114:0/2196282885 shutdown_connections 2024-09-16T05:14:11.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 1 -- 172.21.15.114:0/2196282885 wait complete. 2024-09-16T05:14:11.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:11.404+0000 7f97c254f640 1 librados: shutdown 2024-09-16T05:14:11.947 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP:163:head 2024-09-16T05:14:11.947 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:5d165639:::164:head 2024-09-16T05:14:11.947 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:f43765fc:::165:head 2024-09-16T05:14:11.947 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:b4c720e9:::166:head 2024-09-16T05:14:11.947 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:e694b040:::167:head 2024-09-16T05:14:11.947 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:afa38db2:::168:head 2024-09-16T05:14:11.947 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:77ba9f53:::169:head 2024-09-16T05:14:11.947 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:87495034:::170:head 2024-09-16T05:14:11.947 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:7c96bf0e:::171:head 2024-09-16T05:14:11.947 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:dbe346cc:::172:head 2024-09-16T05:14:11.947 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:e943ec24:::173:head 2024-09-16T05:14:11.948 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:f97a9c0c:::174:head 2024-09-16T05:14:11.948 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:6f26e74d:::175:head 2024-09-16T05:14:11.948 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:4f95e106:::176:head 2024-09-16T05:14:11.948 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:0e6f2f8f:::177:head 2024-09-16T05:14:11.948 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:05db05f1:::178:head 2024-09-16T05:14:11.948 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:38a78d66:::179:head 2024-09-16T05:14:11.948 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:d095610b:::180:head 2024-09-16T05:14:11.948 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:a1a9d709:::181:head 2024-09-16T05:14:11.948 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:1e5d39db:::182:head 2024-09-16T05:14:11.948 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:f7df4fb9:::183:head 2024-09-16T05:14:11.948 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:03a7f161:::184:head 2024-09-16T05:14:11.948 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:ba70721e:::185:head 2024-09-16T05:14:11.948 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:28e5662d:::186:head 2024-09-16T05:14:11.949 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:973d52de:::187:head 2024-09-16T05:14:11.949 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:4303eb1c:::188:head 2024-09-16T05:14:11.949 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:b990b48e:::189:head 2024-09-16T05:14:11.949 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:29b8165b:::190:head 2024-09-16T05:14:11.949 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:3547f197:::191:head 2024-09-16T05:14:11.949 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:7e260936:::192:head 2024-09-16T05:14:11.949 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:1abec7b1:::193:head 2024-09-16T05:14:11.949 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:10fdda93:::194:head 2024-09-16T05:14:11.949 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:15817eea:::195:head 2024-09-16T05:14:11.949 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:770bab57:::196:head 2024-09-16T05:14:11.949 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:ed9e13e7:::197:head 2024-09-16T05:14:11.949 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:71471a8f:::198:head 2024-09-16T05:14:11.949 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: checking for 270:10fb1d02:::199:head 2024-09-16T05:14:11.950 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8993 ms) 2024-09-16T05:14:11.950 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-09-16T05:14:11.950 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726463606,0 2024-09-16T05:14:11.950 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: first is 1726463606 2024-09-16T05:14:11.950 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726463606,0 2024-09-16T05:14:11.950 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726463606,0 2024-09-16T05:14:11.950 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726463606,1726463608,1726463609,0 2024-09-16T05:14:11.950 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726463606,1726463608,1726463609,0 2024-09-16T05:14:11.950 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726463606,1726463608,1726463609,0 2024-09-16T05:14:11.950 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726463606,1726463608,1726463609,1726463611,1726463612,0 2024-09-16T05:14:11.950 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726463606,1726463608,1726463609,1726463611,1726463612,0 2024-09-16T05:14:11.950 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726463606,1726463608,1726463609,1726463611,1726463612,0 2024-09-16T05:14:11.950 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726463606,1726463608,1726463609,1726463611,1726463612,1726463614,1726463615,0 2024-09-16T05:14:11.950 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726463606,1726463608,1726463609,1726463611,1726463612,1726463614,1726463615,0 2024-09-16T05:14:11.951 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726463606,1726463608,1726463609,1726463611,1726463612,1726463614,1726463615,0 2024-09-16T05:14:11.951 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726463609,1726463611,1726463612,1726463614,1726463615,1726463617,1726463618,0 2024-09-16T05:14:11.951 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: first now 1726463609, trimmed 2024-09-16T05:14:11.951 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21041 ms) 2024-09-16T05:14:11.951 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-09-16T05:14:11.951 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: foo0 2024-09-16T05:14:11.951 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-16T05:14:11.951 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (13971 ms) 2024-09-16T05:14:11.951 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-09-16T05:14:14.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:14:14.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 10 monclient: get_monmap_and_config 2024-09-16T05:14:14.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 10 monclient: build_initial_monmap 2024-09-16T05:14:14.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 10 monclient: monmap: 2024-09-16T05:14:14.556 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:14.556 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:14.556 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:14.557753+0000 2024-09-16T05:14:14.556 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:14.557753+0000 2024-09-16T05:14:14.556 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:14.556 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:14.556 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:14.556 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:14.557 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:14.557 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:14.557 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:14.557 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:14.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 1 Processor -- start 2024-09-16T05:14:14.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 1 -- start start 2024-09-16T05:14:14.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 10 monclient: init 2024-09-16T05:14:14.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:14.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-16T05:14:14.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 0x7f40940a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:14.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 10 monclient(hunting): picked mon.noname-c con 0x7f409412d7b0 addr v2:172.21.15.149:3300/0 2024-09-16T05:14:14.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f409412db90 addr v1:172.21.15.149:6789/0 2024-09-16T05:14:14.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 0x7f4094072c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:14.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 10 monclient(hunting): picked mon.noname-a con 0x7f409412dfd0 addr v2:172.21.15.114:3300/0 2024-09-16T05:14:14.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40940a13a0 con 0x7f409412db90 2024-09-16T05:14:14.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:14.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f40940a1680 con 0x7f409412dfd0 2024-09-16T05:14:14.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:14.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f409409cab0 con 0x7f409412d7b0 2024-09-16T05:14:14.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:14.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409bdd3640 10 monclient(hunting): authenticate will time out at 1339.187683s 2024-09-16T05:14:14.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409a370640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 0x7f4094072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:14.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409a370640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 0x7f4094072c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42130/0 (socket says 172.21.15.114:42130) 2024-09-16T05:14:14.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409a370640 1 -- 172.21.15.114:0/450079200 learned_addr learned my addr 172.21.15.114:0/450079200 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:14.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409a370640 10 monclient(hunting): get_auth_request con 0x7f409412dfd0 auth_method 0 2024-09-16T05:14:14.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409a370640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:14.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409a370640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:14.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.552+0000 7f409a370640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:14.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.556+0000 7f409a370640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 msgr2=0x7f4094072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:14:14.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.556+0000 7f409a370640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 msgr2=0x7f4094072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:14.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.556+0000 7f409a370640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 0x7f4094072c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:14.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.556+0000 7f409a370640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 0x7f4094072c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:14.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.556+0000 7f4099b6f640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 0x7f40940a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:14.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.556+0000 7f4099b6f640 10 monclient(hunting): get_auth_request con 0x7f409412d7b0 auth_method 0 2024-09-16T05:14:14.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.556+0000 7f4099b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:14.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.556+0000 7f4099b6f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:14.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.556+0000 7f4099b6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:14.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.556+0000 7f4099b6f640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 msgr2=0x7f40940a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:14:14.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.556+0000 7f4099b6f640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 msgr2=0x7f40940a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:14.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.556+0000 7f4099b6f640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 0x7f40940a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:14.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.556+0000 7f4099b6f640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 0x7f40940a1e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:14.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f409a370640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 0x7f4094072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:14.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f409a370640 10 monclient(hunting): get_auth_request con 0x7f409412dfd0 auth_method 0 2024-09-16T05:14:14.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f409a370640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:14.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f409a370640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:14.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f409a370640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:14.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f4099b6f640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 0x7f40940a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:14.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f4099b6f640 10 monclient(hunting): get_auth_request con 0x7f409412d7b0 auth_method 0 2024-09-16T05:14:14.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f4099b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:14.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f4099b6f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:14.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f4099b6f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:14.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f409a370640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 msgr2=0x7f4094072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:14:14.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f4099b6f640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 msgr2=0x7f40940a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:14:14.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f4099b6f640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 msgr2=0x7f40940a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:14.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f409a370640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 msgr2=0x7f4094072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:14.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f4099b6f640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 0x7f40940a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:14.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f409a370640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 0x7f4094072c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:14.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f4099b6f640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 0x7f40940a1e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:14:14.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:14.760+0000 7f409a370640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 0x7f4094072c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:14:15.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f409a370640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 0x7f4094072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:15.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f409a370640 10 monclient(hunting): get_auth_request con 0x7f409412dfd0 auth_method 0 2024-09-16T05:14:15.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f409a370640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:15.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f409a370640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:15.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f409a370640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:15.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f4099b6f640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 0x7f40940a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:15.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f4099b6f640 10 monclient(hunting): get_auth_request con 0x7f409412d7b0 auth_method 0 2024-09-16T05:14:15.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f4099b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:15.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f4099b6f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:15.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f4099b6f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:15.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f409a370640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 msgr2=0x7f4094072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:14:15.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f409a370640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 msgr2=0x7f4094072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:15.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f409a370640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 0x7f4094072c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:15.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f409a370640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 0x7f4094072c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:14:15.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f4099b6f640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 msgr2=0x7f40940a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:14:15.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f4099b6f640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 msgr2=0x7f40940a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:15.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f4099b6f640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 0x7f40940a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:15.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.164+0000 7f4099b6f640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 0x7f40940a1e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:14:15.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f409a370640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 0x7f4094072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:15.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f409a370640 10 monclient(hunting): get_auth_request con 0x7f409412dfd0 auth_method 0 2024-09-16T05:14:15.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f409a370640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:15.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f409a370640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:15.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f409a370640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:15.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f4099b6f640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 0x7f40940a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:15.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f409a370640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 msgr2=0x7f4094072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:14:15.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f409a370640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 msgr2=0x7f4094072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:15.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f409a370640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 0x7f4094072c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:15.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f4099b6f640 10 monclient(hunting): get_auth_request con 0x7f409412d7b0 auth_method 0 2024-09-16T05:14:15.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f4099b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:15.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f4099b6f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:15.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f4099b6f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:15.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f409a370640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 0x7f4094072c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:14:15.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f4099b6f640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 msgr2=0x7f40940a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:14:15.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f4099b6f640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 msgr2=0x7f40940a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:15.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f4099b6f640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 0x7f40940a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:15.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:15.968+0000 7f4099b6f640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 0x7f40940a1e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:14:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 10 monclient(hunting): tick 2024-09-16T05:14:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:14:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 1 monclient(hunting): continuing hunt 2024-09-16T05:14:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:14:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 msgr2=0x7f40940a1e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:14:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.149:3300/0 conn(0x7f409412d7b0 0x7f40940a1e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 1 -- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 msgr2=0x7f4094072c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:14:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 1 --2- 172.21.15.114:0/450079200 >> v2:172.21.15.114:3300/0 conn(0x7f409412dfd0 0x7f4094072c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 1 -- 172.21.15.114:0/450079200 >> v1:172.21.15.149:6789/0 conn(0x7f409412db90 legacy=0x7f4094072540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 10 monclient: _add_conns ranks=[0,2] 2024-09-16T05:14:17.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40780036c0 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:17.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 0x7f4078009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:17.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f4078007370 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:17.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 1 -- 172.21.15.114:0/450079200 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40940684b0 con 0x7f40780036c0 2024-09-16T05:14:17.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:17.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 1 -- 172.21.15.114:0/450079200 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f40940a1680 con 0x7f4078007370 2024-09-16T05:14:17.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f408b7fe640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:17.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f409936e640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 0x7f4078009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:17.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f409936e640 10 monclient(hunting): get_auth_request con 0x7f4078007370 auth_method 0 2024-09-16T05:14:17.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f409936e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:17.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f409936e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:17.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f409936e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:17.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f409936e640 1 -- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 msgr2=0x7f4078009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:14:17.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f409936e640 1 -- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 msgr2=0x7f4078009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:17.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f409936e640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 0x7f4078009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:17.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.556+0000 7f409936e640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 0x7f4078009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.760+0000 7f409936e640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 0x7f4078009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.760+0000 7f409936e640 10 monclient(hunting): get_auth_request con 0x7f4078007370 auth_method 0 2024-09-16T05:14:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.760+0000 7f409936e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.760+0000 7f409936e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:17.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.760+0000 7f409936e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:17.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.760+0000 7f409936e640 1 -- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 msgr2=0x7f4078009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:14:17.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.760+0000 7f409936e640 1 -- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 msgr2=0x7f4078009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:17.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.760+0000 7f409936e640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 0x7f4078009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:17.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:17.760+0000 7f409936e640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 0x7f4078009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:14:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.164+0000 7f409936e640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 0x7f4078009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.164+0000 7f409936e640 10 monclient(hunting): get_auth_request con 0x7f4078007370 auth_method 0 2024-09-16T05:14:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.164+0000 7f409936e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.164+0000 7f409936e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.164+0000 7f409936e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:18.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.164+0000 7f409936e640 1 -- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 msgr2=0x7f4078009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:14:18.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.164+0000 7f409936e640 1 -- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 msgr2=0x7f4078009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:18.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.164+0000 7f409936e640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 0x7f4078009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:18.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.164+0000 7f409936e640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 0x7f4078009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:14:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.812+0000 7f4098b6d640 1 -- 172.21.15.114:0/450079200 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1315736797 0 0) 0x7f40940684b0 con 0x7f40780036c0 2024-09-16T05:14:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.812+0000 7f4098b6d640 10 monclient(hunting): _negotiate 2024-09-16T05:14:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.812+0000 7f4098b6d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.812+0000 7f4098b6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.812+0000 7f4098b6d640 10 monclient(hunting): my global_id is 74097 2024-09-16T05:14:18.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.812+0000 7f4098b6d640 1 -- 172.21.15.114:0/450079200 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4080003530 con 0x7f40780036c0 2024-09-16T05:14:18.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 1 -- 172.21.15.114:0/450079200 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1763156926 0 0) 0x7f4080003530 con 0x7f40780036c0 2024-09-16T05:14:18.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 1 -- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 msgr2=0x7f4078009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:14:18.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4078007370 0x7f4078009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:18.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient: _finish_hunting 0 2024-09-16T05:14:18.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:14:18.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:18.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:14:18.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 1 -- 172.21.15.114:0/450079200 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f407800b000 con 0x7f40780036c0 2024-09-16T05:14:18.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient: _finish_auth 0 2024-09-16T05:14:18.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient: _check_auth_tickets 2024-09-16T05:14:18.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:18.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 1 -- 172.21.15.114:0/450079200 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f409000f040 con 0x7f40780036c0 2024-09-16T05:14:18.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:18.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:14:18.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient: dump: 2024-09-16T05:14:18.821 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:18.821 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:18.821 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:18.821 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:18.822 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:18.822 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:18.822 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:18.822 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:18.822 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:18.822 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:18.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:14:18.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:18.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 1 -- 172.21.15.114:0/450079200 >> v1:172.21.15.114:6789/0 conn(0x7f40780036c0 legacy=0x7f4078005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:18.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 1 -- 172.21.15.114:0/450079200 reap_dead start 2024-09-16T05:14:18.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:14:18.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40800039c0 0x7f4080003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:18.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient(hunting): picked mon.b con 0x7f40800039c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:18.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4080004810 0x7f4080004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:18.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient(hunting): picked mon.a con 0x7f4080004810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:18.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 0x7f408000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:18.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient(hunting): picked mon.c con 0x7f40800052a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:18.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:18.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 1 -- 172.21.15.114:0/450079200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f40940a1680 con 0x7f4080004810 2024-09-16T05:14:18.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:18.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 1 -- 172.21.15.114:0/450079200 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f408000c1f0 con 0x7f40800039c0 2024-09-16T05:14:18.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:18.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 1 -- 172.21.15.114:0/450079200 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f408000c3f0 con 0x7f40800052a0 2024-09-16T05:14:18.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4098b6d640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:14:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 0x7f408000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409936e640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4080004810 0x7f4080004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 10 monclient(hunting): get_auth_request con 0x7f40800052a0 auth_method 0 2024-09-16T05:14:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409936e640 10 monclient(hunting): get_auth_request con 0x7f4080004810 auth_method 0 2024-09-16T05:14:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409936e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409936e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409936e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:18.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:18.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:18.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:14:18.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409936e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:18.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4099b6f640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40800039c0 0x7f4080003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:18.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409936e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:18.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409936e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:14:18.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4099b6f640 10 monclient(hunting): get_auth_request con 0x7f40800039c0 auth_method 0 2024-09-16T05:14:18.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4099b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:18.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4099b6f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:18.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f4099b6f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:18.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 10 monclient(hunting): handle_auth_done global_id 74097 payload 293 2024-09-16T05:14:18.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 1 -- 172.21.15.114:0/450079200 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40800039c0 msgr2=0x7f4080003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:18.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40800039c0 0x7f4080003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:18.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 1 -- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4080004810 msgr2=0x7f4080004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:18.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4080004810 0x7f4080004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:18.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 10 monclient: _finish_hunting 0 2024-09-16T05:14:18.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 1 monclient: found mon.c 2024-09-16T05:14:18.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:18.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:18.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 1 -- 172.21.15.114:0/450079200 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4080003530 con 0x7f40800052a0 2024-09-16T05:14:18.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 10 monclient: _finish_auth 0 2024-09-16T05:14:18.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 10 monclient: _check_auth_tickets 2024-09-16T05:14:18.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409a370640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:18.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409bdd3640 5 monclient: authenticate success, global_id 74097 2024-09-16T05:14:18.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.816+0000 7f409bdd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.820+0000 7f409a370640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 0x7f408000b9d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f409412e3b0 tx=0x7f408c025840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c32e33070f7aadb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.820+0000 7f4098b6d640 1 -- 172.21.15.114:0/450079200 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c0096d0 con 0x7f40800052a0 2024-09-16T05:14:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.820+0000 7f4098b6d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.820+0000 7f4098b6d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.820+0000 7f4098b6d640 10 monclient: dump: 2024-09-16T05:14:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:18.830 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.820+0000 7f4098b6d640 1 -- 172.21.15.114:0/450079200 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408c0b9070 con 0x7f40800052a0 2024-09-16T05:14:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.820+0000 7f4098b6d640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.820+0000 7f4098b6d640 1 -- 172.21.15.114:0/450079200 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c0b4410 con 0x7f40800052a0 2024-09-16T05:14:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.820+0000 7f4098b6d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.820+0000 7f4098b6d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.820+0000 7f4098b6d640 10 monclient: dump: 2024-09-16T05:14:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:18.831 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:18.831 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:18.831 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:18.831 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:18.831 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:18.831 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:18.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.820+0000 7f409bdd3640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:18.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.820+0000 7f409bdd3640 4 set_mon_vals no callback set 2024-09-16T05:14:18.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 10 monclient: shutdown 2024-09-16T05:14:18.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:18.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 -- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 msgr2=0x7f408000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:18.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 0x7f408000b9d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f409412e3b0 tx=0x7f408c025840 comp rx=0 tx=0).stop 2024-09-16T05:14:18.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 -- 172.21.15.114:0/450079200 shutdown_connections 2024-09-16T05:14:18.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4080004810 0x7f4080004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:18.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 0x7f408000b9d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:18.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 --2- 172.21.15.114:0/450079200 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40800039c0 0x7f4080003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:18.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 -- 172.21.15.114:0/450079200 >> 172.21.15.114:0/450079200 conn(0x7f40940a6380 msgr2=0x7f40940a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:18.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 -- 172.21.15.114:0/450079200 shutdown_connections 2024-09-16T05:14:18.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 -- 172.21.15.114:0/450079200 wait complete. 2024-09-16T05:14:18.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 10 monclient: build_initial_monmap 2024-09-16T05:14:18.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 10 monclient: monmap: 2024-09-16T05:14:18.832 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:18.832 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:18.833 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:18.833 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:18.833 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:18.833 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:18.833 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:18.833 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:18.833 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:18.833 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:18.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 librados: starting msgr at 2024-09-16T05:14:18.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 librados: starting objecter 2024-09-16T05:14:18.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 Processor -- start 2024-09-16T05:14:18.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 -- start start 2024-09-16T05:14:18.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 librados: setting wanted keys 2024-09-16T05:14:18.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 librados: calling monclient init 2024-09-16T05:14:18.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 10 monclient: init 2024-09-16T05:14:18.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:18.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:14:18.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40800039c0 0x7f409411c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:18.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 10 monclient(hunting): picked mon.noname-b con 0x7f40800039c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:18.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4080004810 0x7f409411c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:18.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 10 monclient(hunting): picked mon.noname-a con 0x7f4080004810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:18.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 0x7f409411cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:18.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 10 monclient(hunting): picked mon.noname-c con 0x7f40800052a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:18.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:18.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4094070ce0 con 0x7f4080004810 2024-09-16T05:14:18.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:18.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f4094070b60 con 0x7f40800039c0 2024-09-16T05:14:18.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:18.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4094070e60 con 0x7f40800052a0 2024-09-16T05:14:18.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:18.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409bdd3640 10 monclient(hunting): authenticate will time out at 1343.459337s 2024-09-16T05:14:18.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4080004810 0x7f409411c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:18.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409a370640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 0x7f409411cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:18.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4080004810 0x7f409411c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37120/0 (socket says 172.21.15.114:37120) 2024-09-16T05:14:18.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409a370640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 0x7f409411cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51238/0 (socket says 172.21.15.114:51238) 2024-09-16T05:14:18.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 1 -- 172.21.15.114:0/723364856 learned_addr learned my addr 172.21.15.114:0/723364856 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:18.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 10 monclient(hunting): get_auth_request con 0x7f4080004810 auth_method 0 2024-09-16T05:14:18.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:18.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:18.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:18.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409a370640 10 monclient(hunting): get_auth_request con 0x7f40800052a0 auth_method 0 2024-09-16T05:14:18.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409a370640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:18.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409a370640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409a370640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409a370640 1 -- 172.21.15.114:0/723364856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 msgr2=0x7f409411cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T05:14:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409a370640 1 -- 172.21.15.114:0/723364856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 msgr2=0x7f409411cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:14:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f4099b6f640 1 --2- 172.21.15.114:0/723364856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40800039c0 0x7f409411c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409a370640 1 --2- 172.21.15.114:0/723364856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 0x7f409411cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409a370640 1 --2- 172.21.15.114:0/723364856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 0x7f409411cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f4099b6f640 10 monclient(hunting): get_auth_request con 0x7f40800039c0 auth_method 0 2024-09-16T05:14:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f4099b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f4099b6f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f4099b6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:18.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 10 monclient(hunting): handle_auth_done global_id 74100 payload 824 2024-09-16T05:14:18.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 1 -- 172.21.15.114:0/723364856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 msgr2=0x7f409411cdd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:14:18.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 1 --2- 172.21.15.114:0/723364856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 0x7f409411cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:18.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 1 -- 172.21.15.114:0/723364856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40800039c0 msgr2=0x7f409411c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:18.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 1 --2- 172.21.15.114:0/723364856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40800039c0 0x7f409411c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:18.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 10 monclient: _finish_hunting 0 2024-09-16T05:14:18.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 1 monclient: found mon.noname-a 2024-09-16T05:14:18.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:18.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:18.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f409936e640 1 -- 172.21.15.114:0/723364856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409411f5b0 con 0x7f4080004810 2024-09-16T05:14:18.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.824+0000 7f4099b6f640 1 --2- 172.21.15.114:0/723364856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40800039c0 0x7f409411c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:14:18.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f409936e640 1 --2- 172.21.15.114:0/723364856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4080004810 0x7f409411c890 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4084014730 tx=0x7f4084014bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:18.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 10 client.?.objecter ms_handle_connect 0x7f4080004810 2024-09-16T05:14:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 1 -- 172.21.15.114:0/723364856 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408401a070 con 0x7f4080004810 2024-09-16T05:14:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 10 monclient: dump: 2024-09-16T05:14:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:18.844 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:14:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:14:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 1 -- 172.21.15.114:0/723364856 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4084014e40 con 0x7f4080004810 2024-09-16T05:14:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f409bdd3640 5 monclient: authenticate success, global_id 74100 2024-09-16T05:14:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f409bdd3640 10 monclient: _renew_subs 2024-09-16T05:14:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f409bdd3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f409bdd3640 1 -- 172.21.15.114:0/723364856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f409411f890 con 0x7f4080004810 2024-09-16T05:14:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:18.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 1 -- 172.21.15.114:0/723364856 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40840128a0 con 0x7f4080004810 2024-09-16T05:14:18.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:18.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:18.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 10 monclient: dump: 2024-09-16T05:14:18.845 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:18.845 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:18.845 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:18.845 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:18.845 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:18.845 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:18.845 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:18.845 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:18.845 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:18.846 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408bfff640 4 set_mon_vals no callback set 2024-09-16T05:14:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f409bdd3640 10 client.74100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f409bdd3640 10 monclient: _renew_subs 2024-09-16T05:14:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f409bdd3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f409bdd3640 1 -- 172.21.15.114:0/723364856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f409411e3e0 con 0x7f4080004810 2024-09-16T05:14:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f409bdd3640 1 librados: init done 2024-09-16T05:14:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 1 -- 172.21.15.114:0/723364856 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f408401d070 con 0x7f4080004810 2024-09-16T05:14:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f409bdd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f409bdd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:18.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f409bdd3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:18.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 1 --2- 172.21.15.114:0/723364856 >> v2:172.21.15.149:6812/1185531135 conn(0x7f40780385e0 0x7f407803aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:18.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f409bdd3640 1 -- 172.21.15.114:0/723364856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4094071190 con 0x7f4080004810 2024-09-16T05:14:18.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 1 -- 172.21.15.114:0/723364856 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(285..285 src has 1..285) ==== 5811+0+0 (secure 0 0 0) 0x7f4084049580 con 0x7f4080004810 2024-09-16T05:14:18.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 10 client.74100.objecter ms_dispatch 0x7f4094000ba0 osd_map(285..285 src has 1..285) 2024-09-16T05:14:18.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 3 client.74100.objecter handle_osd_map got epochs [285,285] > 0 2024-09-16T05:14:18.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 3 client.74100.objecter handle_osd_map decoding full epoch 285 2024-09-16T05:14:18.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 20 client.74100.objecter dump_active .. 0 homeless 2024-09-16T05:14:18.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f4099b6f640 1 --2- 172.21.15.114:0/723364856 >> v2:172.21.15.149:6812/1185531135 conn(0x7f40780385e0 0x7f407803aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:18.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f4099b6f640 10 monclient: get_auth_request con 0x7f40780385e0 auth_method 0 2024-09-16T05:14:18.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f408a7fc640 10 client.74100.objecter ms_handle_connect 0x7f40780385e0 2024-09-16T05:14:18.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.828+0000 7f4099b6f640 1 --2- 172.21.15.114:0/723364856 >> v2:172.21.15.149:6812/1185531135 conn(0x7f40780385e0 0x7f407803aaa0 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 crypto rx=0x7f4090006aa0 tx=0x7f4090006a30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:18.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.832+0000 7f408a7fc640 1 -- 172.21.15.114:0/723364856 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f4084012a40 con 0x7f4080004810 2024-09-16T05:14:18.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.832+0000 7f408a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:18.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:18.832+0000 7f408a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:19.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.064+0000 7f409bdd3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:14:19.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.064+0000 7f409bdd3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:19.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.064+0000 7f409bdd3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:19.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.064+0000 7f409bdd3640 1 -- 172.21.15.114:0/723364856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f409406bf20 con 0x7f4080004810 2024-09-16T05:14:19.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.064+0000 7f408a7fc640 1 -- 172.21.15.114:0/723364856 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f40840157b0 con 0x7f4080004810 2024-09-16T05:14:19.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.064+0000 7f408a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:19.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.064+0000 7f408a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:19.070 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":26,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:14:19.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 10 librados: watch_flush enter 2024-09-16T05:14:19.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 10 librados: watch_flush exit 2024-09-16T05:14:19.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 20 client.74100.objecter shutdown clearing up homeless session... 2024-09-16T05:14:19.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 10 client.74100.objecter successfully canceled tick 2024-09-16T05:14:19.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 1 -- 172.21.15.114:0/723364856 >> v2:172.21.15.149:6812/1185531135 conn(0x7f40780385e0 msgr2=0x7f407803aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:19.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 1 --2- 172.21.15.114:0/723364856 >> v2:172.21.15.149:6812/1185531135 conn(0x7f40780385e0 0x7f407803aaa0 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 crypto rx=0x7f4090006aa0 tx=0x7f4090006a30 comp rx=0 tx=0).stop 2024-09-16T05:14:19.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 10 monclient: shutdown 2024-09-16T05:14:19.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:19.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 1 -- 172.21.15.114:0/723364856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4080004810 msgr2=0x7f409411c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:19.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 1 --2- 172.21.15.114:0/723364856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4080004810 0x7f409411c890 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4084014730 tx=0x7f4084014bf0 comp rx=0 tx=0).stop 2024-09-16T05:14:19.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 1 -- 172.21.15.114:0/723364856 shutdown_connections 2024-09-16T05:14:19.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 1 --2- 172.21.15.114:0/723364856 >> v2:172.21.15.149:6812/1185531135 conn(0x7f40780385e0 0x7f407803aaa0 unknown :-1 s=CLOSED pgs=3493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:19.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 1 --2- 172.21.15.114:0/723364856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4080004810 0x7f409411c890 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:19.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 1 --2- 172.21.15.114:0/723364856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40800052a0 0x7f409411cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:19.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 1 --2- 172.21.15.114:0/723364856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40800039c0 0x7f409411c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:19.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 1 -- 172.21.15.114:0/723364856 >> 172.21.15.114:0/723364856 conn(0x7f40940a6380 msgr2=0x7f409409efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:19.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 1 -- 172.21.15.114:0/723364856 shutdown_connections 2024-09-16T05:14:19.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 1 -- 172.21.15.114:0/723364856 wait complete. 2024-09-16T05:14:19.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:19.068+0000 7f409bdd3640 1 librados: shutdown 2024-09-16T05:14:22.089 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:14:22.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 10 monclient: get_monmap_and_config 2024-09-16T05:14:22.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 10 monclient: build_initial_monmap 2024-09-16T05:14:22.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 10 monclient: monmap: 2024-09-16T05:14:22.220 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:22.220 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:22.220 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:22.221531+0000 2024-09-16T05:14:22.220 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:22.221531+0000 2024-09-16T05:14:22.220 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:22.220 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:22.220 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:22.220 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:22.220 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:22.220 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:22.220 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:22.220 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:22.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 1 Processor -- start 2024-09-16T05:14:22.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 1 -- start start 2024-09-16T05:14:22.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 10 monclient: init 2024-09-16T05:14:22.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:22.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T05:14:22.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71b8137730 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:22.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b813c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:22.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 10 monclient(hunting): picked mon.noname-b con 0x7f71b80a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:22.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71b813ca00 addr v1:172.21.15.149:6789/0 2024-09-16T05:14:22.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71b80a55e0 con 0x7f71b8137730 2024-09-16T05:14:22.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71b80a3710 con 0x7f71b813ca00 2024-09-16T05:14:22.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:22.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f71b809c450 con 0x7f71b80a5d00 2024-09-16T05:14:22.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:22.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bf06a640 10 monclient(hunting): authenticate will time out at 1346.851474s 2024-09-16T05:14:22.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71b7fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b813c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:22.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bce06640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f71b8137730 0x7f71b8137b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38554/0 (socket says 172.21.15.114:38554) 2024-09-16T05:14:22.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71bce06640 1 -- 172.21.15.114:0/1249733576 learned_addr learned my addr 172.21.15.114:0/1249733576 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:14:22.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71b7fff640 10 monclient(hunting): get_auth_request con 0x7f71b80a5d00 auth_method 0 2024-09-16T05:14:22.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:22.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:22.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:22.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71b7fff640 1 -- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 msgr2=0x7f71b813c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:14:22.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71b7fff640 1 -- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 msgr2=0x7f71b813c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:22.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71b7fff640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b813c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:22.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.216+0000 7f71b7fff640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b813c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.420+0000 7f71b7fff640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b813c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.420+0000 7f71b7fff640 10 monclient(hunting): get_auth_request con 0x7f71b80a5d00 auth_method 0 2024-09-16T05:14:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.420+0000 7f71b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.420+0000 7f71b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.420+0000 7f71b7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.420+0000 7f71b7fff640 1 -- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 msgr2=0x7f71b813c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:14:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.420+0000 7f71b7fff640 1 -- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 msgr2=0x7f71b813c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:22.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.420+0000 7f71b7fff640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b813c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:22.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.420+0000 7f71b7fff640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b813c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:14:22.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.824+0000 7f71b7fff640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b813c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:22.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.824+0000 7f71b7fff640 10 monclient(hunting): get_auth_request con 0x7f71b80a5d00 auth_method 0 2024-09-16T05:14:22.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.824+0000 7f71b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:22.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.824+0000 7f71b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:22.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.824+0000 7f71b7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:22.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.824+0000 7f71b7fff640 1 -- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 msgr2=0x7f71b813c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:14:22.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.824+0000 7f71b7fff640 1 -- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 msgr2=0x7f71b813c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:22.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.824+0000 7f71b7fff640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b813c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:22.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:22.824+0000 7f71b7fff640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b813c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:14:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:23.628+0000 7f71b7fff640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b813c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:23.628+0000 7f71b7fff640 10 monclient(hunting): get_auth_request con 0x7f71b80a5d00 auth_method 0 2024-09-16T05:14:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:23.628+0000 7f71b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:23.628+0000 7f71b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:23.628+0000 7f71b7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:23.628+0000 7f71b7fff640 1 -- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 msgr2=0x7f71b813c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:14:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:23.628+0000 7f71b7fff640 1 -- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 msgr2=0x7f71b813c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:23.628+0000 7f71b7fff640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b813c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:23.628+0000 7f71b7fff640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b813c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:14:24.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1683456820 0 0) 0x7f71b80a3710 con 0x7f71b813ca00 2024-09-16T05:14:24.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:14:24.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:24.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:24.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): my global_id is 64330 2024-09-16T05:14:24.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71a0003530 con 0x7f71b813ca00 2024-09-16T05:14:24.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1049803991 0 0) 0x7f71b80a55e0 con 0x7f71b8137730 2024-09-16T05:14:24.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:14:24.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:24.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:24.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): my global_id is 74103 2024-09-16T05:14:24.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71b80a3710 con 0x7f71b8137730 2024-09-16T05:14:24.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 <== mon.1 v1:172.21.15.149:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1462088117 0 0) 0x7f71a0003530 con 0x7f71b813ca00 2024-09-16T05:14:24.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 msgr2=0x7f71b813c4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:14:24.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b813c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:24.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 >> v1:172.21.15.114:6789/0 conn(0x7f71b8137730 legacy=0x7f71b8137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:24.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient: _finish_hunting 0 2024-09-16T05:14:24.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 monclient: found mon.noname-c-legacy 2024-09-16T05:14:24.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:24.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.149:6789/0 2024-09-16T05:14:24.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 --> v1:172.21.15.149:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b809be00 con 0x7f71b813ca00 2024-09-16T05:14:24.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient: _finish_auth 0 2024-09-16T05:14:24.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient: _check_auth_tickets 2024-09-16T05:14:24.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:24.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 <== mon.1 v1:172.21.15.149:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f71b0001680 con 0x7f71b813ca00 2024-09-16T05:14:24.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bf06a640 5 monclient: authenticate success, global_id 64330 2024-09-16T05:14:24.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bf06a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:24.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:24.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T05:14:24.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient: dump: 2024-09-16T05:14:24.772 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:24.772 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:24.772 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:24.773 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:24.773 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:24.773 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:24.773 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:24.773 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:24.773 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:24.773 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:24.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] but i'm connected to v1:172.21.15.149:6789/0, reconnecting 2024-09-16T05:14:24.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:24.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 >> v1:172.21.15.149:6789/0 conn(0x7f71b813ca00 legacy=0x7f71b813edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:24.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:14:24.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71a0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:24.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): picked mon.c con 0x7f71b80a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:24.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a0004100 0x7f71a0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:24.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): picked mon.a con 0x7f71a0004100 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:24.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f71a0004da0 0x7f71a0005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:24.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): picked mon.b con 0x7f71a0004da0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:24.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:24.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f71b809c450 con 0x7f71a0004100 2024-09-16T05:14:24.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:24.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f71a000bcb0 con 0x7f71a0004da0 2024-09-16T05:14:24.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:24.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f71a000beb0 con 0x7f71b80a5d00 2024-09-16T05:14:24.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:24.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71b77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:14:24.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bf06a640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:24.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bce06640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71a0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:24.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bce06640 10 monclient(hunting): get_auth_request con 0x7f71b80a5d00 auth_method 0 2024-09-16T05:14:24.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bce06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:24.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bce06640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:24.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bce06640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:24.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bd607640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f71a0004da0 0x7f71a0005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:24.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bce06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:24.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bce06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:24.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bce06640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:14:24.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bd607640 10 monclient(hunting): get_auth_request con 0x7f71a0004da0 auth_method 0 2024-09-16T05:14:24.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bd607640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:24.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bd607640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:24.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bd607640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:24.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bd607640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:24.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bd607640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:24.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.764+0000 7f71bd607640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:14:24.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.768+0000 7f71bce06640 10 monclient(hunting): handle_auth_done global_id 64330 payload 293 2024-09-16T05:14:24.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.768+0000 7f71bce06640 1 -- 172.21.15.114:0/1249733576 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f71a0004da0 msgr2=0x7f71a0005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:24.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.768+0000 7f71bce06640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f71a0004da0 0x7f71a0005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:24.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.768+0000 7f71bce06640 1 -- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a0004100 msgr2=0x7f71a0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:24.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.768+0000 7f71bce06640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71a0004100 0x7f71a0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:24.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.768+0000 7f71bce06640 10 monclient: _finish_hunting 0 2024-09-16T05:14:24.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.768+0000 7f71bce06640 1 monclient: found mon.c 2024-09-16T05:14:24.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.768+0000 7f71bce06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:24.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.768+0000 7f71bce06640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:24.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.768+0000 7f71bce06640 1 -- 172.21.15.114:0/1249733576 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a0003530 con 0x7f71b80a5d00 2024-09-16T05:14:24.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.768+0000 7f71bce06640 10 monclient: _finish_auth 0 2024-09-16T05:14:24.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.768+0000 7f71bce06640 10 monclient: _check_auth_tickets 2024-09-16T05:14:24.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.768+0000 7f71bce06640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:24.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71bce06640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71a0003b50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f71a800c2f0 tx=0x7f71a801cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41a70ec176d850ba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:24.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a8002070 con 0x7f71b80a5d00 2024-09-16T05:14:24.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:24.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:24.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71b77fe640 10 monclient: dump: 2024-09-16T05:14:24.779 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:24.779 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:24.779 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:24.779 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:24.779 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:24.779 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:24.779 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:24.779 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:24.779 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:24.779 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:24.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71a80a9070 con 0x7f71b80a5d00 2024-09-16T05:14:24.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71bf06a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:24.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71b77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:24.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71bf06a640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:24.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71bf06a640 4 set_mon_vals no callback set 2024-09-16T05:14:24.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71b77fe640 1 -- 172.21.15.114:0/1249733576 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a80a5ca0 con 0x7f71b80a5d00 2024-09-16T05:14:24.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:24.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:24.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71b77fe640 10 monclient: dump: 2024-09-16T05:14:24.780 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:24.780 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:24.780 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:24.780 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:24.780 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:24.780 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:24.780 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:24.781 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:24.781 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:24.781 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:24.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71bf06a640 10 monclient: shutdown 2024-09-16T05:14:24.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71bf06a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:24.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71bf06a640 1 -- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 msgr2=0x7f71a0003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:24.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71bf06a640 1 --2- 172.21.15.114:0/1249733576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71a0003b50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f71a800c2f0 tx=0x7f71a801cd80 comp rx=0 tx=0).stop 2024-09-16T05:14:24.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71bd607640 1 -- 172.21.15.114:0/1249733576 reap_dead start 2024-09-16T05:14:24.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71bf06a640 1 -- 172.21.15.114:0/1249733576 shutdown_connections 2024-09-16T05:14:24.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71bf06a640 1 -- 172.21.15.114:0/1249733576 >> 172.21.15.114:0/1249733576 conn(0x7f71b80a9c70 msgr2=0x7f71b80aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:24.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71bf06a640 1 -- 172.21.15.114:0/1249733576 shutdown_connections 2024-09-16T05:14:24.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.772+0000 7f71bf06a640 1 -- 172.21.15.114:0/1249733576 wait complete. 2024-09-16T05:14:24.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient: build_initial_monmap 2024-09-16T05:14:24.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient: monmap: 2024-09-16T05:14:24.782 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:24.782 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:24.782 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:24.782 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:24.782 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:24.782 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:24.782 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:24.782 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:24.782 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:24.782 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:24.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 1 librados: starting msgr at 2024-09-16T05:14:24.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 1 librados: starting objecter 2024-09-16T05:14:24.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 1 Processor -- start 2024-09-16T05:14:24.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 1 -- start start 2024-09-16T05:14:24.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 1 librados: setting wanted keys 2024-09-16T05:14:24.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 1 librados: calling monclient init 2024-09-16T05:14:24.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient: init 2024-09-16T05:14:24.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:24.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:14:24.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b81cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:24.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient(hunting): picked mon.noname-c con 0x7f71b80a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:24.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71b8136420 0x7f71b8136860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:24.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient(hunting): picked mon.noname-a con 0x7f71b8136420 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:24.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f71b80a0be0 0x7f71b80a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:24.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient(hunting): picked mon.noname-b con 0x7f71b80a0be0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:24.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:24.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f71b8140ea0 con 0x7f71b8136420 2024-09-16T05:14:24.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:24.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f71b8140d20 con 0x7f71b80a0be0 2024-09-16T05:14:24.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:24.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f71b8141020 con 0x7f71b80a5d00 2024-09-16T05:14:24.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:24.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient(hunting): authenticate will time out at 1349.410392s 2024-09-16T05:14:24.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bce06640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b81cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:24.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bce06640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b81cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51286/0 (socket says 172.21.15.114:51286) 2024-09-16T05:14:24.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bce06640 1 -- 172.21.15.114:0/1708060877 learned_addr learned my addr 172.21.15.114:0/1708060877 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:24.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bce06640 10 monclient(hunting): get_auth_request con 0x7f71b80a5d00 auth_method 0 2024-09-16T05:14:24.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bce06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:24.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bce06640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:24.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bce06640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:24.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 1 --2- 172.21.15.114:0/1708060877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71b8136420 0x7f71b8136860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:24.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 10 monclient(hunting): get_auth_request con 0x7f71b8136420 auth_method 0 2024-09-16T05:14:24.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:24.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:24.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:24.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bce06640 1 -- 172.21.15.114:0/1708060877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 msgr2=0x7f71b81cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:14:24.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bce06640 1 -- 172.21.15.114:0/1708060877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 msgr2=0x7f71b81cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:14:24.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bce06640 1 --2- 172.21.15.114:0/1708060877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b81cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:24.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bce06640 1 --2- 172.21.15.114:0/1708060877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b81cf1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:24.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:24.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:24.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:24.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bd607640 1 --2- 172.21.15.114:0/1708060877 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f71b80a0be0 0x7f71b80a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:24.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bd607640 10 monclient(hunting): get_auth_request con 0x7f71b80a0be0 auth_method 0 2024-09-16T05:14:24.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bd607640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:24.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bd607640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:24.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bd607640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:24.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 10 monclient(hunting): handle_auth_done global_id 74106 payload 824 2024-09-16T05:14:24.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 1 -- 172.21.15.114:0/1708060877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 msgr2=0x7f71b81cf1b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:14:24.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 1 --2- 172.21.15.114:0/1708060877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b81cf1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:24.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 1 -- 172.21.15.114:0/1708060877 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f71b80a0be0 msgr2=0x7f71b80a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:24.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 1 --2- 172.21.15.114:0/1708060877 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f71b80a0be0 0x7f71b80a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:24.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:14:24.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 1 monclient: found mon.noname-a 2024-09-16T05:14:24.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:24.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:24.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 1 -- 172.21.15.114:0/1708060877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b81203b0 con 0x7f71b8136420 2024-09-16T05:14:24.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b7fff640 1 --2- 172.21.15.114:0/1708060877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71b8136420 0x7f71b8136860 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f71b80a6100 tx=0x7f71ac014df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:24.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b57fa640 10 client.?.objecter ms_handle_connect 0x7f71b8136420 2024-09-16T05:14:24.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b57fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:24.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b57fa640 1 -- 172.21.15.114:0/1708060877 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac01b030 con 0x7f71b8136420 2024-09-16T05:14:24.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:24.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:24.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b57fa640 10 monclient: dump: 2024-09-16T05:14:24.792 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:24.793 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:24.793 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:24.793 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:24.793 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:24.793 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:24.793 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:24.793 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:24.793 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:24.793 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:24.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b57fa640 10 monclient: _finish_auth 0 2024-09-16T05:14:24.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b57fa640 10 monclient: _check_auth_tickets 2024-09-16T05:14:24.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:24.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b57fa640 1 -- 172.21.15.114:0/1708060877 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71ac012820 con 0x7f71b8136420 2024-09-16T05:14:24.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:24.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b57fa640 1 -- 172.21.15.114:0/1708060877 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac00f040 con 0x7f71b8136420 2024-09-16T05:14:24.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:24.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:24.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b57fa640 10 monclient: dump: 2024-09-16T05:14:24.794 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:24.794 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:24.794 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:24.794 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:24.794 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:24.794 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:24.794 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:24.794 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:24.795 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:24.795 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:24.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 5 monclient: authenticate success, global_id 74106 2024-09-16T05:14:24.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71b6ffd640 4 set_mon_vals no callback set 2024-09-16T05:14:24.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient: _renew_subs 2024-09-16T05:14:24.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:24.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 1 -- 172.21.15.114:0/1708060877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71b81cf6f0 con 0x7f71b8136420 2024-09-16T05:14:24.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 client.74106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:24.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient: _renew_subs 2024-09-16T05:14:24.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:24.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 1 -- 172.21.15.114:0/1708060877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71b80a9160 con 0x7f71b8136420 2024-09-16T05:14:24.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.776+0000 7f71bf06a640 1 librados: init done 2024-09-16T05:14:24.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.780+0000 7f71b57fa640 1 -- 172.21.15.114:0/1708060877 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f71ac01e030 con 0x7f71b8136420 2024-09-16T05:14:24.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.780+0000 7f71bf06a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.780+0000 7f71bf06a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.780+0000 7f71bf06a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.780+0000 7f71bf06a640 1 -- 172.21.15.114:0/1708060877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71b8137b10 con 0x7f71b8136420 2024-09-16T05:14:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.780+0000 7f71b57fa640 1 --2- 172.21.15.114:0/1708060877 >> v2:172.21.15.149:6812/1185531135 conn(0x7f71a4037810 0x7f71a4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.780+0000 7f71b57fa640 1 -- 172.21.15.114:0/1708060877 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 6224+0+0 (secure 0 0 0) 0x7f71ac049170 con 0x7f71b8136420 2024-09-16T05:14:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.780+0000 7f71b57fa640 10 client.74106.objecter ms_dispatch 0x7f71b8000ba0 osd_map(286..286 src has 1..286) 2024-09-16T05:14:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.780+0000 7f71b57fa640 3 client.74106.objecter handle_osd_map got epochs [286,286] > 0 2024-09-16T05:14:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.780+0000 7f71b57fa640 3 client.74106.objecter handle_osd_map decoding full epoch 286 2024-09-16T05:14:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.780+0000 7f71b57fa640 20 client.74106.objecter dump_active .. 0 homeless 2024-09-16T05:14:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.780+0000 7f71bce06640 1 --2- 172.21.15.114:0/1708060877 >> v2:172.21.15.149:6812/1185531135 conn(0x7f71a4037810 0x7f71a4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.780+0000 7f71bce06640 10 monclient: get_auth_request con 0x7f71a4037810 auth_method 0 2024-09-16T05:14:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.780+0000 7f71b57fa640 10 client.74106.objecter ms_handle_connect 0x7f71a4037810 2024-09-16T05:14:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.780+0000 7f71bce06640 1 --2- 172.21.15.114:0/1708060877 >> v2:172.21.15.149:6812/1185531135 conn(0x7f71a4037810 0x7f71a4039cd0 secure :-1 s=READY pgs=3494 cs=0 l=1 rev1=1 crypto rx=0x7f71a800c190 tx=0x7f71a80a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:24.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.784+0000 7f71b57fa640 1 -- 172.21.15.114:0/1708060877 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f71ac01fc50 con 0x7f71b8136420 2024-09-16T05:14:24.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.784+0000 7f71b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:24.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:24.784+0000 7f71b57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:25.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.012+0000 7f71bf06a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:14:25.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.012+0000 7f71bf06a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:25.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.012+0000 7f71bf06a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:25.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.012+0000 7f71bf06a640 1 -- 172.21.15.114:0/1708060877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f71b8068650 con 0x7f71b8136420 2024-09-16T05:14:25.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.012+0000 7f71b57fa640 1 -- 172.21.15.114:0/1708060877 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f71ac012bf0 con 0x7f71b8136420 2024-09-16T05:14:25.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.012+0000 7f71b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:25.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.012+0000 7f71b57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:25.017 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:14:25.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 10 librados: watch_flush enter 2024-09-16T05:14:25.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 10 librados: watch_flush exit 2024-09-16T05:14:25.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 20 client.74106.objecter shutdown clearing up homeless session... 2024-09-16T05:14:25.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 10 client.74106.objecter successfully canceled tick 2024-09-16T05:14:25.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 1 -- 172.21.15.114:0/1708060877 >> v2:172.21.15.149:6812/1185531135 conn(0x7f71a4037810 msgr2=0x7f71a4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:25.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 1 --2- 172.21.15.114:0/1708060877 >> v2:172.21.15.149:6812/1185531135 conn(0x7f71a4037810 0x7f71a4039cd0 secure :-1 s=READY pgs=3494 cs=0 l=1 rev1=1 crypto rx=0x7f71a800c190 tx=0x7f71a80a7070 comp rx=0 tx=0).stop 2024-09-16T05:14:25.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 10 monclient: shutdown 2024-09-16T05:14:25.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:25.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 1 -- 172.21.15.114:0/1708060877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71b8136420 msgr2=0x7f71b8136860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:25.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 1 --2- 172.21.15.114:0/1708060877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71b8136420 0x7f71b8136860 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f71b80a6100 tx=0x7f71ac014df0 comp rx=0 tx=0).stop 2024-09-16T05:14:25.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 1 -- 172.21.15.114:0/1708060877 shutdown_connections 2024-09-16T05:14:25.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 1 --2- 172.21.15.114:0/1708060877 >> v2:172.21.15.149:6812/1185531135 conn(0x7f71a4037810 0x7f71a4039cd0 unknown :-1 s=CLOSED pgs=3494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:25.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 1 --2- 172.21.15.114:0/1708060877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71b8136420 0x7f71b8136860 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:25.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 1 --2- 172.21.15.114:0/1708060877 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f71b80a0be0 0x7f71b80a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:25.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 1 --2- 172.21.15.114:0/1708060877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71b80a5d00 0x7f71b81cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:25.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 1 -- 172.21.15.114:0/1708060877 >> 172.21.15.114:0/1708060877 conn(0x7f71b80a9c70 msgr2=0x7f71b8139130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:25.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 1 -- 172.21.15.114:0/1708060877 shutdown_connections 2024-09-16T05:14:25.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 1 -- 172.21.15.114:0/1708060877 wait complete. 2024-09-16T05:14:25.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:25.016+0000 7f71bf06a640 1 librados: shutdown 2024-09-16T05:14:28.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:14:28.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 10 monclient: get_monmap_and_config 2024-09-16T05:14:28.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 10 monclient: build_initial_monmap 2024-09-16T05:14:28.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 10 monclient: monmap: 2024-09-16T05:14:28.168 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:28.168 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.168 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:28.169612+0000 2024-09-16T05:14:28.168 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:28.169612+0000 2024-09-16T05:14:28.168 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:28.168 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.168 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:28.168 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:28.168 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:28.168 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:28.168 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:28.168 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 1 Processor -- start 2024-09-16T05:14:28.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 1 -- start start 2024-09-16T05:14:28.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 10 monclient: init 2024-09-16T05:14:28.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:28.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T05:14:28.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9cc0a0ec0 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:28.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff9cc09e1d0 0x7ff9cc09e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 10 monclient(hunting): picked mon.noname-c con 0x7ff9cc09e1d0 addr v2:172.21.15.149:3300/0 2024-09-16T05:14:28.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9cc139640 addr v1:172.21.15.149:6789/0 2024-09-16T05:14:28.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9cc09a760 con 0x7ff9cc0a0ec0 2024-09-16T05:14:28.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9cc0a1870 con 0x7ff9cc139640 2024-09-16T05:14:28.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7ff9cc0994b0 con 0x7ff9cc09e1d0 2024-09-16T05:14:28.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:28.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 10 monclient(hunting): authenticate will time out at 1352.799039s 2024-09-16T05:14:28.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9caffd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff9cc0a0ec0 0x7ff9cc09dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:47710/0 (socket says 172.21.15.114:47710) 2024-09-16T05:14:28.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9caffd640 1 -- 172.21.15.114:0/2715322892 learned_addr learned my addr 172.21.15.114:0/2715322892 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:14:28.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 -- 172.21.15.114:0/2715322892 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 724146180 0 0) 0x7ff9cc09a760 con 0x7ff9cc0a0ec0 2024-09-16T05:14:28.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:14:28.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient(hunting): my global_id is 74109 2024-09-16T05:14:28.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9ca7fc640 1 --2- 172.21.15.114:0/2715322892 >> v2:172.21.15.149:3300/0 conn(0x7ff9cc09e1d0 0x7ff9cc09e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 -- 172.21.15.114:0/2715322892 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9b4003530 con 0x7ff9cc0a0ec0 2024-09-16T05:14:28.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9ca7fc640 10 monclient(hunting): get_auth_request con 0x7ff9cc09e1d0 auth_method 0 2024-09-16T05:14:28.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 -- 172.21.15.114:0/2715322892 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2019178263 0 0) 0x7ff9b4003530 con 0x7ff9cc0a0ec0 2024-09-16T05:14:28.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 -- 172.21.15.114:0/2715322892 >> v2:172.21.15.149:3300/0 conn(0x7ff9cc09e1d0 msgr2=0x7ff9cc09e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:28.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 --2- 172.21.15.114:0/2715322892 >> v2:172.21.15.149:3300/0 conn(0x7ff9cc09e1d0 0x7ff9cc09e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 -- 172.21.15.114:0/2715322892 >> v1:172.21.15.149:6789/0 conn(0x7ff9cc139640 legacy=0x7ff9cc13ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:14:28.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:14:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:14:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 -- 172.21.15.114:0/2715322892 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9cc099790 con 0x7ff9cc0a0ec0 2024-09-16T05:14:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:14:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:14:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 5 monclient: authenticate success, global_id 74109 2024-09-16T05:14:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 -- 172.21.15.114:0/2715322892 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7ff9bc0016a0 con 0x7ff9cc0a0ec0 2024-09-16T05:14:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:14:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient: dump: 2024-09-16T05:14:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.176 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.176 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.176 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.176 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.176 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.176 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.176 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.176 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:14:28.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:28.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 -- 172.21.15.114:0/2715322892 >> v1:172.21.15.114:6789/0 conn(0x7ff9cc0a0ec0 legacy=0x7ff9cc09dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:14:28.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 --2- 172.21.15.114:0/2715322892 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9b40039c0 0x7ff9b4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient(hunting): picked mon.b con 0x7ff9b40039c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:28.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 --2- 172.21.15.114:0/2715322892 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9b4004810 0x7ff9b4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient(hunting): picked mon.c con 0x7ff9b4004810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:28.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 --2- 172.21.15.114:0/2715322892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9b40052f0 0x7ff9b400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient(hunting): picked mon.a con 0x7ff9b40052f0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:28.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 -- 172.21.15.114:0/2715322892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc0994b0 con 0x7ff9b40052f0 2024-09-16T05:14:28.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 -- 172.21.15.114:0/2715322892 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b400c1c0 con 0x7ff9b40039c0 2024-09-16T05:14:28.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 1 -- 172.21.15.114:0/2715322892 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff9b400c3c0 con 0x7ff9b4004810 2024-09-16T05:14:28.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:28.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9c9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:14:28.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.164+0000 7ff9d12bf640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:28.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9cb7fe640 1 --2- 172.21.15.114:0/2715322892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9b40052f0 0x7ff9b400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 1 --2- 172.21.15.114:0/2715322892 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9b4004810 0x7ff9b4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9cb7fe640 10 monclient(hunting): get_auth_request con 0x7ff9b40052f0 auth_method 0 2024-09-16T05:14:28.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9cb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:28.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 10 monclient(hunting): get_auth_request con 0x7ff9b4004810 auth_method 0 2024-09-16T05:14:28.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:28.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9caffd640 1 --2- 172.21.15.114:0/2715322892 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9b40039c0 0x7ff9b4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:14:28.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:14:28.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9caffd640 10 monclient(hunting): get_auth_request con 0x7ff9b40039c0 auth_method 0 2024-09-16T05:14:28.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9caffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:28.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 10 monclient(hunting): handle_auth_done global_id 74109 payload 293 2024-09-16T05:14:28.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 1 -- 172.21.15.114:0/2715322892 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9b40039c0 msgr2=0x7ff9b4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:28.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 1 --2- 172.21.15.114:0/2715322892 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9b40039c0 0x7ff9b4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 1 -- 172.21.15.114:0/2715322892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9b40052f0 msgr2=0x7ff9b400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:28.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 1 --2- 172.21.15.114:0/2715322892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9b40052f0 0x7ff9b400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:14:28.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 1 monclient: found mon.c 2024-09-16T05:14:28.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:28.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:28.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 1 -- 172.21.15.114:0/2715322892 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b4003530 con 0x7ff9b4004810 2024-09-16T05:14:28.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 10 monclient: _finish_auth 0 2024-09-16T05:14:28.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:14:28.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:28.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9cb7fe640 1 --2- 172.21.15.114:0/2715322892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9b40052f0 0x7ff9b400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:14:28.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9cb7fe640 1 -- 172.21.15.114:0/2715322892 reap_dead start 2024-09-16T05:14:28.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9ca7fc640 1 --2- 172.21.15.114:0/2715322892 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9b4004810 0x7ff9b4004c90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000ba00 tx=0x7ff9c001c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74c56442c35f0464 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9c9ffb640 1 -- 172.21.15.114:0/2715322892 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c00b2070 con 0x7ff9b4004810 2024-09-16T05:14:28.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9c9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:28.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9c9ffb640 10 monclient: dump: 2024-09-16T05:14:28.187 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.187 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.187 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.187 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.187 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.187 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.187 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.187 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.187 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.187 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9d12bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:28.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9c9ffb640 1 -- 172.21.15.114:0/2715322892 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c00ad640 con 0x7ff9b4004810 2024-09-16T05:14:28.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:28.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9d12bf640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:28.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9d12bf640 4 set_mon_vals no callback set 2024-09-16T05:14:28.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9c9ffb640 1 -- 172.21.15.114:0/2715322892 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c000b600 con 0x7ff9b4004810 2024-09-16T05:14:28.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9c9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:28.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9c9ffb640 10 monclient: dump: 2024-09-16T05:14:28.188 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.188 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.188 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.189 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.189 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.189 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.189 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.189 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.189 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.189 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9d12bf640 10 monclient: shutdown 2024-09-16T05:14:28.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9d12bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:28.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9d12bf640 1 -- 172.21.15.114:0/2715322892 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9b4004810 msgr2=0x7ff9b4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9d12bf640 1 --2- 172.21.15.114:0/2715322892 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9b4004810 0x7ff9b4004c90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000ba00 tx=0x7ff9c001c640 comp rx=0 tx=0).stop 2024-09-16T05:14:28.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9d12bf640 1 -- 172.21.15.114:0/2715322892 shutdown_connections 2024-09-16T05:14:28.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9d12bf640 1 --2- 172.21.15.114:0/2715322892 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9b4004810 0x7ff9b4004c90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9d12bf640 1 -- 172.21.15.114:0/2715322892 >> 172.21.15.114:0/2715322892 conn(0x7ff9cc0a1fb0 msgr2=0x7ff9cc0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:28.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9d12bf640 1 -- 172.21.15.114:0/2715322892 shutdown_connections 2024-09-16T05:14:28.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9d12bf640 1 -- 172.21.15.114:0/2715322892 wait complete. 2024-09-16T05:14:28.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9d12bf640 10 monclient: build_initial_monmap 2024-09-16T05:14:28.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.168+0000 7ff9d12bf640 10 monclient: monmap: 2024-09-16T05:14:28.190 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:28.190 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:28.190 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:28.190 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:28.190 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:28.190 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.190 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:28.190 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:28.190 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:28.190 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 librados: starting msgr at 2024-09-16T05:14:28.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 librados: starting objecter 2024-09-16T05:14:28.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 Processor -- start 2024-09-16T05:14:28.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 -- start start 2024-09-16T05:14:28.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 librados: setting wanted keys 2024-09-16T05:14:28.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 librados: calling monclient init 2024-09-16T05:14:28.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient: init 2024-09-16T05:14:28.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:28.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:14:28.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9b4004810 0x7ff9cc098b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient(hunting): picked mon.noname-b con 0x7ff9b4004810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:28.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9cc08e630 0x7ff9cc08ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient(hunting): picked mon.noname-a con 0x7ff9cc08e630 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:28.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9cc08efb0 0x7ff9cc1ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient(hunting): picked mon.noname-c con 0x7ff9cc08efb0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:28.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc0a1df0 con 0x7ff9cc08e630 2024-09-16T05:14:28.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc0a1c70 con 0x7ff9b4004810 2024-09-16T05:14:28.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff9cc09c230 con 0x7ff9cc08efb0 2024-09-16T05:14:28.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:28.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient(hunting): authenticate will time out at 1352.804819s 2024-09-16T05:14:28.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9cb7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9cc08efb0 0x7ff9cc1ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9cb7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9cc08efb0 0x7ff9cc1ce800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:48582/0 (socket says 172.21.15.114:48582) 2024-09-16T05:14:28.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9cb7fe640 1 -- 172.21.15.114:0/776424574 learned_addr learned my addr 172.21.15.114:0/776424574 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:28.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9cb7fe640 10 monclient(hunting): get_auth_request con 0x7ff9cc08efb0 auth_method 0 2024-09-16T05:14:28.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 1 --2- 172.21.15.114:0/776424574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9cc08e630 0x7ff9cc08ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 10 monclient(hunting): get_auth_request con 0x7ff9cc08e630 auth_method 0 2024-09-16T05:14:28.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9caffd640 1 --2- 172.21.15.114:0/776424574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9b4004810 0x7ff9cc098b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9caffd640 10 monclient(hunting): get_auth_request con 0x7ff9b4004810 auth_method 0 2024-09-16T05:14:28.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 10 monclient(hunting): handle_auth_done global_id 74112 payload 824 2024-09-16T05:14:28.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 1 -- 172.21.15.114:0/776424574 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9cc08efb0 msgr2=0x7ff9cc1ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 1 --2- 172.21.15.114:0/776424574 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9cc08efb0 0x7ff9cc1ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 1 -- 172.21.15.114:0/776424574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9b4004810 msgr2=0x7ff9cc098b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 1 --2- 172.21.15.114:0/776424574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9b4004810 0x7ff9cc098b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:14:28.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 1 monclient: found mon.noname-a 2024-09-16T05:14:28.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 1 -- 172.21.15.114:0/776424574 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9cc0a34b0 con 0x7ff9cc08e630 2024-09-16T05:14:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9cb7fe640 1 --2- 172.21.15.114:0/776424574 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9cc08efb0 0x7ff9cc1ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:14:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9caffd640 1 --2- 172.21.15.114:0/776424574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9b4004810 0x7ff9cc098b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:14:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9ca7fc640 1 --2- 172.21.15.114:0/776424574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9cc08e630 0x7ff9cc08ea70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400e7a0 tx=0x7ff9b4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 10 client.?.objecter ms_handle_connect 0x7ff9cc08e630 2024-09-16T05:14:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 1 -- 172.21.15.114:0/776424574 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c00b2040 con 0x7ff9cc08e630 2024-09-16T05:14:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 10 monclient: dump: 2024-09-16T05:14:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.196 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.196 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.196 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.196 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.196 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.196 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.196 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 10 monclient: _finish_auth 0 2024-09-16T05:14:28.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:14:28.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:28.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 1 -- 172.21.15.114:0/776424574 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c0004230 con 0x7ff9cc08e630 2024-09-16T05:14:28.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 5 monclient: authenticate success, global_id 74112 2024-09-16T05:14:28.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient: _renew_subs 2024-09-16T05:14:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 -- 172.21.15.114:0/776424574 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9cc091420 con 0x7ff9cc08e630 2024-09-16T05:14:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 1 -- 172.21.15.114:0/776424574 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c00a7070 con 0x7ff9cc08e630 2024-09-16T05:14:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 10 monclient: dump: 2024-09-16T05:14:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.198 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9c97fa640 4 set_mon_vals no callback set 2024-09-16T05:14:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 client.74112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient: _renew_subs 2024-09-16T05:14:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 -- 172.21.15.114:0/776424574 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9cc092260 con 0x7ff9cc08e630 2024-09-16T05:14:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 librados: init done 2024-09-16T05:14:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 1 -- 172.21.15.114:0/776424574 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff9c00ad4c0 con 0x7ff9cc08e630 2024-09-16T05:14:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:28.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9d12bf640 1 -- 172.21.15.114:0/776424574 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9cc0a5140 con 0x7ff9cc08e630 2024-09-16T05:14:28.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 1 --2- 172.21.15.114:0/776424574 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff9b8037810 0x7ff9b8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 1 -- 172.21.15.114:0/776424574 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(288..288 src has 1..288) ==== 5832+0+0 (secure 0 0 0) 0x7ff9c00e3170 con 0x7ff9cc08e630 2024-09-16T05:14:28.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 10 client.74112.objecter ms_dispatch 0x7ff9cc000ba0 osd_map(288..288 src has 1..288) 2024-09-16T05:14:28.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 3 client.74112.objecter handle_osd_map got epochs [288,288] > 0 2024-09-16T05:14:28.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 3 client.74112.objecter handle_osd_map decoding full epoch 288 2024-09-16T05:14:28.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9b37fe640 20 client.74112.objecter dump_active .. 0 homeless 2024-09-16T05:14:28.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9caffd640 1 --2- 172.21.15.114:0/776424574 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff9b8037810 0x7ff9b8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.172+0000 7ff9caffd640 10 monclient: get_auth_request con 0x7ff9b8037810 auth_method 0 2024-09-16T05:14:28.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.176+0000 7ff9b37fe640 10 client.74112.objecter ms_handle_connect 0x7ff9b8037810 2024-09-16T05:14:28.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.176+0000 7ff9caffd640 1 --2- 172.21.15.114:0/776424574 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff9b8037810 0x7ff9b8039cd0 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc09a220 tx=0x7ff9bc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.176+0000 7ff9b37fe640 1 -- 172.21.15.114:0/776424574 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ff9cc0a34b0 con 0x7ff9cc08e630 2024-09-16T05:14:28.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.176+0000 7ff9b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:28.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.176+0000 7ff9b37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:28.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.404+0000 7ff9d12bf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:14:28.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.404+0000 7ff9d12bf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:28.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.404+0000 7ff9d12bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:28.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.404+0000 7ff9d12bf640 1 -- 172.21.15.114:0/776424574 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff9cc09e670 con 0x7ff9cc08e630 2024-09-16T05:14:28.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.404+0000 7ff9b37fe640 1 -- 172.21.15.114:0/776424574 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7ff9c00ba020 con 0x7ff9cc08e630 2024-09-16T05:14:28.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.404+0000 7ff9b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:28.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.404+0000 7ff9b37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:28.409 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:14:28.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 10 librados: watch_flush enter 2024-09-16T05:14:28.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 10 librados: watch_flush exit 2024-09-16T05:14:28.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 20 client.74112.objecter shutdown clearing up homeless session... 2024-09-16T05:14:28.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 10 client.74112.objecter successfully canceled tick 2024-09-16T05:14:28.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 1 -- 172.21.15.114:0/776424574 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff9b8037810 msgr2=0x7ff9b8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 1 --2- 172.21.15.114:0/776424574 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff9b8037810 0x7ff9b8039cd0 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc09a220 tx=0x7ff9bc009040 comp rx=0 tx=0).stop 2024-09-16T05:14:28.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 10 monclient: shutdown 2024-09-16T05:14:28.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:28.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 1 -- 172.21.15.114:0/776424574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9cc08e630 msgr2=0x7ff9cc08ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 1 --2- 172.21.15.114:0/776424574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9cc08e630 0x7ff9cc08ea70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400e7a0 tx=0x7ff9b4005840 comp rx=0 tx=0).stop 2024-09-16T05:14:28.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 1 -- 172.21.15.114:0/776424574 shutdown_connections 2024-09-16T05:14:28.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 1 --2- 172.21.15.114:0/776424574 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff9b8037810 0x7ff9b8039cd0 unknown :-1 s=CLOSED pgs=3495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 1 --2- 172.21.15.114:0/776424574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9cc08e630 0x7ff9cc08ea70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 1 --2- 172.21.15.114:0/776424574 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff9cc08efb0 0x7ff9cc1ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 1 --2- 172.21.15.114:0/776424574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9b4004810 0x7ff9cc098b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 1 -- 172.21.15.114:0/776424574 >> 172.21.15.114:0/776424574 conn(0x7ff9cc0a1fb0 msgr2=0x7ff9cc138a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:28.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 1 -- 172.21.15.114:0/776424574 shutdown_connections 2024-09-16T05:14:28.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 1 -- 172.21.15.114:0/776424574 wait complete. 2024-09-16T05:14:28.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.408+0000 7ff9d12bf640 1 librados: shutdown 2024-09-16T05:14:28.428 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:14:28.428 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:14:28.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:14:28.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 10 monclient: get_monmap_and_config 2024-09-16T05:14:28.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 10 monclient: build_initial_monmap 2024-09-16T05:14:28.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 10 monclient: monmap: 2024-09-16T05:14:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:28.554554+0000 2024-09-16T05:14:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:28.554554+0000 2024-09-16T05:14:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:28.553 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 1 Processor -- start 2024-09-16T05:14:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 1 -- start start 2024-09-16T05:14:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 10 monclient: init 2024-09-16T05:14:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-16T05:14:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f68f8137730 0x7f68f8137b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 10 monclient(hunting): picked mon.noname-a con 0x7f68f8137730 addr v2:172.21.15.114:3300/0 2024-09-16T05:14:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68f80a5d00 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68f80a4980 0x7f68f813f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 10 monclient(hunting): picked mon.noname-b con 0x7f68f80a4980 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f80a55e0 con 0x7f68f80a5d00 2024-09-16T05:14:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f68f809c450 con 0x7f68f8137730 2024-09-16T05:14:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f68f80a3710 con 0x7f68f80a4980 2024-09-16T05:14:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f6900639640 10 monclient(hunting): authenticate will time out at 1353.183758s 2024-09-16T05:14:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f68fe3d5640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f68f8137730 0x7f68f8137b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.548+0000 7f68febd6640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68f80a4980 0x7f68f813f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68f80a4980 0x7f68f813f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:48594/0 (socket says 172.21.15.114:48594) 2024-09-16T05:14:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fdbd4640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f68f80a5d00 0x7f68f80a4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:47720/0 (socket says 172.21.15.114:47720) 2024-09-16T05:14:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fe3d5640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f68f8137730 0x7f68f8137b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36668/0 (socket says 172.21.15.114:36668) 2024-09-16T05:14:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 1 -- 172.21.15.114:0/2046357677 learned_addr learned my addr 172.21.15.114:0/2046357677 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 10 monclient(hunting): get_auth_request con 0x7f68f80a4980 auth_method 0 2024-09-16T05:14:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fe3d5640 10 monclient(hunting): get_auth_request con 0x7f68f8137730 auth_method 0 2024-09-16T05:14:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fe3d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fe3d5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fe3d5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fe3d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fe3d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fe3d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 10 monclient(hunting): handle_auth_done global_id 54620 payload 274 2024-09-16T05:14:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 1 -- 172.21.15.114:0/2046357677 >> v2:172.21.15.114:3300/0 conn(0x7f68f8137730 msgr2=0x7f68f8137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 1 --2- 172.21.15.114:0/2046357677 >> v2:172.21.15.114:3300/0 conn(0x7f68f8137730 0x7f68f8137b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 1 -- 172.21.15.114:0/2046357677 >> v1:172.21.15.114:6789/0 conn(0x7f68f80a5d00 legacy=0x7f68f80a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 10 monclient: _finish_hunting 0 2024-09-16T05:14:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 1 monclient: found mon.noname-b 2024-09-16T05:14:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:28.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:14:28.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 1 -- 172.21.15.114:0/2046357677 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f809be00 con 0x7f68f80a4980 2024-09-16T05:14:28.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fe3d5640 1 --2- 172.21.15.114:0/2046357677 >> v2:172.21.15.114:3300/0 conn(0x7f68f8137730 0x7f68f8137b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:14:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68febd6640 1 --2- 172.21.15.114:0/2046357677 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68f80a4980 0x7f68f813f4e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f68f0009e50 tx=0x7f68f001c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfa8d3498aef674d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fd3d3640 1 -- 172.21.15.114:0/2046357677 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f00a5640 con 0x7f68f80a4980 2024-09-16T05:14:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fd3d3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fd3d3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:14:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fd3d3640 10 monclient: dump: 2024-09-16T05:14:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.571 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fd3d3640 10 monclient: _finish_auth 0 2024-09-16T05:14:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fd3d3640 10 monclient: _check_auth_tickets 2024-09-16T05:14:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fd3d3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fd3d3640 1 -- 172.21.15.114:0/2046357677 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68f00a57e0 con 0x7f68f80a4980 2024-09-16T05:14:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 5 monclient: authenticate success, global_id 54620 2024-09-16T05:14:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fd3d3640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fd3d3640 1 -- 172.21.15.114:0/2046357677 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f00a5bb0 con 0x7f68f80a4980 2024-09-16T05:14:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fd3d3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 4 set_mon_vals no callback set 2024-09-16T05:14:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fd3d3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f68fd3d3640 10 monclient: dump: 2024-09-16T05:14:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.573 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.573 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.573 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.573 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.573 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.573 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.573 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 10 monclient: shutdown 2024-09-16T05:14:28.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:28.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 1 -- 172.21.15.114:0/2046357677 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68f80a4980 msgr2=0x7f68f813f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 1 --2- 172.21.15.114:0/2046357677 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68f80a4980 0x7f68f813f4e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f68f0009e50 tx=0x7f68f001c8f0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 1 -- 172.21.15.114:0/2046357677 shutdown_connections 2024-09-16T05:14:28.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 1 --2- 172.21.15.114:0/2046357677 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68f80a4980 0x7f68f813f4e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 1 --2- 172.21.15.114:0/2046357677 >> v2:172.21.15.114:3300/0 conn(0x7f68f8137730 0x7f68f8137b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 1 -- 172.21.15.114:0/2046357677 >> 172.21.15.114:0/2046357677 conn(0x7f68f80a9c70 msgr2=0x7f68f80aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:28.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 1 -- 172.21.15.114:0/2046357677 shutdown_connections 2024-09-16T05:14:28.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 1 -- 172.21.15.114:0/2046357677 wait complete. 2024-09-16T05:14:28.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 10 monclient: build_initial_monmap 2024-09-16T05:14:28.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.552+0000 7f6900639640 10 monclient: monmap: 2024-09-16T05:14:28.574 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:28.574 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:28.574 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:28.574 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:28.574 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:28.574 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.574 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:28.574 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:28.574 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:28.575 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 librados: starting msgr at 2024-09-16T05:14:28.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 librados: starting objecter 2024-09-16T05:14:28.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 Processor -- start 2024-09-16T05:14:28.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 -- start start 2024-09-16T05:14:28.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 librados: setting wanted keys 2024-09-16T05:14:28.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 librados: calling monclient init 2024-09-16T05:14:28.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient: init 2024-09-16T05:14:28.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:28.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:14:28.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f80a4980 0x7f68f81cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient(hunting): picked mon.noname-a con 0x7f68f80a4980 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:28.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f68f80a5d00 0x7f68f8136360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient(hunting): picked mon.noname-b con 0x7f68f80a5d00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:28.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68f8137730 0x7f68f813c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient(hunting): picked mon.noname-c con 0x7f68f8137730 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:28.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f68f8137320 con 0x7f68f80a4980 2024-09-16T05:14:28.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f68f81371a0 con 0x7f68f80a5d00 2024-09-16T05:14:28.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f68f8142240 con 0x7f68f8137730 2024-09-16T05:14:28.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:28.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient(hunting): authenticate will time out at 1353.188852s 2024-09-16T05:14:28.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f80a4980 0x7f68f81cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68f8137730 0x7f68f813c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f80a4980 0x7f68f81cf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36682/0 (socket says 172.21.15.114:36682) 2024-09-16T05:14:28.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 10 monclient(hunting): get_auth_request con 0x7f68f8137730 auth_method 0 2024-09-16T05:14:28.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 1 -- 172.21.15.114:0/3412456261 learned_addr learned my addr 172.21.15.114:0/3412456261 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:28.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 10 monclient(hunting): get_auth_request con 0x7f68f80a4980 auth_method 0 2024-09-16T05:14:28.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 10 monclient(hunting): handle_auth_done global_id 54623 payload 824 2024-09-16T05:14:28.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 1 -- 172.21.15.114:0/3412456261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f68f80a5d00 msgr2=0x7f68f8136360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fdbd4640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f68f80a5d00 0x7f68f8136360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f68f80a5d00 0x7f68f8136360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 1 -- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f80a4980 msgr2=0x7f68f81cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f80a4980 0x7f68f81cf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 10 monclient: _finish_hunting 0 2024-09-16T05:14:28.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 1 monclient: found mon.noname-c 2024-09-16T05:14:28.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:28.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:14:28.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 1 -- 172.21.15.114:0/3412456261 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f8120a30 con 0x7f68f8137730 2024-09-16T05:14:28.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f80a4980 0x7f68f81cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:14:28.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68febd6640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68f8137730 0x7f68f813c510 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f68f80a3890 tx=0x7f68f0019690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 client.?.objecter ms_handle_connect 0x7f68f8137730 2024-09-16T05:14:28.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:28.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 1 -- 172.21.15.114:0/3412456261 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f00bc020 con 0x7f68f8137730 2024-09-16T05:14:28.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:14:28.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 monclient: dump: 2024-09-16T05:14:28.579 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.579 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.579 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.579 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.579 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.579 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.580 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.580 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.580 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.580 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 monclient: _finish_auth 0 2024-09-16T05:14:28.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 monclient: _check_auth_tickets 2024-09-16T05:14:28.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:28.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 1 -- 172.21.15.114:0/3412456261 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68f00b8670 con 0x7f68f8137730 2024-09-16T05:14:28.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 5 monclient: authenticate success, global_id 54623 2024-09-16T05:14:28.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient: _renew_subs 2024-09-16T05:14:28.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:28.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 -- 172.21.15.114:0/3412456261 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f813f500 con 0x7f68f8137730 2024-09-16T05:14:28.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:28.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 1 -- 172.21.15.114:0/3412456261 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f0009510 con 0x7f68f8137730 2024-09-16T05:14:28.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:28.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 monclient: dump: 2024-09-16T05:14:28.581 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.581 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.581 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.581 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.581 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.581 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.581 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.581 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.581 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.581 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 client.54623.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:28.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient: _renew_subs 2024-09-16T05:14:28.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:28.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fcbd2640 4 set_mon_vals no callback set 2024-09-16T05:14:28.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 -- 172.21.15.114:0/3412456261 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f8073840 con 0x7f68f8137730 2024-09-16T05:14:28.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 librados: init done 2024-09-16T05:14:28.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:28.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f802c650 0x7f68f802ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f6900639640 1 -- 172.21.15.114:0/3412456261 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f68f8068ef0 con 0x7f68f802c650 2024-09-16T05:14:28.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f802c650 0x7f68f802ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 10 monclient: get_auth_request con 0x7f68f802c650 auth_method 0 2024-09-16T05:14:28.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 1 -- 172.21.15.114:0/3412456261 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f68f00bf020 con 0x7f68f8137730 2024-09-16T05:14:28.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 1 --2- 172.21.15.114:0/3412456261 >> v2:172.21.15.149:6812/1185531135 conn(0x7f68dc037810 0x7f68dc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 10 monclient(hunting): handle_auth_done global_id 74124 payload 824 2024-09-16T05:14:28.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 1 -- 172.21.15.114:0/3412456261 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f68f8137b10 con 0x7f68f802c650 2024-09-16T05:14:28.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 1 -- 172.21.15.114:0/3412456261 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(289..289 src has 1..289) ==== 5505+0+0 (secure 0 0 0) 0x7f68f00e3c20 con 0x7f68f8137730 2024-09-16T05:14:28.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 client.54623.objecter ms_dispatch 0x7f68f8000ba0 osd_map(289..289 src has 1..289) 2024-09-16T05:14:28.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 3 client.54623.objecter handle_osd_map got epochs [289,289] > 0 2024-09-16T05:14:28.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 3 client.54623.objecter handle_osd_map decoding full epoch 289 2024-09-16T05:14:28.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 20 client.54623.objecter dump_active .. 0 homeless 2024-09-16T05:14:28.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 client.54623.objecter ms_handle_connect 0x7f68f802c650 2024-09-16T05:14:28.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 client.54623.objecter resend_mon_ops 2024-09-16T05:14:28.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fe3d5640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f802c650 0x7f68f802ca30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f68f4009ef0 tx=0x7f68f400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fdbd4640 1 --2- 172.21.15.114:0/3412456261 >> v2:172.21.15.149:6812/1185531135 conn(0x7f68dc037810 0x7f68dc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fdbd4640 10 monclient: get_auth_request con 0x7f68dc037810 auth_method 0 2024-09-16T05:14:28.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 1 -- 172.21.15.114:0/3412456261 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f400fe50 con 0x7f68f802c650 2024-09-16T05:14:28.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:28.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 monclient: dump: 2024-09-16T05:14:28.584 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.585 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.585 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.585 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.585 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.585 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.585 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.585 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.585 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.585 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 1 -- 172.21.15.114:0/3412456261 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f68f400fe50 con 0x7f68f802c650 2024-09-16T05:14:28.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:28.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:28.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 1 -- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f802c650 msgr2=0x7f68f802ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f802c650 0x7f68f802ca30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f68f4009ef0 tx=0x7f68f400f610 comp rx=0 tx=0).stop 2024-09-16T05:14:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 1 -- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f802c650 msgr2=0x7f68f802ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f802c650 0x7f68f802ca30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68eeffd640 10 client.54623.objecter ms_handle_connect 0x7f68dc037810 2024-09-16T05:14:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.556+0000 7f68fdbd4640 1 --2- 172.21.15.114:0/3412456261 >> v2:172.21.15.149:6812/1185531135 conn(0x7f68dc037810 0x7f68dc039cd0 secure :-1 s=READY pgs=3496 cs=0 l=1 rev1=1 crypto rx=0x7f68f8120c40 tx=0x7f68e8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:14:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f802cf70 0x7f68f802d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 1 -- 172.21.15.114:0/3412456261 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f68f80a2050 con 0x7f68f802cf70 2024-09-16T05:14:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68fe3d5640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f802cf70 0x7f68f802d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68fe3d5640 10 monclient: get_auth_request con 0x7f68f802cf70 auth_method 0 2024-09-16T05:14:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68fe3d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68fe3d5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68fe3d5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68fe3d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68fe3d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68fe3d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68fe3d5640 10 monclient(hunting): handle_auth_done global_id 74127 payload 824 2024-09-16T05:14:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68fe3d5640 1 -- 172.21.15.114:0/3412456261 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f68f813e930 con 0x7f68f802cf70 2024-09-16T05:14:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68fe3d5640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f802cf70 0x7f68f802d380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f68f401c000 tx=0x7f68f40046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68eeffd640 10 client.54623.objecter ms_handle_connect 0x7f68f802cf70 2024-09-16T05:14:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68eeffd640 10 client.54623.objecter resend_mon_ops 2024-09-16T05:14:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68eeffd640 1 -- 172.21.15.114:0/3412456261 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f401d900 con 0x7f68f802cf70 2024-09-16T05:14:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68eeffd640 10 monclient: dump: 2024-09-16T05:14:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.588 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68eeffd640 1 -- 172.21.15.114:0/3412456261 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7f68f402a3e0 con 0x7f68f802cf70 2024-09-16T05:14:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68eeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:14:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68eeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68eeffd640 1 -- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f802cf70 msgr2=0x7f68f802d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68eeffd640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f802cf70 0x7f68f802d380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f68f401c000 tx=0x7f68f40046d0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68eeffd640 1 -- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f802cf70 msgr2=0x7f68f802d380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68eeffd640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68f802cf70 0x7f68f802d380 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f68f401c000 tx=0x7f68f40046d0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.589 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:14:28.589 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:14:28.590 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:14:28.590 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:14:28.590 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 34, 2024-09-16T05:14:28.590 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 27619, 2024-09-16T05:14:28.590 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:14:28.590 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:14:28.590 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:14:28.590 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:14:28.590 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.590 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:14:28.591 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:28.591 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:14:28.591 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:14:28.591 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:28.591 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:28.591 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:28.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:28.591 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:28.591 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:28.591 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:28.591 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:28.592 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:28.592 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:28.592 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:28.592 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.592 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:14:28.592 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:14:28.592 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:28.592 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:28.592 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:28.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:28.592 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:28.593 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:28.593 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:28.593 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:28.593 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:28.593 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:28.593 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:28.593 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.593 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.593 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:14:28.593 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:14:28.594 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:14:28.594 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:14:28.594 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:14:28.594 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:14:28.594 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:28.594 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:28.594 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:14:28.594 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:14:28.594 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:14:28.594 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:14:28.594 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:14:28.595 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:14:28.595 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:14:28.595 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:28.595 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:14:28.595 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:28.595 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:28.595 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:28.595 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:28.595 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:28.595 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:28.595 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:28.595 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:14:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:14:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:14:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:14:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:14:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:14:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.598 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.598 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.598 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.598 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:14:28.598 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:14:28.598 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:28.598 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:28.598 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:28.598 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.598 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.598 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:14:28.598 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:14:28.598 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:28.599 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:28.599 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.599 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:28.599 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:14:28.599 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.599 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.599 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.599 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:28.599 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:14:28.599 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.599 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.599 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.599 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.600 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:14:28.600 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:14:28.600 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:28.600 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:28.600 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:28.600 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.600 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:14:28.600 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:14:28.600 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:28.600 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:28.600 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.600 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:28.601 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:14:28.601 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.601 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.601 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:28.601 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:14:28.601 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.601 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.601 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.601 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:14:28.601 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:14:28.601 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:28.601 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:28.602 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:28.602 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.602 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.602 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.602 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:14:28.602 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:14:28.602 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.602 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:28.602 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:28.602 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:14:28.602 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.602 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.602 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:14:28.603 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.603 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:28.603 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:28.603 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:14:28.603 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.603 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.603 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:14:28.603 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.603 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:28.603 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:28.603 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:14:28.603 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.603 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.604 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:14:28.604 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:14:28.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 10 librados: watch_flush enter 2024-09-16T05:14:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 10 librados: watch_flush exit 2024-09-16T05:14:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 20 client.54623.objecter shutdown clearing up homeless session... 2024-09-16T05:14:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 10 client.54623.objecter successfully canceled tick 2024-09-16T05:14:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 1 -- 172.21.15.114:0/3412456261 >> v2:172.21.15.149:6812/1185531135 conn(0x7f68dc037810 msgr2=0x7f68dc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 1 --2- 172.21.15.114:0/3412456261 >> v2:172.21.15.149:6812/1185531135 conn(0x7f68dc037810 0x7f68dc039cd0 secure :-1 s=READY pgs=3496 cs=0 l=1 rev1=1 crypto rx=0x7f68f8120c40 tx=0x7f68e8009040 comp rx=0 tx=0).stop 2024-09-16T05:14:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 10 monclient: shutdown 2024-09-16T05:14:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 1 -- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68f8137730 msgr2=0x7f68f813c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f68febd6640 1 -- 172.21.15.114:0/3412456261 reap_dead start 2024-09-16T05:14:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68f8137730 0x7f68f813c510 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f68f80a3890 tx=0x7f68f0019690 comp rx=0 tx=0).stop 2024-09-16T05:14:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 1 -- 172.21.15.114:0/3412456261 shutdown_connections 2024-09-16T05:14:28.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 1 --2- 172.21.15.114:0/3412456261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68f8137730 0x7f68f813c510 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 1 -- 172.21.15.114:0/3412456261 >> 172.21.15.114:0/3412456261 conn(0x7f68f80a9c70 msgr2=0x7f68f80a4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 1 -- 172.21.15.114:0/3412456261 shutdown_connections 2024-09-16T05:14:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 1 -- 172.21.15.114:0/3412456261 wait complete. 2024-09-16T05:14:28.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.564+0000 7f6900639640 1 librados: shutdown 2024-09-16T05:14:28.607 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:14:28.607 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:14:28.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:14:28.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 10 monclient: get_monmap_and_config 2024-09-16T05:14:28.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 10 monclient: build_initial_monmap 2024-09-16T05:14:28.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 10 monclient: monmap: 2024-09-16T05:14:28.723 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:28.723 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.723 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:28.724807+0000 2024-09-16T05:14:28.723 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:28.724807+0000 2024-09-16T05:14:28.723 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:28.723 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.723 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:28.723 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:28.723 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:28.724 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:28.724 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:28.724 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 1 Processor -- start 2024-09-16T05:14:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 1 -- start start 2024-09-16T05:14:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 10 monclient: init 2024-09-16T05:14:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-16T05:14:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e74098500 0x7f9e740988e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 10 monclient(hunting): picked mon.noname-b con 0x7f9e74098500 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f9e7409c9b0 0x7f9e7409cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 10 monclient(hunting): picked mon.noname-c con 0x7f9e7409c9b0 addr v2:172.21.15.149:3300/0 2024-09-16T05:14:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e7409d2d0 addr v1:172.21.15.149:6789/0 2024-09-16T05:14:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e7409db80 con 0x7f9e7409d2d0 2024-09-16T05:14:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f9e7409e070 con 0x7f9e7409c9b0 2024-09-16T05:14:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9e7409f550 con 0x7f9e74098500 2024-09-16T05:14:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7c766640 10 monclient(hunting): authenticate will time out at 1353.353823s 2024-09-16T05:14:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e74098500 0x7f9e740988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e74098500 0x7f9e740988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:48610/0 (socket says 172.21.15.114:48610) 2024-09-16T05:14:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 1 -- 172.21.15.114:0/2472644760 learned_addr learned my addr 172.21.15.114:0/2472644760 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 10 monclient(hunting): get_auth_request con 0x7f9e74098500 auth_method 0 2024-09-16T05:14:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e79d01640 1 --2- 172.21.15.114:0/2472644760 >> v2:172.21.15.149:3300/0 conn(0x7f9e7409c9b0 0x7f9e7409cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e79d01640 10 monclient(hunting): get_auth_request con 0x7f9e7409c9b0 auth_method 0 2024-09-16T05:14:28.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e79d01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e79d01640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e79d01640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 10 monclient(hunting): handle_auth_done global_id 54626 payload 274 2024-09-16T05:14:28.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 1 -- 172.21.15.114:0/2472644760 >> v2:172.21.15.149:3300/0 conn(0x7f9e7409c9b0 msgr2=0x7f9e7409cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:28.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 1 --2- 172.21.15.114:0/2472644760 >> v2:172.21.15.149:3300/0 conn(0x7f9e7409c9b0 0x7f9e7409cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 1 -- 172.21.15.114:0/2472644760 >> v1:172.21.15.149:6789/0 conn(0x7f9e7409d2d0 legacy=0x7f9e7413a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:28.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 10 monclient: _finish_hunting 0 2024-09-16T05:14:28.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 1 monclient: found mon.noname-b 2024-09-16T05:14:28.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:28.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:14:28.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.720+0000 7f9e7a502640 1 -- 172.21.15.114:0/2472644760 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e7409e2d0 con 0x7f9e74098500 2024-09-16T05:14:28.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 1 --2- 172.21.15.114:0/2472644760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e74098500 0x7f9e740988e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9e70009870 tx=0x7f9e7001d4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e3d4ab8daf60e36 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79500640 1 -- 172.21.15.114:0/2472644760 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e700b1070 con 0x7f9e74098500 2024-09-16T05:14:28.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79500640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79500640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:14:28.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79500640 10 monclient: dump: 2024-09-16T05:14:28.736 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.736 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.736 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.736 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.736 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.736 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.736 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.736 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.736 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.736 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79500640 10 monclient: _finish_auth 0 2024-09-16T05:14:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79500640 10 monclient: _check_auth_tickets 2024-09-16T05:14:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79500640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79500640 1 -- 172.21.15.114:0/2472644760 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e70004230 con 0x7f9e74098500 2024-09-16T05:14:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 5 monclient: authenticate success, global_id 54626 2024-09-16T05:14:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79500640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 4 set_mon_vals no callback set 2024-09-16T05:14:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79500640 1 -- 172.21.15.114:0/2472644760 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e70003620 con 0x7f9e74098500 2024-09-16T05:14:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79500640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79500640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79500640 10 monclient: dump: 2024-09-16T05:14:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.738 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 10 monclient: shutdown 2024-09-16T05:14:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 -- 172.21.15.114:0/2472644760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e74098500 msgr2=0x7f9e740988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 --2- 172.21.15.114:0/2472644760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e74098500 0x7f9e740988e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9e70009870 tx=0x7f9e7001d4f0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 -- 172.21.15.114:0/2472644760 shutdown_connections 2024-09-16T05:14:28.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 --2- 172.21.15.114:0/2472644760 >> v2:172.21.15.149:3300/0 conn(0x7f9e7409c9b0 0x7f9e7409cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 --2- 172.21.15.114:0/2472644760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e74098500 0x7f9e740988e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 -- 172.21.15.114:0/2472644760 >> 172.21.15.114:0/2472644760 conn(0x7f9e740a1fb0 msgr2=0x7f9e740a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:28.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 -- 172.21.15.114:0/2472644760 shutdown_connections 2024-09-16T05:14:28.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 -- 172.21.15.114:0/2472644760 wait complete. 2024-09-16T05:14:28.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 10 monclient: build_initial_monmap 2024-09-16T05:14:28.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 10 monclient: monmap: 2024-09-16T05:14:28.739 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:28.739 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:28.739 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:28.739 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:28.739 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:28.740 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 librados: starting msgr at 2024-09-16T05:14:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 librados: starting objecter 2024-09-16T05:14:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 Processor -- start 2024-09-16T05:14:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 -- start start 2024-09-16T05:14:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 librados: setting wanted keys 2024-09-16T05:14:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 librados: calling monclient init 2024-09-16T05:14:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 10 monclient: init 2024-09-16T05:14:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:14:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e74098500 0x7f9e74097030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 10 monclient(hunting): picked mon.noname-a con 0x7f9e74098500 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7409c9b0 0x7f9e74091940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 10 monclient(hunting): picked mon.noname-c con 0x7f9e7409c9b0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e7409d2d0 0x7f9e74091e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 10 monclient(hunting): picked mon.noname-b con 0x7f9e7409d2d0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9e74098fa0 con 0x7f9e74098500 2024-09-16T05:14:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f9e74098e20 con 0x7f9e7409d2d0 2024-09-16T05:14:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9e74099e20 con 0x7f9e7409c9b0 2024-09-16T05:14:28.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:28.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7c766640 10 monclient(hunting): authenticate will time out at 1353.359002s 2024-09-16T05:14:28.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e74098500 0x7f9e74097030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79d01640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7409c9b0 0x7f9e74091940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e74098500 0x7f9e74097030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36708/0 (socket says 172.21.15.114:36708) 2024-09-16T05:14:28.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 1 -- 172.21.15.114:0/2094196206 learned_addr learned my addr 172.21.15.114:0/2094196206 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:28.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 10 monclient(hunting): get_auth_request con 0x7f9e74098500 auth_method 0 2024-09-16T05:14:28.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79d01640 10 monclient(hunting): get_auth_request con 0x7f9e7409c9b0 auth_method 0 2024-09-16T05:14:28.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79d01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79d01640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79d01640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79d01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79d01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e79d01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7ad03640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e7409d2d0 0x7f9e74091e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7ad03640 10 monclient(hunting): get_auth_request con 0x7f9e7409d2d0 auth_method 0 2024-09-16T05:14:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7ad03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7ad03640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7ad03640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 10 monclient(hunting): handle_auth_done global_id 74130 payload 824 2024-09-16T05:14:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 1 -- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7409c9b0 msgr2=0x7f9e74091940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7409c9b0 0x7f9e74091940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 1 -- 172.21.15.114:0/2094196206 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e7409d2d0 msgr2=0x7f9e74091e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e7409d2d0 0x7f9e74091e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 10 monclient: _finish_hunting 0 2024-09-16T05:14:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 1 monclient: found mon.noname-a 2024-09-16T05:14:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 1 -- 172.21.15.114:0/2094196206 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e74094140 con 0x7f9e74098500 2024-09-16T05:14:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e7a502640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e74098500 0x7f9e74097030 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9e700af040 tx=0x7f9e7001a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e6affd640 10 client.?.objecter ms_handle_connect 0x7f9e74098500 2024-09-16T05:14:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e6affd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e6affd640 1 -- 172.21.15.114:0/2094196206 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e700ba070 con 0x7f9e74098500 2024-09-16T05:14:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e6affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e6affd640 10 monclient: dump: 2024-09-16T05:14:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.745 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e6affd640 10 monclient: _finish_auth 0 2024-09-16T05:14:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e6affd640 10 monclient: _check_auth_tickets 2024-09-16T05:14:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e6affd640 1 -- 172.21.15.114:0/2094196206 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e70003de0 con 0x7f9e74098500 2024-09-16T05:14:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e6affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e6affd640 1 -- 172.21.15.114:0/2094196206 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e700b1040 con 0x7f9e74098500 2024-09-16T05:14:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e6affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e6affd640 10 monclient: dump: 2024-09-16T05:14:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.746 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.747 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.747 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.747 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.747 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.724+0000 7f9e78cff640 4 set_mon_vals no callback set 2024-09-16T05:14:28.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7c766640 5 monclient: authenticate success, global_id 74130 2024-09-16T05:14:28.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7c766640 10 monclient: _renew_subs 2024-09-16T05:14:28.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7c766640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:28.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7c766640 1 -- 172.21.15.114:0/2094196206 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e74072df0 con 0x7f9e74098500 2024-09-16T05:14:28.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7c766640 10 client.74130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:28.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7c766640 10 monclient: _renew_subs 2024-09-16T05:14:28.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7c766640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:28.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7c766640 1 -- 172.21.15.114:0/2094196206 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e74095310 con 0x7f9e74098500 2024-09-16T05:14:28.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7c766640 1 librados: init done 2024-09-16T05:14:28.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7c766640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:28.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7c766640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e74095550 0x7f9e740923c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7c766640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7c766640 1 -- 172.21.15.114:0/2094196206 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9e74069390 con 0x7f9e74095550 2024-09-16T05:14:28.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e79d01640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e74095550 0x7f9e740923c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 1 -- 172.21.15.114:0/2094196206 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9e700ac440 con 0x7f9e74098500 2024-09-16T05:14:28.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e79d01640 10 monclient: get_auth_request con 0x7f9e74095550 auth_method 0 2024-09-16T05:14:28.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e79d01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e79d01640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e79d01640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e79d01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e79d01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e79d01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e79d01640 10 monclient(hunting): handle_auth_done global_id 54632 payload 824 2024-09-16T05:14:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 1 --2- 172.21.15.114:0/2094196206 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9e58037810 0x7f9e58039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e79d01640 1 -- 172.21.15.114:0/2094196206 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9e740988e0 con 0x7f9e74095550 2024-09-16T05:14:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 1 -- 172.21.15.114:0/2094196206 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(289..289 src has 1..289) ==== 5505+0+0 (secure 0 0 0) 0x7f9e700e20a0 con 0x7f9e74098500 2024-09-16T05:14:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 10 client.74130.objecter ms_dispatch 0x7f9e74000ba0 osd_map(289..289 src has 1..289) 2024-09-16T05:14:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 3 client.74130.objecter handle_osd_map got epochs [289,289] > 0 2024-09-16T05:14:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 3 client.74130.objecter handle_osd_map decoding full epoch 289 2024-09-16T05:14:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 20 client.74130.objecter dump_active .. 0 homeless 2024-09-16T05:14:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e79d01640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e74095550 0x7f9e740923c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9e74097a30 tx=0x7f9e64010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 10 client.74130.objecter ms_handle_connect 0x7f9e74095550 2024-09-16T05:14:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 10 client.74130.objecter resend_mon_ops 2024-09-16T05:14:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 1 -- 172.21.15.114:0/2094196206 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e64018020 con 0x7f9e74095550 2024-09-16T05:14:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7ad03640 1 --2- 172.21.15.114:0/2094196206 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9e58037810 0x7f9e58039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:28.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 10 monclient: dump: 2024-09-16T05:14:28.750 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.750 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.750 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.750 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.750 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.750 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.750 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.750 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.750 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.750 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7ad03640 10 monclient: get_auth_request con 0x7f9e58037810 auth_method 0 2024-09-16T05:14:28.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 1 -- 172.21.15.114:0/2094196206 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9e64005bb0 con 0x7f9e74095550 2024-09-16T05:14:28.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:28.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:28.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 1 -- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e74095550 msgr2=0x7f9e740923c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e74095550 0x7f9e740923c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9e74097a30 tx=0x7f9e64010c80 comp rx=0 tx=0).stop 2024-09-16T05:14:28.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 1 -- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e74095550 msgr2=0x7f9e740923c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:28.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e74095550 0x7f9e740923c0 secure :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9e74097a30 tx=0x7f9e64010c80 comp rx=0 tx=0).stop 2024-09-16T05:14:28.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e6affd640 10 client.74130.objecter ms_handle_connect 0x7f9e58037810 2024-09-16T05:14:28.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.728+0000 7f9e7ad03640 1 --2- 172.21.15.114:0/2094196206 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9e58037810 0x7f9e58039cd0 secure :-1 s=READY pgs=3497 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0054b0 tx=0x7f9e6c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.732+0000 7f9e7c766640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:14:28.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.732+0000 7f9e7c766640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7408e010 0x7f9e7408e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.732+0000 7f9e7c766640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.732+0000 7f9e7c766640 1 -- 172.21.15.114:0/2094196206 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9e74093ec0 con 0x7f9e7408e010 2024-09-16T05:14:28.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.732+0000 7f9e79d01640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7408e010 0x7f9e7408e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e79d01640 10 monclient: get_auth_request con 0x7f9e7408e010 auth_method 0 2024-09-16T05:14:28.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e79d01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e79d01640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e79d01640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e79d01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e79d01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e79d01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e79d01640 10 monclient(hunting): handle_auth_done global_id 54635 payload 824 2024-09-16T05:14:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e79d01640 1 -- 172.21.15.114:0/2094196206 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9e740686f0 con 0x7f9e7408e010 2024-09-16T05:14:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e6affd640 10 client.74130.objecter ms_handle_connect 0x7f9e7408e010 2024-09-16T05:14:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e6affd640 10 client.74130.objecter resend_mon_ops 2024-09-16T05:14:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e79d01640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7408e010 0x7f9e7408e420 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9e74097a30 tx=0x7f9e640046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e6affd640 1 -- 172.21.15.114:0/2094196206 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e64018030 con 0x7f9e7408e010 2024-09-16T05:14:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:28.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e6affd640 10 monclient: dump: 2024-09-16T05:14:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.755 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e6affd640 1 -- 172.21.15.114:0/2094196206 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f9e6400ea70 con 0x7f9e7408e010 2024-09-16T05:14:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e6affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:14:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e6affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:28.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e6affd640 1 -- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7408e010 msgr2=0x7f9e7408e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e6affd640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7408e010 0x7f9e7408e420 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9e74097a30 tx=0x7f9e640046d0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e6affd640 1 -- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7408e010 msgr2=0x7f9e7408e420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:28.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e6affd640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e7408e010 0x7f9e7408e420 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9e74097a30 tx=0x7f9e640046d0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.756 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:14:28.756 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:14:28.756 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:14:28.756 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:14:28.756 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 34, 2024-09-16T05:14:28.756 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 282872, 2024-09-16T05:14:28.757 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:14:28.757 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:14:28.757 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:14:28.757 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:14:28.757 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.757 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:14:28.757 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:28.757 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:14:28.757 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:14:28.757 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:28.757 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:28.757 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:28.757 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:28.757 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:28.758 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:28.758 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:28.758 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:28.758 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:28.758 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:28.758 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:28.758 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.758 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:14:28.758 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:14:28.758 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:28.758 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:28.758 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:28.758 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:28.758 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:28.759 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:28.759 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:28.759 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:28.759 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:28.759 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:28.759 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:28.759 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.759 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.759 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:14:28.759 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:14:28.759 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:14:28.759 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:14:28.759 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:14:28.759 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:14:28.760 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:28.760 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:28.760 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:14:28.760 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:14:28.760 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:14:28.760 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:14:28.760 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:14:28.760 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:14:28.760 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:14:28.760 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:28.760 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:14:28.760 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:28.760 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:28.761 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:28.761 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:28.761 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:28.761 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:28.761 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:28.761 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:28.761 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:28.761 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:28.761 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:28.761 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.761 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:14:28.761 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.761 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:14:28.762 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.762 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:14:28.762 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:14:28.762 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:28.762 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:28.762 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.762 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:28.762 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:14:28.762 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.762 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.762 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.762 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:28.762 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:14:28.762 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.763 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.763 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.763 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.763 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:14:28.763 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:14:28.763 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:28.763 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:28.763 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:28.763 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.763 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.763 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:14:28.763 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:14:28.763 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:28.763 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:28.764 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.764 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:28.764 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:14:28.764 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.764 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.764 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.764 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:28.764 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:14:28.764 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.764 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.764 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.764 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.764 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:14:28.764 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:14:28.765 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:28.765 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:28.765 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:28.765 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.765 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.765 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:14:28.765 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:14:28.765 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:28.765 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:28.765 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.765 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:28.765 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:14:28.765 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.766 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.766 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.766 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:28.766 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:14:28.766 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.766 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.766 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.766 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.766 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:14:28.766 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:14:28.766 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:28.766 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:28.766 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:28.767 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.767 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.767 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.767 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:14:28.767 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:14:28.767 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.767 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:28.767 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:28.767 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:14:28.767 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.767 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.767 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:14:28.767 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.767 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:28.768 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:28.768 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:14:28.768 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.768 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.768 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:14:28.768 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.768 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:28.768 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:28.768 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:14:28.768 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.768 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.768 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.768 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:14:28.768 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:14:28.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 10 librados: watch_flush enter 2024-09-16T05:14:28.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 10 librados: watch_flush exit 2024-09-16T05:14:28.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 20 client.74130.objecter shutdown clearing up homeless session... 2024-09-16T05:14:28.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 10 client.74130.objecter successfully canceled tick 2024-09-16T05:14:28.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 1 -- 172.21.15.114:0/2094196206 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9e58037810 msgr2=0x7f9e58039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 1 --2- 172.21.15.114:0/2094196206 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9e58037810 0x7f9e58039cd0 secure :-1 s=READY pgs=3497 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0054b0 tx=0x7f9e6c009040 comp rx=0 tx=0).stop 2024-09-16T05:14:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 10 monclient: shutdown 2024-09-16T05:14:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 1 -- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e74098500 msgr2=0x7f9e74097030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e74098500 0x7f9e74097030 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9e700af040 tx=0x7f9e7001a290 comp rx=0 tx=0).stop 2024-09-16T05:14:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7ad03640 1 -- 172.21.15.114:0/2094196206 reap_dead start 2024-09-16T05:14:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 1 -- 172.21.15.114:0/2094196206 shutdown_connections 2024-09-16T05:14:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 1 --2- 172.21.15.114:0/2094196206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e74098500 0x7f9e74097030 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 1 -- 172.21.15.114:0/2094196206 >> 172.21.15.114:0/2094196206 conn(0x7f9e740a1fb0 msgr2=0x7f9e74071bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 1 -- 172.21.15.114:0/2094196206 shutdown_connections 2024-09-16T05:14:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 1 -- 172.21.15.114:0/2094196206 wait complete. 2024-09-16T05:14:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.736+0000 7f9e7c766640 1 librados: shutdown 2024-09-16T05:14:28.772 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:14:28.772 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:14:28.772 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:14:28.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 10 monclient: get_monmap_and_config 2024-09-16T05:14:28.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 10 monclient: build_initial_monmap 2024-09-16T05:14:28.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 10 monclient: monmap: 2024-09-16T05:14:28.890 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:28.891 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.891 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:28.892411+0000 2024-09-16T05:14:28.891 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:28.892411+0000 2024-09-16T05:14:28.891 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:28.891 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.891 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:28.891 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:28.891 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:28.891 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:28.891 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:28.891 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 1 Processor -- start 2024-09-16T05:14:28.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 1 -- start start 2024-09-16T05:14:28.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 10 monclient: init 2024-09-16T05:14:28.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:28.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-16T05:14:28.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a581376e0 addr v1:172.21.15.149:6789/0 2024-09-16T05:14:28.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f0a581370a0 0x7f0a58137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 10 monclient(hunting): picked mon.noname-c con 0x7f0a581370a0 addr v2:172.21.15.149:3300/0 2024-09-16T05:14:28.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a580a5d00 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:28.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a580a5090 con 0x7f0a580a5d00 2024-09-16T05:14:28.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a5809bd20 con 0x7f0a581376e0 2024-09-16T05:14:28.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f0a58138000 con 0x7f0a581370a0 2024-09-16T05:14:28.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:28.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 10 monclient(hunting): authenticate will time out at 1353.521586s 2024-09-16T05:14:28.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5759e640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0a580a5d00 0x7f0a580a4980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:47722/0 (socket says 172.21.15.114:47722) 2024-09-16T05:14:28.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5759e640 1 -- 172.21.15.114:0/1099800524 learned_addr learned my addr 172.21.15.114:0/1099800524 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:14:28.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5659c640 1 --2- 172.21.15.114:0/1099800524 >> v2:172.21.15.149:3300/0 conn(0x7f0a581370a0 0x7f0a58137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 -- 172.21.15.114:0/1099800524 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3306921669 0 0) 0x7f0a580a5090 con 0x7f0a580a5d00 2024-09-16T05:14:28.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:14:28.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient(hunting): my global_id is 74133 2024-09-16T05:14:28.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 -- 172.21.15.114:0/1099800524 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a40003530 con 0x7f0a580a5d00 2024-09-16T05:14:28.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5659c640 10 monclient(hunting): get_auth_request con 0x7f0a581370a0 auth_method 0 2024-09-16T05:14:28.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 -- 172.21.15.114:0/1099800524 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3825914381 0 0) 0x7f0a40003530 con 0x7f0a580a5d00 2024-09-16T05:14:28.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 -- 172.21.15.114:0/1099800524 >> v2:172.21.15.149:3300/0 conn(0x7f0a581370a0 msgr2=0x7f0a58137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:28.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 --2- 172.21.15.114:0/1099800524 >> v2:172.21.15.149:3300/0 conn(0x7f0a581370a0 0x7f0a58137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 -- 172.21.15.114:0/1099800524 >> v1:172.21.15.149:6789/0 conn(0x7f0a581376e0 legacy=0x7f0a580a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:14:28.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:14:28.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:28.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:14:28.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 -- 172.21.15.114:0/1099800524 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a5809c000 con 0x7f0a580a5d00 2024-09-16T05:14:28.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient: _finish_auth 0 2024-09-16T05:14:28.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:14:28.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:28.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 -- 172.21.15.114:0/1099800524 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f0a50001a00 con 0x7f0a580a5d00 2024-09-16T05:14:28.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:14:28.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient: dump: 2024-09-16T05:14:28.906 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.907 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.907 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.907 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.907 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.907 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.907 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.907 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.907 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.907 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:14:28.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:28.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 -- 172.21.15.114:0/1099800524 >> v1:172.21.15.114:6789/0 conn(0x7f0a580a5d00 legacy=0x7f0a580a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:14:28.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 --2- 172.21.15.114:0/1099800524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a40003a10 0x7f0a40003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient(hunting): picked mon.c con 0x7f0a40003a10 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:28.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 --2- 172.21.15.114:0/1099800524 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a40004810 0x7f0a40004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient(hunting): picked mon.b con 0x7f0a40004810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:28.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 --2- 172.21.15.114:0/1099800524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a400052f0 0x7f0a4000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient(hunting): picked mon.a con 0x7f0a400052f0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:28.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 -- 172.21.15.114:0/1099800524 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4000c1c0 con 0x7f0a400052f0 2024-09-16T05:14:28.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 -- 172.21.15.114:0/1099800524 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4000c370 con 0x7f0a40004810 2024-09-16T05:14:28.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 1 -- 172.21.15.114:0/1099800524 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f0a4000c570 con 0x7f0a40003a10 2024-09-16T05:14:28.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:28.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a55d9b640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:14:28.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5759e640 1 --2- 172.21.15.114:0/1099800524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a400052f0 0x7f0a4000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 1 --2- 172.21.15.114:0/1099800524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a40003a10 0x7f0a40003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 10 monclient(hunting): get_auth_request con 0x7f0a40003a10 auth_method 0 2024-09-16T05:14:28.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:28.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5759e640 10 monclient(hunting): get_auth_request con 0x7f0a400052f0 auth_method 0 2024-09-16T05:14:28.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5759e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:28.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:14:28.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5759e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:14:28.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5659c640 1 --2- 172.21.15.114:0/1099800524 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a40004810 0x7f0a40004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5659c640 10 monclient(hunting): get_auth_request con 0x7f0a40004810 auth_method 0 2024-09-16T05:14:28.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:28.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 10 monclient(hunting): handle_auth_done global_id 74133 payload 293 2024-09-16T05:14:28.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 1 -- 172.21.15.114:0/1099800524 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a40004810 msgr2=0x7f0a40004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:28.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 1 --2- 172.21.15.114:0/1099800524 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a40004810 0x7f0a40004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 1 -- 172.21.15.114:0/1099800524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a400052f0 msgr2=0x7f0a4000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:28.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 1 --2- 172.21.15.114:0/1099800524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a400052f0 0x7f0a4000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 10 monclient: _finish_hunting 0 2024-09-16T05:14:28.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 1 monclient: found mon.c 2024-09-16T05:14:28.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:28.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:28.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 1 -- 172.21.15.114:0/1099800524 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a40003530 con 0x7f0a40003a10 2024-09-16T05:14:28.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 10 monclient: _finish_auth 0 2024-09-16T05:14:28.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 10 monclient: _check_auth_tickets 2024-09-16T05:14:28.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a56d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:28.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 5 monclient: authenticate success, global_id 74133 2024-09-16T05:14:28.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5d05b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:28.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5759e640 1 --2- 172.21.15.114:0/1099800524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a400052f0 0x7f0a4000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:14:28.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.888+0000 7f0a5759e640 1 -- 172.21.15.114:0/1099800524 reap_dead start 2024-09-16T05:14:28.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 1 --2- 172.21.15.114:0/1099800524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a40003a10 0x7f0a40003e30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0a48001780 tx=0x7f0a4801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33c234ced729e93a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a55d9b640 1 -- 172.21.15.114:0/1099800524 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a480013a0 con 0x7f0a40003a10 2024-09-16T05:14:28.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a55d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a55d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:28.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a55d9b640 10 monclient: dump: 2024-09-16T05:14:28.913 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.913 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.913 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.913 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.913 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.913 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.913 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.913 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.913 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.913 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:28.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a55d9b640 1 -- 172.21.15.114:0/1099800524 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a480052f0 con 0x7f0a40003a10 2024-09-16T05:14:28.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a55d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:28.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:28.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 4 set_mon_vals no callback set 2024-09-16T05:14:28.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a55d9b640 1 -- 172.21.15.114:0/1099800524 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48003840 con 0x7f0a40003a10 2024-09-16T05:14:28.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a55d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a55d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:28.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a55d9b640 10 monclient: dump: 2024-09-16T05:14:28.914 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.914 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.914 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.914 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.914 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.914 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.914 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.915 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 10 monclient: shutdown 2024-09-16T05:14:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 -- 172.21.15.114:0/1099800524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a40003a10 msgr2=0x7f0a40003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 --2- 172.21.15.114:0/1099800524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a40003a10 0x7f0a40003e30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0a48001780 tx=0x7f0a4801c3e0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 -- 172.21.15.114:0/1099800524 shutdown_connections 2024-09-16T05:14:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 --2- 172.21.15.114:0/1099800524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a40003a10 0x7f0a40003e30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 -- 172.21.15.114:0/1099800524 >> 172.21.15.114:0/1099800524 conn(0x7f0a580a9a40 msgr2=0x7f0a580a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 -- 172.21.15.114:0/1099800524 shutdown_connections 2024-09-16T05:14:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 -- 172.21.15.114:0/1099800524 wait complete. 2024-09-16T05:14:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 10 monclient: build_initial_monmap 2024-09-16T05:14:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 10 monclient: monmap: 2024-09-16T05:14:28.916 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:28.916 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:28.916 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:28.916 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:28.916 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:28.916 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.916 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:28.916 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:28.916 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:28.916 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 librados: starting msgr at 2024-09-16T05:14:28.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 librados: starting objecter 2024-09-16T05:14:28.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 Processor -- start 2024-09-16T05:14:28.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 -- start start 2024-09-16T05:14:28.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 librados: setting wanted keys 2024-09-16T05:14:28.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 librados: calling monclient init 2024-09-16T05:14:28.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 10 monclient: init 2024-09-16T05:14:28.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:14:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a40003a10 0x7f0a581cf890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 10 monclient(hunting): picked mon.noname-c con 0x7f0a40003a10 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a581cfdd0 0x7f0a58124a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 10 monclient(hunting): picked mon.noname-b con 0x7f0a581cfdd0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a58124f40 0x7f0a581253c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 10 monclient(hunting): picked mon.noname-a con 0x7f0a58124f40 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0a580a4e10 con 0x7f0a58124f40 2024-09-16T05:14:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f0a580a4c90 con 0x7f0a581cfdd0 2024-09-16T05:14:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f0a5809c5c0 con 0x7f0a40003a10 2024-09-16T05:14:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5d05b640 10 monclient(hunting): authenticate will time out at 1353.527469s 2024-09-16T05:14:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a40003a10 0x7f0a581cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a40003a10 0x7f0a581cf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:48662/0 (socket says 172.21.15.114:48662) 2024-09-16T05:14:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 1 -- 172.21.15.114:0/4290777811 learned_addr learned my addr 172.21.15.114:0/4290777811 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 10 monclient(hunting): get_auth_request con 0x7f0a40003a10 auth_method 0 2024-09-16T05:14:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5759e640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a58124f40 0x7f0a581253c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5759e640 10 monclient(hunting): get_auth_request con 0x7f0a58124f40 auth_method 0 2024-09-16T05:14:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5759e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5659c640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a581cfdd0 0x7f0a58124a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5659c640 10 monclient(hunting): get_auth_request con 0x7f0a581cfdd0 auth_method 0 2024-09-16T05:14:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 10 monclient(hunting): handle_auth_done global_id 54638 payload 824 2024-09-16T05:14:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 1 -- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a581cfdd0 msgr2=0x7f0a58124a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a581cfdd0 0x7f0a58124a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 1 -- 172.21.15.114:0/4290777811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a58124f40 msgr2=0x7f0a581253c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a58124f40 0x7f0a581253c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 10 monclient: _finish_hunting 0 2024-09-16T05:14:28.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 1 monclient: found mon.noname-c 2024-09-16T05:14:28.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:28.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:14:28.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a56d9d640 1 -- 172.21.15.114:0/4290777811 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a58129320 con 0x7f0a40003a10 2024-09-16T05:14:28.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.892+0000 7f0a5759e640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a58124f40 0x7f0a581253c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:14:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a56d9d640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a40003a10 0x7f0a581cf890 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0a48097bd0 tx=0x7f0a40005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 client.?.objecter ms_handle_connect 0x7f0a40003a10 2024-09-16T05:14:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 1 -- 172.21.15.114:0/4290777811 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a480b8020 con 0x7f0a40003a10 2024-09-16T05:14:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:14:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 monclient: dump: 2024-09-16T05:14:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.923 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:14:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:14:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 1 -- 172.21.15.114:0/4290777811 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a480054a0 con 0x7f0a40003a10 2024-09-16T05:14:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5d05b640 5 monclient: authenticate success, global_id 54638 2024-09-16T05:14:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5d05b640 10 monclient: _renew_subs 2024-09-16T05:14:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5d05b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5d05b640 1 -- 172.21.15.114:0/4290777811 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a58126af0 con 0x7f0a40003a10 2024-09-16T05:14:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 1 -- 172.21.15.114:0/4290777811 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48007a40 con 0x7f0a40003a10 2024-09-16T05:14:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 monclient: dump: 2024-09-16T05:14:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.925 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5d05b640 10 client.54638.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:28.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5d05b640 10 monclient: _renew_subs 2024-09-16T05:14:28.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5d05b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:28.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5d05b640 1 -- 172.21.15.114:0/4290777811 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a5809d4f0 con 0x7f0a40003a10 2024-09-16T05:14:28.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5559a640 4 set_mon_vals no callback set 2024-09-16T05:14:28.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5d05b640 1 librados: init done 2024-09-16T05:14:28.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 1 -- 172.21.15.114:0/4290777811 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0a480bb020 con 0x7f0a40003a10 2024-09-16T05:14:28.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5d05b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:28.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5d05b640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a58125900 0x7f0a58128180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5d05b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5d05b640 1 -- 172.21.15.114:0/4290777811 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f0a580a2000 con 0x7f0a58125900 2024-09-16T05:14:28.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 1 --2- 172.21.15.114:0/4290777811 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0a44037810 0x7f0a44039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 1 -- 172.21.15.114:0/4290777811 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(289..289 src has 1..289) ==== 5505+0+0 (secure 0 0 0) 0x7f0a480e2c60 con 0x7f0a40003a10 2024-09-16T05:14:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 client.54638.objecter ms_dispatch 0x7f0a58000ba0 osd_map(289..289 src has 1..289) 2024-09-16T05:14:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 3 client.54638.objecter handle_osd_map got epochs [289,289] > 0 2024-09-16T05:14:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 3 client.54638.objecter handle_osd_map decoding full epoch 289 2024-09-16T05:14:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 20 client.54638.objecter dump_active .. 0 homeless 2024-09-16T05:14:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5659c640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a58125900 0x7f0a58128180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5659c640 10 monclient: get_auth_request con 0x7f0a58125900 auth_method 0 2024-09-16T05:14:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5759e640 1 --2- 172.21.15.114:0/4290777811 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0a44037810 0x7f0a44039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5759e640 10 monclient: get_auth_request con 0x7f0a44037810 auth_method 0 2024-09-16T05:14:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5659c640 10 monclient(hunting): handle_auth_done global_id 64357 payload 824 2024-09-16T05:14:28.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5659c640 1 -- 172.21.15.114:0/4290777811 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0a580a5050 con 0x7f0a58125900 2024-09-16T05:14:28.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 client.54638.objecter ms_handle_connect 0x7f0a44037810 2024-09-16T05:14:28.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5759e640 1 --2- 172.21.15.114:0/4290777811 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0a44037810 0x7f0a44039cd0 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7f0a58073760 tx=0x7f0a50006040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 client.54638.objecter ms_handle_connect 0x7f0a58125900 2024-09-16T05:14:28.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 client.54638.objecter resend_mon_ops 2024-09-16T05:14:28.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a5659c640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a58125900 0x7f0a58128180 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c009fd0 tx=0x7f0a4c0139a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 1 -- 172.21.15.114:0/4290777811 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c01c070 con 0x7f0a58125900 2024-09-16T05:14:28.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:14:28.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 monclient: dump: 2024-09-16T05:14:28.927 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.928 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 1 -- 172.21.15.114:0/4290777811 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0a4c0118f0 con 0x7f0a58125900 2024-09-16T05:14:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 1 -- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a58125900 msgr2=0x7f0a58128180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a58125900 0x7f0a58128180 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c009fd0 tx=0x7f0a4c0139a0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 1 -- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a58125900 msgr2=0x7f0a58128180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:28.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.896+0000 7f0a3f7fe640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a58125900 0x7f0a58128180 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c009fd0 tx=0x7f0a4c0139a0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a5d05b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:14:28.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a5d05b640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a58120ed0 0x7f0a581212e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:28.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a5d05b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a5d05b640 1 -- 172.21.15.114:0/4290777811 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f0a58068820 con 0x7f0a58120ed0 2024-09-16T05:14:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a5659c640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a58120ed0 0x7f0a581212e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a5659c640 10 monclient: get_auth_request con 0x7f0a58120ed0 auth_method 0 2024-09-16T05:14:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a5659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a5659c640 10 monclient(hunting): handle_auth_done global_id 64360 payload 824 2024-09-16T05:14:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a5659c640 1 -- 172.21.15.114:0/4290777811 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0a58125ce0 con 0x7f0a58120ed0 2024-09-16T05:14:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a5659c640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a58120ed0 0x7f0a581212e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c009fd0 tx=0x7f0a4c004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a3f7fe640 10 client.54638.objecter ms_handle_connect 0x7f0a58120ed0 2024-09-16T05:14:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a3f7fe640 10 client.54638.objecter resend_mon_ops 2024-09-16T05:14:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a3f7fe640 1 -- 172.21.15.114:0/4290777811 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c004030 con 0x7f0a58120ed0 2024-09-16T05:14:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:14:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a3f7fe640 10 monclient: dump: 2024-09-16T05:14:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:28.933 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:28.933 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:28.933 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:28.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a3f7fe640 1 -- 172.21.15.114:0/4290777811 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f0a4c0063d0 con 0x7f0a58120ed0 2024-09-16T05:14:28.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a3f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:14:28.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.904+0000 7f0a3f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:28.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a3f7fe640 1 -- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a58120ed0 msgr2=0x7f0a581212e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a3f7fe640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a58120ed0 0x7f0a581212e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c009fd0 tx=0x7f0a4c004740 comp rx=0 tx=0).stop 2024-09-16T05:14:28.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a3f7fe640 1 -- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a58120ed0 msgr2=0x7f0a581212e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:28.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a3f7fe640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0a58120ed0 0x7f0a581212e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.934 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:14:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:14:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:14:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:14:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 34, 2024-09-16T05:14:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 153045, 2024-09-16T05:14:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:14:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:14:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:14:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:14:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:14:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:14:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:14:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:14:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:14:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:14:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:14:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:14:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:14:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:14:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:14:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:14:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:14:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:14:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:14:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:14:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:14:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:14:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:14:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:14:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:14:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:14:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:14:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:14:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:14:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:14:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:14:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:14:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:14:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:14:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:14:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:14:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:14:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:14:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:14:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:14:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:14:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:14:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:14:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:14:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:14:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:14:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:14:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:14:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:14:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:14:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:14:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:14:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:28.948 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:28.948 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:14:28.948 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:14:28.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 10 librados: watch_flush enter 2024-09-16T05:14:28.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 10 librados: watch_flush exit 2024-09-16T05:14:28.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 20 client.54638.objecter shutdown clearing up homeless session... 2024-09-16T05:14:28.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 10 client.54638.objecter successfully canceled tick 2024-09-16T05:14:28.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 1 -- 172.21.15.114:0/4290777811 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0a44037810 msgr2=0x7f0a44039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 1 --2- 172.21.15.114:0/4290777811 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0a44037810 0x7f0a44039cd0 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7f0a58073760 tx=0x7f0a50006040 comp rx=0 tx=0).stop 2024-09-16T05:14:28.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 10 monclient: shutdown 2024-09-16T05:14:28.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:28.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 1 -- 172.21.15.114:0/4290777811 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a40003a10 msgr2=0x7f0a581cf890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:28.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a40003a10 0x7f0a581cf890 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0a48097bd0 tx=0x7f0a40005840 comp rx=0 tx=0).stop 2024-09-16T05:14:28.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5759e640 1 -- 172.21.15.114:0/4290777811 reap_dead start 2024-09-16T05:14:28.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 1 -- 172.21.15.114:0/4290777811 shutdown_connections 2024-09-16T05:14:28.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 1 --2- 172.21.15.114:0/4290777811 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a40003a10 0x7f0a581cf890 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:28.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 1 -- 172.21.15.114:0/4290777811 >> 172.21.15.114:0/4290777811 conn(0x7f0a580a9a40 msgr2=0x7f0a580a0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:28.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 1 -- 172.21.15.114:0/4290777811 shutdown_connections 2024-09-16T05:14:28.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 1 -- 172.21.15.114:0/4290777811 wait complete. 2024-09-16T05:14:28.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:28.908+0000 7f0a5d05b640 1 librados: shutdown 2024-09-16T05:14:28.951 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:14:28.951 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:14:28.951 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:14:29.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 10 monclient: get_monmap_and_config 2024-09-16T05:14:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 10 monclient: build_initial_monmap 2024-09-16T05:14:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 10 monclient: monmap: 2024-09-16T05:14:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:29.066694+0000 2024-09-16T05:14:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:29.066694+0000 2024-09-16T05:14:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:29.069 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:29.069 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:29.069 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:29.069 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:29.069 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:29.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 1 Processor -- start 2024-09-16T05:14:29.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 1 -- start start 2024-09-16T05:14:29.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 10 monclient: init 2024-09-16T05:14:29.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:29.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T05:14:29.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb64809b6c0 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:29.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb6480a15d0 0x7fb64813a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:29.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 10 monclient(hunting): picked mon.noname-a con 0x7fb6480a15d0 addr v2:172.21.15.114:3300/0 2024-09-16T05:14:29.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb6480a19e0 0x7fb64813cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:29.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 10 monclient(hunting): picked mon.noname-c con 0x7fb6480a19e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:14:29.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb64809d4e0 con 0x7fb64809b6c0 2024-09-16T05:14:29.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:29.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb64809e390 con 0x7fb6480a15d0 2024-09-16T05:14:29.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:29.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fb64809ece0 con 0x7fb6480a19e0 2024-09-16T05:14:29.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:29.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb64e41f640 10 monclient(hunting): authenticate will time out at 1353.695837s 2024-09-16T05:14:29.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.060+0000 7fb6477fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb6480a15d0 0x7fb64813a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:29.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb6480a15d0 0x7fb64813a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36730/0 (socket says 172.21.15.114:36730) 2024-09-16T05:14:29.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 1 -- 172.21.15.114:0/3691769988 learned_addr learned my addr 172.21.15.114:0/3691769988 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:29.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 10 monclient(hunting): get_auth_request con 0x7fb6480a15d0 auth_method 0 2024-09-16T05:14:29.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:29.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:29.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:29.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:29.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:29.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:29.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb64c9bc640 1 --2- 172.21.15.114:0/3691769988 >> v2:172.21.15.149:3300/0 conn(0x7fb6480a19e0 0x7fb64813cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:29.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 10 monclient(hunting): handle_auth_done global_id 74139 payload 274 2024-09-16T05:14:29.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 1 -- 172.21.15.114:0/3691769988 >> v2:172.21.15.149:3300/0 conn(0x7fb6480a19e0 msgr2=0x7fb64813cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:29.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 1 --2- 172.21.15.114:0/3691769988 >> v2:172.21.15.149:3300/0 conn(0x7fb6480a19e0 0x7fb64813cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:29.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 1 -- 172.21.15.114:0/3691769988 >> v1:172.21.15.114:6789/0 conn(0x7fb64809b6c0 legacy=0x7fb6480a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:29.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 10 monclient: _finish_hunting 0 2024-09-16T05:14:29.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 1 monclient: found mon.noname-a 2024-09-16T05:14:29.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:29.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:29.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 1 -- 172.21.15.114:0/3691769988 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb64809f430 con 0x7fb6480a15d0 2024-09-16T05:14:29.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb64c9bc640 10 monclient: get_auth_request con 0x7fb6480a19e0 auth_method 0 2024-09-16T05:14:29.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb64c9bc640 1 --2- 172.21.15.114:0/3691769988 >> v2:172.21.15.149:3300/0 conn(0x7fb6480a19e0 0x7fb64813cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:14:29.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb6477fe640 1 --2- 172.21.15.114:0/3691769988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6480a15d0 0x7fb64813a650 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb64809d220 tx=0x7fb63001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248df8d0b75d4690 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:29.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb646ffd640 1 -- 172.21.15.114:0/3691769988 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6300b2070 con 0x7fb6480a15d0 2024-09-16T05:14:29.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb646ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:29.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb646ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:29.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb646ffd640 10 monclient: dump: 2024-09-16T05:14:29.078 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:29.078 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:29.078 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:29.078 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:29.079 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:29.079 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:29.079 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:29.079 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:29.079 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:29.079 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:29.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb646ffd640 10 monclient: _finish_auth 0 2024-09-16T05:14:29.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb646ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:14:29.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb646ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:29.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb646ffd640 1 -- 172.21.15.114:0/3691769988 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6300a9070 con 0x7fb6480a15d0 2024-09-16T05:14:29.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb646ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:29.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb646ffd640 1 -- 172.21.15.114:0/3691769988 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6300ad880 con 0x7fb6480a15d0 2024-09-16T05:14:29.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb64e41f640 5 monclient: authenticate success, global_id 74139 2024-09-16T05:14:29.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb64e41f640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:29.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb64e41f640 4 set_mon_vals no callback set 2024-09-16T05:14:29.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb646ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:29.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb646ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:29.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb646ffd640 10 monclient: dump: 2024-09-16T05:14:29.080 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:29.080 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:29.080 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:29.080 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:29.080 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:29.080 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:29.080 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:29.080 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:29.080 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:29.081 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:29.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb64e41f640 10 monclient: shutdown 2024-09-16T05:14:29.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb64e41f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:29.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb64e41f640 1 -- 172.21.15.114:0/3691769988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6480a15d0 msgr2=0x7fb64813a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:29.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb64e41f640 1 --2- 172.21.15.114:0/3691769988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6480a15d0 0x7fb64813a650 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb64809d220 tx=0x7fb63001c470 comp rx=0 tx=0).stop 2024-09-16T05:14:29.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb64e41f640 1 -- 172.21.15.114:0/3691769988 shutdown_connections 2024-09-16T05:14:29.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb64e41f640 1 --2- 172.21.15.114:0/3691769988 >> v2:172.21.15.149:3300/0 conn(0x7fb6480a19e0 0x7fb64813cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:29.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb64e41f640 1 --2- 172.21.15.114:0/3691769988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6480a15d0 0x7fb64813a650 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:29.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb64e41f640 1 -- 172.21.15.114:0/3691769988 >> 172.21.15.114:0/3691769988 conn(0x7fb6480a1fb0 msgr2=0x7fb6480a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:29.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.064+0000 7fb64e41f640 1 -- 172.21.15.114:0/3691769988 shutdown_connections 2024-09-16T05:14:29.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 -- 172.21.15.114:0/3691769988 wait complete. 2024-09-16T05:14:29.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient: build_initial_monmap 2024-09-16T05:14:29.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient: monmap: 2024-09-16T05:14:29.082 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:29.082 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:29.082 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:29.082 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:29.082 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:29.082 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:29.082 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:29.082 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:29.082 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:29.082 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:29.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 librados: starting msgr at 2024-09-16T05:14:29.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 librados: starting objecter 2024-09-16T05:14:29.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 Processor -- start 2024-09-16T05:14:29.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 -- start start 2024-09-16T05:14:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 librados: setting wanted keys 2024-09-16T05:14:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 librados: calling monclient init 2024-09-16T05:14:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient: init 2024-09-16T05:14:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:14:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb64809b6c0 0x7fb648096bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient(hunting): picked mon.noname-a con 0x7fb64809b6c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6480a15d0 0x7fb6480994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient(hunting): picked mon.noname-c con 0x7fb6480a15d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6480a19e0 0x7fb648099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient(hunting): picked mon.noname-b con 0x7fb6480a19e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb64813f900 con 0x7fb64809b6c0 2024-09-16T05:14:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb64809cad0 con 0x7fb6480a19e0 2024-09-16T05:14:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb64813fa80 con 0x7fb6480a15d0 2024-09-16T05:14:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient(hunting): authenticate will time out at 1353.701416s 2024-09-16T05:14:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6480a15d0 0x7fb6480994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb647fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb64809b6c0 0x7fb648096bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6480a15d0 0x7fb6480994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:48674/0 (socket says 172.21.15.114:48674) 2024-09-16T05:14:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 1 -- 172.21.15.114:0/494506782 learned_addr learned my addr 172.21.15.114:0/494506782 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 10 monclient(hunting): get_auth_request con 0x7fb6480a15d0 auth_method 0 2024-09-16T05:14:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:29.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:29.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb647fff640 10 monclient(hunting): get_auth_request con 0x7fb64809b6c0 auth_method 0 2024-09-16T05:14:29.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:29.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb647fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:29.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb647fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:29.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:29.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:29.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:29.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb647fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:29.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb647fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:29.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb647fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:29.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64c9bc640 1 --2- 172.21.15.114:0/494506782 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6480a19e0 0x7fb648099a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:29.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64c9bc640 10 monclient(hunting): get_auth_request con 0x7fb6480a19e0 auth_method 0 2024-09-16T05:14:29.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64c9bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:29.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64c9bc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:29.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64c9bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:29.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 10 monclient(hunting): handle_auth_done global_id 54641 payload 824 2024-09-16T05:14:29.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 1 -- 172.21.15.114:0/494506782 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6480a19e0 msgr2=0x7fb648099a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:29.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 1 --2- 172.21.15.114:0/494506782 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6480a19e0 0x7fb648099a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:29.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 1 -- 172.21.15.114:0/494506782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb64809b6c0 msgr2=0x7fb648096bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:29.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 1 --2- 172.21.15.114:0/494506782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb64809b6c0 0x7fb648096bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:29.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 10 monclient: _finish_hunting 0 2024-09-16T05:14:29.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 1 monclient: found mon.noname-c 2024-09-16T05:14:29.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:29.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:14:29.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 1 -- 172.21.15.114:0/494506782 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6480973d0 con 0x7fb6480a15d0 2024-09-16T05:14:29.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb647fff640 1 --2- 172.21.15.114:0/494506782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb64809b6c0 0x7fb648096bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:14:29.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb6477fe640 1 --2- 172.21.15.114:0/494506782 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6480a15d0 0x7fb6480994e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb630019080 tx=0x7fb630019210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:29.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 10 client.?.objecter ms_handle_connect 0x7fb6480a15d0 2024-09-16T05:14:29.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:29.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 1 -- 172.21.15.114:0/494506782 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6300b2050 con 0x7fb6480a15d0 2024-09-16T05:14:29.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:29.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:14:29.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 10 monclient: dump: 2024-09-16T05:14:29.088 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:29.088 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:29.088 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:29.088 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:29.088 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:29.089 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:29.089 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:29.089 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:29.089 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:29.089 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:29.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 10 monclient: _finish_auth 0 2024-09-16T05:14:29.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:14:29.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:29.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 1 -- 172.21.15.114:0/494506782 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6300bc070 con 0x7fb6480a15d0 2024-09-16T05:14:29.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:29.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 1 -- 172.21.15.114:0/494506782 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb630003dd0 con 0x7fb6480a15d0 2024-09-16T05:14:29.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 5 monclient: authenticate success, global_id 54641 2024-09-16T05:14:29.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient: _renew_subs 2024-09-16T05:14:29.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 -- 172.21.15.114:0/494506782 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6480988d0 con 0x7fb6480a15d0 2024-09-16T05:14:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb645ffb640 4 set_mon_vals no callback set 2024-09-16T05:14:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 10 monclient: dump: 2024-09-16T05:14:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:29.091 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:29.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 client.54641.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:29.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient: _renew_subs 2024-09-16T05:14:29.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:29.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 -- 172.21.15.114:0/494506782 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb648072c50 con 0x7fb6480a15d0 2024-09-16T05:14:29.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 librados: init done 2024-09-16T05:14:29.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:29.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:29.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:29.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb64e41f640 1 -- 172.21.15.114:0/494506782 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb64806dfd0 con 0x7fb6480a15d0 2024-09-16T05:14:29.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 1 -- 172.21.15.114:0/494506782 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb6300ab070 con 0x7fb6480a15d0 2024-09-16T05:14:29.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 1 --2- 172.21.15.114:0/494506782 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb628037810 0x7fb628039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:29.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 1 -- 172.21.15.114:0/494506782 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(289..289 src has 1..289) ==== 5505+0+0 (secure 0 0 0) 0x7fb6300e2db0 con 0x7fb6480a15d0 2024-09-16T05:14:29.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 10 client.54641.objecter ms_dispatch 0x7fb648000ba0 osd_map(289..289 src has 1..289) 2024-09-16T05:14:29.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 3 client.54641.objecter handle_osd_map got epochs [289,289] > 0 2024-09-16T05:14:29.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 3 client.54641.objecter handle_osd_map decoding full epoch 289 2024-09-16T05:14:29.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb644ff9640 20 client.54641.objecter dump_active .. 0 homeless 2024-09-16T05:14:29.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb647fff640 1 --2- 172.21.15.114:0/494506782 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb628037810 0x7fb628039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:29.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.068+0000 7fb647fff640 10 monclient: get_auth_request con 0x7fb628037810 auth_method 0 2024-09-16T05:14:29.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.072+0000 7fb644ff9640 10 client.54641.objecter ms_handle_connect 0x7fb628037810 2024-09-16T05:14:29.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.072+0000 7fb647fff640 1 --2- 172.21.15.114:0/494506782 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb628037810 0x7fb628039cd0 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 crypto rx=0x7fb638001b60 tx=0x7fb638009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:29.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.076+0000 7fb644ff9640 1 -- 172.21.15.114:0/494506782 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb6300160f0 con 0x7fb6480a15d0 2024-09-16T05:14:29.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.076+0000 7fb644ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:29.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.076+0000 7fb644ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.296+0000 7fb64e41f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:14:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.296+0000 7fb64e41f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:14:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.296+0000 7fb64e41f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.296+0000 7fb64e41f640 1 -- 172.21.15.114:0/494506782 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fb648098b00 con 0x7fb6480a15d0 2024-09-16T05:14:29.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.300+0000 7fb644ff9640 1 -- 172.21.15.114:0/494506782 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fb6300c1840 con 0x7fb6480a15d0 2024-09-16T05:14:29.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.300+0000 7fb644ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:14:29.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.300+0000 7fb644ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:29.305 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.300+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.305 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.300+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=12,logm=61} crc {auth=3183603690,config=2586702516,health=2646929418,kv=2757161437,logm=3482200618}) 2024-09-16T05:14:29.305 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.300+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=12,logm=60} crc {auth=3183603690,config=2586702516,health=1995790572,kv=2757161437,logm=3245530284}) 2024-09-16T05:14:29.305 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.300+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.305 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.300+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=12,logm=61} crc {auth=3183603690,config=2586702516,health=2646929418,kv=2757161437,logm=3482200618}) 2024-09-16T05:14:29.305 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.300+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=1,health=12,kv=11,logm=64} crc {auth=722418564,config=1228700967,health=1995790572,kv=1569878674,logm=2803046409}) 2024-09-16T05:14:29.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.300+0000 7fb64e41f640 10 librados: watch_flush enter 2024-09-16T05:14:29.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.300+0000 7fb64e41f640 10 librados: watch_flush exit 2024-09-16T05:14:29.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.300+0000 7fb64e41f640 20 client.54641.objecter shutdown clearing up homeless session... 2024-09-16T05:14:29.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.300+0000 7fb64e41f640 10 client.54641.objecter successfully canceled tick 2024-09-16T05:14:29.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.300+0000 7fb64e41f640 1 -- 172.21.15.114:0/494506782 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb628037810 msgr2=0x7fb628039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:29.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.300+0000 7fb64e41f640 1 --2- 172.21.15.114:0/494506782 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb628037810 0x7fb628039cd0 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 crypto rx=0x7fb638001b60 tx=0x7fb638009040 comp rx=0 tx=0).stop 2024-09-16T05:14:29.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.300+0000 7fb64e41f640 10 monclient: shutdown 2024-09-16T05:14:29.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.300+0000 7fb64e41f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:29.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.300+0000 7fb64e41f640 1 -- 172.21.15.114:0/494506782 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6480a15d0 msgr2=0x7fb6480994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:29.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.300+0000 7fb64e41f640 1 --2- 172.21.15.114:0/494506782 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6480a15d0 0x7fb6480994e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb630019080 tx=0x7fb630019210 comp rx=0 tx=0).stop 2024-09-16T05:14:29.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.304+0000 7fb64e41f640 1 -- 172.21.15.114:0/494506782 shutdown_connections 2024-09-16T05:14:29.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.304+0000 7fb64e41f640 1 --2- 172.21.15.114:0/494506782 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb628037810 0x7fb628039cd0 unknown :-1 s=CLOSED pgs=3499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:29.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.304+0000 7fb64e41f640 1 --2- 172.21.15.114:0/494506782 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6480a19e0 0x7fb648099a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:29.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.304+0000 7fb64e41f640 1 --2- 172.21.15.114:0/494506782 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6480a15d0 0x7fb6480994e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:29.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.304+0000 7fb64e41f640 1 --2- 172.21.15.114:0/494506782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb64809b6c0 0x7fb648096bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:29.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.304+0000 7fb64e41f640 1 -- 172.21.15.114:0/494506782 >> 172.21.15.114:0/494506782 conn(0x7fb6480a1fb0 msgr2=0x7fb648073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:29.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.304+0000 7fb64e41f640 1 -- 172.21.15.114:0/494506782 shutdown_connections 2024-09-16T05:14:29.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.304+0000 7fb64e41f640 1 -- 172.21.15.114:0/494506782 wait complete. 2024-09-16T05:14:29.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:29.304+0000 7fb64e41f640 1 librados: shutdown 2024-09-16T05:14:29.308 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.304+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.309 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.304+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063721675}) 2024-09-16T05:14:29.309 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.304+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4045820103}) 2024-09-16T05:14:29.309 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.304+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.309 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.304+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063721675}) 2024-09-16T05:14:29.309 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.304+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2304774380}) 2024-09-16T05:14:29.309 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.304+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.309 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.304+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1058499986}) 2024-09-16T05:14:29.310 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.304+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1695672753}) 2024-09-16T05:14:29.310 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.304+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.310 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.304+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1058499986}) 2024-09-16T05:14:29.310 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.304+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497275421}) 2024-09-16T05:14:29.311 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.308+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.311 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.308+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3348527791}) 2024-09-16T05:14:29.311 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.308+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3786177713}) 2024-09-16T05:14:29.311 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.308+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.311 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.308+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3348527791}) 2024-09-16T05:14:29.311 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.308+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2866528015}) 2024-09-16T05:14:29.313 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.308+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.313 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.308+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2240938215}) 2024-09-16T05:14:29.313 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.308+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1061582401}) 2024-09-16T05:14:29.313 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.308+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.313 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.308+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2240938215}) 2024-09-16T05:14:29.313 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.308+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1241837925}) 2024-09-16T05:14:29.314 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.314 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1172629850}) 2024-09-16T05:14:29.314 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3073728848}) 2024-09-16T05:14:29.314 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.314 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1172629850}) 2024-09-16T05:14:29.315 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=966942691}) 2024-09-16T05:14:29.316 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.316 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1164031047}) 2024-09-16T05:14:29.316 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3931114659}) 2024-09-16T05:14:29.316 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.316 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1164031047}) 2024-09-16T05:14:29.316 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2436905561}) 2024-09-16T05:14:29.318 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.318 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285103274}) 2024-09-16T05:14:29.318 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1027611326}) 2024-09-16T05:14:29.318 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.318 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285103274}) 2024-09-16T05:14:29.318 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.312+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282429657}) 2024-09-16T05:14:29.319 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.316+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.319 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.316+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307082953}) 2024-09-16T05:14:29.319 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.316+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4243625935}) 2024-09-16T05:14:29.319 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.316+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.319 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.316+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307082953}) 2024-09-16T05:14:29.320 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.316+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3183647716}) 2024-09-16T05:14:29.321 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.316+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.321 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.316+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159271460}) 2024-09-16T05:14:29.321 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.316+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3934892208}) 2024-09-16T05:14:29.321 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.316+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.321 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.316+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159271460}) 2024-09-16T05:14:29.321 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.316+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3520083280}) 2024-09-16T05:14:29.322 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:14:29.323 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.320+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.323 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.320+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4237957718}) 2024-09-16T05:14:29.323 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.320+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2817630368}) 2024-09-16T05:14:29.323 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.320+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.323 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.320+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4237957718}) 2024-09-16T05:14:29.323 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.320+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3107144571}) 2024-09-16T05:14:29.324 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.320+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.324 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.320+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3070895611}) 2024-09-16T05:14:29.324 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.320+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3459121330}) 2024-09-16T05:14:29.325 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.320+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.325 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.320+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3070895611}) 2024-09-16T05:14:29.325 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.320+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1646238424}) 2024-09-16T05:14:29.326 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.326 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412452712}) 2024-09-16T05:14:29.326 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3186697158}) 2024-09-16T05:14:29.326 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.326 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412452712}) 2024-09-16T05:14:29.326 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4045893901}) 2024-09-16T05:14:29.328 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.328 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1584318487}) 2024-09-16T05:14:29.328 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1848711854}) 2024-09-16T05:14:29.328 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.328 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1584318487}) 2024-09-16T05:14:29.328 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2020636098}) 2024-09-16T05:14:29.330 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.330 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615095097}) 2024-09-16T05:14:29.330 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1940055995}) 2024-09-16T05:14:29.330 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.330 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615095097}) 2024-09-16T05:14:29.330 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.324+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=274297545}) 2024-09-16T05:14:29.331 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.328+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.332 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.328+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=736010565}) 2024-09-16T05:14:29.332 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.328+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1312911667}) 2024-09-16T05:14:29.332 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.328+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.332 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.328+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=736010565}) 2024-09-16T05:14:29.332 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.328+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=143866137}) 2024-09-16T05:14:29.333 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.328+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.334 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.328+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=421765550}) 2024-09-16T05:14:29.334 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.328+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1575606169}) 2024-09-16T05:14:29.334 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.328+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.334 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.328+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=421765550}) 2024-09-16T05:14:29.334 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.328+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2575688220}) 2024-09-16T05:14:29.335 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.332+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.336 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.332+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19107601}) 2024-09-16T05:14:29.336 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.332+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4235433159}) 2024-09-16T05:14:29.336 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.332+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.336 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.332+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19107601}) 2024-09-16T05:14:29.336 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.332+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4012321559}) 2024-09-16T05:14:29.337 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.332+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.337 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.332+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2309328598}) 2024-09-16T05:14:29.337 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.332+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=358332594}) 2024-09-16T05:14:29.337 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.332+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.337 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.332+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2309328598}) 2024-09-16T05:14:29.337 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.332+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2261912549}) 2024-09-16T05:14:29.338 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.336+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.338 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.336+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4059676623}) 2024-09-16T05:14:29.338 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.336+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3909291019}) 2024-09-16T05:14:29.339 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.336+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.339 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.336+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4059676623}) 2024-09-16T05:14:29.339 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.336+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1601213223}) 2024-09-16T05:14:29.340 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.336+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.340 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.336+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703050684}) 2024-09-16T05:14:29.340 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.336+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1639046991}) 2024-09-16T05:14:29.340 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.336+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.340 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.336+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703050684}) 2024-09-16T05:14:29.341 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.336+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2757360672}) 2024-09-16T05:14:29.342 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.336+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.342 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3989650635}) 2024-09-16T05:14:29.342 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2940414475}) 2024-09-16T05:14:29.342 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.342 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3989650635}) 2024-09-16T05:14:29.342 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3989205017}) 2024-09-16T05:14:29.344 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.344 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038721338}) 2024-09-16T05:14:29.344 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1222387144}) 2024-09-16T05:14:29.344 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.344 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038721338}) 2024-09-16T05:14:29.344 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1401827042}) 2024-09-16T05:14:29.345 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.346 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2310915206}) 2024-09-16T05:14:29.346 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=478679003}) 2024-09-16T05:14:29.346 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.346 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2310915206}) 2024-09-16T05:14:29.346 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.340+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1364890005}) 2024-09-16T05:14:29.347 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.344+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.347 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.344+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4208040988}) 2024-09-16T05:14:29.347 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.344+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4055626184}) 2024-09-16T05:14:29.347 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.344+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.348 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.344+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4208040988}) 2024-09-16T05:14:29.348 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.344+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1629949933}) 2024-09-16T05:14:29.349 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.344+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.349 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.344+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=639547546}) 2024-09-16T05:14:29.349 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.344+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3429848833}) 2024-09-16T05:14:29.349 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.344+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.349 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.344+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=639547546}) 2024-09-16T05:14:29.350 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.344+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=482137348}) 2024-09-16T05:14:29.351 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.348+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.351 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.348+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=445228386}) 2024-09-16T05:14:29.351 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.348+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2423611737}) 2024-09-16T05:14:29.351 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.348+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.351 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.348+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=445228386}) 2024-09-16T05:14:29.351 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.348+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2207907984}) 2024-09-16T05:14:29.353 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.348+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.353 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.348+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329808003}) 2024-09-16T05:14:29.353 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.348+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2311817551}) 2024-09-16T05:14:29.353 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.348+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.353 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.348+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329808003}) 2024-09-16T05:14:29.353 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.348+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3062770375}) 2024-09-16T05:14:29.355 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.352+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.355 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.352+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311309342}) 2024-09-16T05:14:29.355 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.352+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2309471120}) 2024-09-16T05:14:29.355 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.352+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.355 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.352+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311309342}) 2024-09-16T05:14:29.355 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.352+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2701992718}) 2024-09-16T05:14:29.357 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.352+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.357 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.352+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149586578}) 2024-09-16T05:14:29.357 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.352+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1285432494}) 2024-09-16T05:14:29.357 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.352+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.357 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.352+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149586578}) 2024-09-16T05:14:29.357 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.352+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=179804322}) 2024-09-16T05:14:29.358 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.356+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.359 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.356+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334386130}) 2024-09-16T05:14:29.359 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.356+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3933588950}) 2024-09-16T05:14:29.359 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.356+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.359 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.356+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334386130}) 2024-09-16T05:14:29.359 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.356+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610150950}) 2024-09-16T05:14:29.360 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.356+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.360 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.356+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098545869}) 2024-09-16T05:14:29.360 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.356+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3995361013}) 2024-09-16T05:14:29.361 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.356+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.361 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.356+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098545869}) 2024-09-16T05:14:29.361 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.356+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2961111354}) 2024-09-16T05:14:29.362 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.360+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.362 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.360+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070821529}) 2024-09-16T05:14:29.362 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.360+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=866853937}) 2024-09-16T05:14:29.362 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.360+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.362 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.360+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070821529}) 2024-09-16T05:14:29.362 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.360+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1214944402}) 2024-09-16T05:14:29.364 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.360+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.364 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.360+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=44,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=15,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=7} crc {logm=2880572103,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=3117416403,mon_config_key=1444020154,monmap=3371862038,osd_metadata=699678260,osd_pg_creating=1677805431,osd_snap=17429743}) 2024-09-16T05:14:29.365 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.360+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=3} crc {logm=1272024625,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=202446750,mon_config_key=3107880615,monmap=3045472550,osd_metadata=487805116,osd_pg_creating=1677805431,osd_snap=3734492510}) 2024-09-16T05:14:29.365 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.360+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.365 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.360+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=44,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=15,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=7} crc {logm=2880572103,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=3117416403,mon_config_key=1444020154,monmap=3371862038,osd_metadata=699678260,osd_pg_creating=1677805431,osd_snap=17429743}) 2024-09-16T05:14:29.365 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.360+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=45,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=6} crc {logm=1288392766,mdsmap=3140514009,mgr=3496963760,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=3117416403,mon_config_key=3241792257,monmap=3045472550,osd_metadata=39865867,osd_pg_creating=1677805431,osd_snap=699840441}) 2024-09-16T05:14:29.367 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.364+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.367 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.364+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=20,osdmap=80} crc {osd_snap=2807149361,osdmap=4166262191}) 2024-09-16T05:14:29.367 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.364+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=20,osdmap=80} crc {osd_snap=3936335937,osdmap=1373732680}) 2024-09-16T05:14:29.367 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.364+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.367 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.364+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=20,osdmap=80} crc {osd_snap=2807149361,osdmap=4166262191}) 2024-09-16T05:14:29.367 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.364+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=2292209064,osdmap=962559678}) 2024-09-16T05:14:29.369 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.364+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.369 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.364+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2373265037}) 2024-09-16T05:14:29.369 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.364+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4032785508}) 2024-09-16T05:14:29.369 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.364+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.369 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.364+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2373265037}) 2024-09-16T05:14:29.369 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.364+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2045482331}) 2024-09-16T05:14:29.371 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.368+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.371 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.368+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3271393586}) 2024-09-16T05:14:29.371 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.368+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2711066498}) 2024-09-16T05:14:29.371 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.368+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.372 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.368+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3271393586}) 2024-09-16T05:14:29.372 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.368+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3093753783}) 2024-09-16T05:14:29.375 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.372+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.375 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.372+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2469784968}) 2024-09-16T05:14:29.375 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.372+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2569331429}) 2024-09-16T05:14:29.376 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.372+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.376 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.372+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2469784968}) 2024-09-16T05:14:29.376 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.372+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=417443579}) 2024-09-16T05:14:29.378 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.372+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.378 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.372+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=46} crc {osdmap=2798279030}) 2024-09-16T05:14:29.378 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.372+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=45} crc {osdmap=3536113028}) 2024-09-16T05:14:29.378 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.372+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:14:29.378 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.372+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=46} crc {osdmap=2798279030}) 2024-09-16T05:14:29.378 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:14:29.372+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=45} crc {osdmap=2175695996}) 2024-09-16T05:14:30.323 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:14:30.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:14:30.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 10 monclient: get_monmap_and_config 2024-09-16T05:14:30.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 10 monclient: build_initial_monmap 2024-09-16T05:14:30.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 10 monclient: monmap: 2024-09-16T05:14:30.452 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:30.452 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:30.452 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:30.453968+0000 2024-09-16T05:14:30.452 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:30.453968+0000 2024-09-16T05:14:30.452 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:30.452 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:30.452 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:30.452 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:30.453 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:30.453 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:30.453 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:30.453 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:30.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 1 Processor -- start 2024-09-16T05:14:30.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 1 -- start start 2024-09-16T05:14:30.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 10 monclient: init 2024-09-16T05:14:30.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:30.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T05:14:30.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdfc809b6c0 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:30.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdfc80a15d0 0x7fdfc813a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:30.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 10 monclient(hunting): picked mon.noname-a con 0x7fdfc80a15d0 addr v2:172.21.15.114:3300/0 2024-09-16T05:14:30.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fdfc80a19e0 0x7fdfc813cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:30.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 10 monclient(hunting): picked mon.noname-c con 0x7fdfc80a19e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:14:30.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfc809d4e0 con 0x7fdfc809b6c0 2024-09-16T05:14:30.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:30.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fdfc809e390 con 0x7fdfc80a15d0 2024-09-16T05:14:30.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:30.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fdfc809ece0 con 0x7fdfc80a19e0 2024-09-16T05:14:30.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:30.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcfc5e640 10 monclient(hunting): authenticate will time out at 1355.083156s 2024-09-16T05:14:30.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdfc80a15d0 0x7fdfc813a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:30.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdfc80a15d0 0x7fdfc813a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36756/0 (socket says 172.21.15.114:36756) 2024-09-16T05:14:30.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 1 -- 172.21.15.114:0/2071547769 learned_addr learned my addr 172.21.15.114:0/2071547769 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:30.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 10 monclient(hunting): get_auth_request con 0x7fdfc80a15d0 auth_method 0 2024-09-16T05:14:30.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:30.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:30.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:30.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:30.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:30.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:30.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfce1fb640 1 --2- 172.21.15.114:0/2071547769 >> v2:172.21.15.149:3300/0 conn(0x7fdfc80a19e0 0x7fdfc813cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:30.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfce1fb640 10 monclient(hunting): get_auth_request con 0x7fdfc80a19e0 auth_method 0 2024-09-16T05:14:30.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfce1fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:30.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfce1fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:30.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfce1fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:30.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 10 monclient(hunting): handle_auth_done global_id 74151 payload 274 2024-09-16T05:14:30.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 1 -- 172.21.15.114:0/2071547769 >> v2:172.21.15.149:3300/0 conn(0x7fdfc80a19e0 msgr2=0x7fdfc813cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:30.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 1 --2- 172.21.15.114:0/2071547769 >> v2:172.21.15.149:3300/0 conn(0x7fdfc80a19e0 0x7fdfc813cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 1 -- 172.21.15.114:0/2071547769 >> v1:172.21.15.114:6789/0 conn(0x7fdfc809b6c0 legacy=0x7fdfc80a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:30.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 10 monclient: _finish_hunting 0 2024-09-16T05:14:30.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 1 monclient: found mon.noname-a 2024-09-16T05:14:30.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:30.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:30.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.448+0000 7fdfcd1f9640 1 -- 172.21.15.114:0/2071547769 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc809f430 con 0x7fdfc80a15d0 2024-09-16T05:14:30.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcd1f9640 1 --2- 172.21.15.114:0/2071547769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfc80a15d0 0x7fdfc813a650 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdfc809d220 tx=0x7fdfc401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f477e70205e2997a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:30.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcc9f8640 1 -- 172.21.15.114:0/2071547769 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc40b2070 con 0x7fdfc80a15d0 2024-09-16T05:14:30.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcc9f8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:30.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcc9f8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:30.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcc9f8640 10 monclient: dump: 2024-09-16T05:14:30.464 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:30.464 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:30.465 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcc9f8640 10 monclient: _finish_auth 0 2024-09-16T05:14:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcc9f8640 10 monclient: _check_auth_tickets 2024-09-16T05:14:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcc9f8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 5 monclient: authenticate success, global_id 74151 2024-09-16T05:14:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcc9f8640 1 -- 172.21.15.114:0/2071547769 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfc40a9070 con 0x7fdfc80a15d0 2024-09-16T05:14:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcc9f8640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 4 set_mon_vals no callback set 2024-09-16T05:14:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcc9f8640 1 -- 172.21.15.114:0/2071547769 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc40ad880 con 0x7fdfc80a15d0 2024-09-16T05:14:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcc9f8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcc9f8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcc9f8640 10 monclient: dump: 2024-09-16T05:14:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:30.467 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 10 monclient: shutdown 2024-09-16T05:14:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 1 -- 172.21.15.114:0/2071547769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfc80a15d0 msgr2=0x7fdfc813a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 1 --2- 172.21.15.114:0/2071547769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfc80a15d0 0x7fdfc813a650 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdfc809d220 tx=0x7fdfc401c470 comp rx=0 tx=0).stop 2024-09-16T05:14:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 1 -- 172.21.15.114:0/2071547769 shutdown_connections 2024-09-16T05:14:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 1 --2- 172.21.15.114:0/2071547769 >> v2:172.21.15.149:3300/0 conn(0x7fdfc80a19e0 0x7fdfc813cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 1 --2- 172.21.15.114:0/2071547769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfc80a15d0 0x7fdfc813a650 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 1 -- 172.21.15.114:0/2071547769 >> 172.21.15.114:0/2071547769 conn(0x7fdfc80a1fb0 msgr2=0x7fdfc80a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:30.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 1 -- 172.21.15.114:0/2071547769 shutdown_connections 2024-09-16T05:14:30.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 1 -- 172.21.15.114:0/2071547769 wait complete. 2024-09-16T05:14:30.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 10 monclient: build_initial_monmap 2024-09-16T05:14:30.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 10 monclient: monmap: 2024-09-16T05:14:30.468 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:30.468 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:30.468 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:30.468 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:30.468 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:30.468 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:30.468 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:30.468 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:30.468 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:30.469 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:30.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 1 librados: starting msgr at 2024-09-16T05:14:30.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 1 librados: starting objecter 2024-09-16T05:14:30.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 1 Processor -- start 2024-09-16T05:14:30.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 1 -- start start 2024-09-16T05:14:30.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 1 librados: setting wanted keys 2024-09-16T05:14:30.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 1 librados: calling monclient init 2024-09-16T05:14:30.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.452+0000 7fdfcfc5e640 10 monclient: init 2024-09-16T05:14:30.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:30.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:14:30.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdfc809b6c0 0x7fdfc8096b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:30.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient(hunting): picked mon.noname-b con 0x7fdfc809b6c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:30.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfc80a15d0 0x7fdfc8093ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:30.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient(hunting): picked mon.noname-a con 0x7fdfc80a15d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:30.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdfc80a19e0 0x7fdfc8094220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:30.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient(hunting): picked mon.noname-c con 0x7fdfc80a19e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:30.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:30.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc813f900 con 0x7fdfc80a15d0 2024-09-16T05:14:30.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:30.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc809cad0 con 0x7fdfc809b6c0 2024-09-16T05:14:30.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:30.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fdfc813fa80 con 0x7fdfc80a19e0 2024-09-16T05:14:30.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:30.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient(hunting): authenticate will time out at 1355.088321s 2024-09-16T05:14:30.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfc80a15d0 0x7fdfc8093ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:30.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfc80a15d0 0x7fdfc8093ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36770/0 (socket says 172.21.15.114:36770) 2024-09-16T05:14:30.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 1 -- 172.21.15.114:0/3992543090 learned_addr learned my addr 172.21.15.114:0/3992543090 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:30.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 10 monclient(hunting): get_auth_request con 0x7fdfc80a15d0 auth_method 0 2024-09-16T05:14:30.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:30.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:30.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:30.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfce1fb640 1 --2- 172.21.15.114:0/3992543090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdfc80a19e0 0x7fdfc8094220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:30.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:30.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:30.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:30.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfce1fb640 10 monclient(hunting): get_auth_request con 0x7fdfc80a19e0 auth_method 0 2024-09-16T05:14:30.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfce1fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:30.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfce1fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:30.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfce1fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:30.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfce1fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:30.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfce1fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:30.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd9fa640 1 --2- 172.21.15.114:0/3992543090 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdfc809b6c0 0x7fdfc8096b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:30.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfce1fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:30.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd9fa640 10 monclient(hunting): get_auth_request con 0x7fdfc809b6c0 auth_method 0 2024-09-16T05:14:30.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd9fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:30.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd9fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:30.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd9fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:30.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 10 monclient(hunting): handle_auth_done global_id 74157 payload 824 2024-09-16T05:14:30.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 1 -- 172.21.15.114:0/3992543090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdfc80a19e0 msgr2=0x7fdfc8094220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:30.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 1 --2- 172.21.15.114:0/3992543090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdfc80a19e0 0x7fdfc8094220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 1 -- 172.21.15.114:0/3992543090 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdfc809b6c0 msgr2=0x7fdfc8096b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:30.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 1 --2- 172.21.15.114:0/3992543090 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdfc809b6c0 0x7fdfc8096b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 10 monclient: _finish_hunting 0 2024-09-16T05:14:30.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 1 monclient: found mon.noname-a 2024-09-16T05:14:30.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:30.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:30.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 1 -- 172.21.15.114:0/3992543090 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc8097090 con 0x7fdfc80a15d0 2024-09-16T05:14:30.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd1f9640 1 --2- 172.21.15.114:0/3992543090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfc80a15d0 0x7fdfc8093ce0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdfc401a0a0 tx=0x7fdfc4019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:30.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 10 client.?.objecter ms_handle_connect 0x7fdfc80a15d0 2024-09-16T05:14:30.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:30.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 1 -- 172.21.15.114:0/3992543090 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc40bb020 con 0x7fdfc80a15d0 2024-09-16T05:14:30.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:30.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:30.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 10 monclient: dump: 2024-09-16T05:14:30.474 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:30.474 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:30.474 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.474 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.474 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:30.474 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:30.474 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:30.474 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:30.474 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:30.475 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:30.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 10 monclient: _finish_auth 0 2024-09-16T05:14:30.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:14:30.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:30.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 1 -- 172.21.15.114:0/3992543090 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfc40b3040 con 0x7fdfc80a15d0 2024-09-16T05:14:30.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:30.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 1 -- 172.21.15.114:0/3992543090 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc40b2050 con 0x7fdfc80a15d0 2024-09-16T05:14:30.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:30.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:30.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 10 monclient: dump: 2024-09-16T05:14:30.475 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:30.475 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:30.475 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.476 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.476 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:30.476 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:30.476 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:30.476 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:30.476 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:30.476 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:30.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbffff640 4 set_mon_vals no callback set 2024-09-16T05:14:30.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 5 monclient: authenticate success, global_id 74157 2024-09-16T05:14:30.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient: _renew_subs 2024-09-16T05:14:30.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:30.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 1 -- 172.21.15.114:0/3992543090 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfc8098b00 con 0x7fdfc80a15d0 2024-09-16T05:14:30.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 client.74157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:30.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient: _renew_subs 2024-09-16T05:14:30.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:30.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 1 -- 172.21.15.114:0/3992543090 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfc809c6a0 con 0x7fdfc80a15d0 2024-09-16T05:14:30.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 1 librados: init done 2024-09-16T05:14:30.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 1 -- 172.21.15.114:0/3992543090 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fdfc40be070 con 0x7fdfc80a15d0 2024-09-16T05:14:30.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:30.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:30.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:30.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcfc5e640 1 -- 172.21.15.114:0/3992543090 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfc809aa90 con 0x7fdfc80a15d0 2024-09-16T05:14:30.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 1 --2- 172.21.15.114:0/3992543090 >> v2:172.21.15.149:6812/1185531135 conn(0x7fdfac037810 0x7fdfac039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:30.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 1 -- 172.21.15.114:0/3992543090 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(290..290 src has 1..290) ==== 5398+0+0 (secure 0 0 0) 0x7fdfc40e2230 con 0x7fdfc80a15d0 2024-09-16T05:14:30.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 10 client.74157.objecter ms_dispatch 0x7fdfc8000ba0 osd_map(290..290 src has 1..290) 2024-09-16T05:14:30.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 3 client.74157.objecter handle_osd_map got epochs [290,290] > 0 2024-09-16T05:14:30.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 3 client.74157.objecter handle_osd_map decoding full epoch 290 2024-09-16T05:14:30.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfbe7fc640 20 client.74157.objecter dump_active .. 0 homeless 2024-09-16T05:14:30.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd9fa640 1 --2- 172.21.15.114:0/3992543090 >> v2:172.21.15.149:6812/1185531135 conn(0x7fdfac037810 0x7fdfac039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:30.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.456+0000 7fdfcd9fa640 10 monclient: get_auth_request con 0x7fdfac037810 auth_method 0 2024-09-16T05:14:30.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.460+0000 7fdfcd9fa640 1 --2- 172.21.15.114:0/3992543090 >> v2:172.21.15.149:6812/1185531135 conn(0x7fdfac037810 0x7fdfac039cd0 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7fdfb80015b0 tx=0x7fdfb8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:30.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.460+0000 7fdfbe7fc640 10 client.74157.objecter ms_handle_connect 0x7fdfac037810 2024-09-16T05:14:30.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.460+0000 7fdfbe7fc640 1 -- 172.21.15.114:0/3992543090 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fdfc40160f0 con 0x7fdfc80a15d0 2024-09-16T05:14:30.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.460+0000 7fdfbe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:30.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.460+0000 7fdfbe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.692+0000 7fdfcfc5e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:14:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.692+0000 7fdfcfc5e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.692+0000 7fdfcfc5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.692+0000 7fdfcfc5e640 1 -- 172.21.15.114:0/3992543090 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdfc8096260 con 0x7fdfc80a15d0 2024-09-16T05:14:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.692+0000 7fdfbe7fc640 1 -- 172.21.15.114:0/3992543090 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fdfc40ad460 con 0x7fdfc80a15d0 2024-09-16T05:14:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.692+0000 7fdfbe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.692+0000 7fdfbe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:30.698 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:14:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 10 librados: watch_flush enter 2024-09-16T05:14:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 10 librados: watch_flush exit 2024-09-16T05:14:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 20 client.74157.objecter shutdown clearing up homeless session... 2024-09-16T05:14:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 10 client.74157.objecter successfully canceled tick 2024-09-16T05:14:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 1 -- 172.21.15.114:0/3992543090 >> v2:172.21.15.149:6812/1185531135 conn(0x7fdfac037810 msgr2=0x7fdfac039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 1 --2- 172.21.15.114:0/3992543090 >> v2:172.21.15.149:6812/1185531135 conn(0x7fdfac037810 0x7fdfac039cd0 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7fdfb80015b0 tx=0x7fdfb8009040 comp rx=0 tx=0).stop 2024-09-16T05:14:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 10 monclient: shutdown 2024-09-16T05:14:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 1 -- 172.21.15.114:0/3992543090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfc80a15d0 msgr2=0x7fdfc8093ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 1 --2- 172.21.15.114:0/3992543090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfc80a15d0 0x7fdfc8093ce0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdfc401a0a0 tx=0x7fdfc4019210 comp rx=0 tx=0).stop 2024-09-16T05:14:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 1 -- 172.21.15.114:0/3992543090 shutdown_connections 2024-09-16T05:14:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 1 --2- 172.21.15.114:0/3992543090 >> v2:172.21.15.149:6812/1185531135 conn(0x7fdfac037810 0x7fdfac039cd0 unknown :-1 s=CLOSED pgs=3501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 1 --2- 172.21.15.114:0/3992543090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfc80a15d0 0x7fdfc8093ce0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 1 --2- 172.21.15.114:0/3992543090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdfc80a19e0 0x7fdfc8094220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 1 --2- 172.21.15.114:0/3992543090 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdfc809b6c0 0x7fdfc8096b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 1 -- 172.21.15.114:0/3992543090 >> 172.21.15.114:0/3992543090 conn(0x7fdfc80a1fb0 msgr2=0x7fdfc80994b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 1 -- 172.21.15.114:0/3992543090 shutdown_connections 2024-09-16T05:14:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 1 -- 172.21.15.114:0/3992543090 wait complete. 2024-09-16T05:14:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.696+0000 7fdfcfc5e640 1 librados: shutdown 2024-09-16T05:14:30.715 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:14:30.716 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:14:30.716 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:14:30.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:14:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 10 monclient: get_monmap_and_config 2024-09-16T05:14:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 10 monclient: build_initial_monmap 2024-09-16T05:14:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 10 monclient: monmap: 2024-09-16T05:14:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:30.838 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:30.840418+0000 2024-09-16T05:14:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:30.840418+0000 2024-09-16T05:14:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:30.839 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:30.839 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:30.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 1 Processor -- start 2024-09-16T05:14:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 1 -- start start 2024-09-16T05:14:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 10 monclient: init 2024-09-16T05:14:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-16T05:14:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9a181376e0 0x7f9a18137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 10 monclient(hunting): picked mon.noname-a con 0x7f9a181376e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:14:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a181370a0 0x7f9a180a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 10 monclient(hunting): picked mon.noname-b con 0x7f9a181370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a180a5d00 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a1809bd20 con 0x7f9a180a5d00 2024-09-16T05:14:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:30.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f9a18138000 con 0x7f9a181376e0 2024-09-16T05:14:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9a1809c5c0 con 0x7f9a181370a0 2024-09-16T05:14:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1fc28640 10 monclient(hunting): authenticate will time out at 1355.469660s 2024-09-16T05:14:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9a181376e0 0x7f9a18137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9a181376e0 0x7f9a18137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36780/0 (socket says 172.21.15.114:36780) 2024-09-16T05:14:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 1 -- 172.21.15.114:0/2693234177 learned_addr learned my addr 172.21.15.114:0/2693234177 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 10 monclient(hunting): get_auth_request con 0x7f9a181376e0 auth_method 0 2024-09-16T05:14:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:30.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d1c3640 1 --2- 172.21.15.114:0/2693234177 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a181370a0 0x7f9a180a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:30.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:30.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:30.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:30.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d1c3640 10 monclient(hunting): get_auth_request con 0x7f9a181370a0 auth_method 0 2024-09-16T05:14:30.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d1c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:30.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d1c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:30.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d1c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:30.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d1c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:30.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d1c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:30.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d1c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:30.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 10 monclient(hunting): handle_auth_done global_id 74160 payload 274 2024-09-16T05:14:30.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 1 -- 172.21.15.114:0/2693234177 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a181370a0 msgr2=0x7f9a180a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:30.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 1 --2- 172.21.15.114:0/2693234177 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a181370a0 0x7f9a180a4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 1 -- 172.21.15.114:0/2693234177 >> v1:172.21.15.114:6789/0 conn(0x7f9a180a5d00 legacy=0x7f9a180a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 10 monclient: _finish_hunting 0 2024-09-16T05:14:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 1 monclient: found mon.noname-a 2024-09-16T05:14:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.836+0000 7f9a1d9c4640 1 -- 172.21.15.114:0/2693234177 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1809c000 con 0x7f9a181376e0 2024-09-16T05:14:30.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 1 --2- 172.21.15.114:0/2693234177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a181376e0 0x7f9a18137ac0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a08009870 tx=0x7f9a0801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed3c4e826fc011c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:30.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1c9c2640 1 -- 172.21.15.114:0/2693234177 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a080b0070 con 0x7f9a181376e0 2024-09-16T05:14:30.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1c9c2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:30.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1c9c2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:30.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1c9c2640 10 monclient: dump: 2024-09-16T05:14:30.852 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:30.852 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:30.852 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.852 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.852 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:30.852 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:30.852 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:30.852 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:30.852 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:30.853 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:30.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1c9c2640 10 monclient: _finish_auth 0 2024-09-16T05:14:30.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1c9c2640 10 monclient: _check_auth_tickets 2024-09-16T05:14:30.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1c9c2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:30.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1c9c2640 1 -- 172.21.15.114:0/2693234177 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a080ab600 con 0x7f9a181376e0 2024-09-16T05:14:30.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 5 monclient: authenticate success, global_id 74160 2024-09-16T05:14:30.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:30.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1c9c2640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:30.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1c9c2640 1 -- 172.21.15.114:0/2693234177 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a080ab9d0 con 0x7f9a181376e0 2024-09-16T05:14:30.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1c9c2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:30.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1c9c2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:30.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1c9c2640 10 monclient: dump: 2024-09-16T05:14:30.854 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:30.854 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:30.854 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.854 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.854 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:30.854 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:30.854 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:30.854 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:30.854 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:30.854 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:30.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:30.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 4 set_mon_vals no callback set 2024-09-16T05:14:30.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient: shutdown 2024-09-16T05:14:30.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:30.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 -- 172.21.15.114:0/2693234177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a181376e0 msgr2=0x7f9a18137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:30.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 --2- 172.21.15.114:0/2693234177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a181376e0 0x7f9a18137ac0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a08009870 tx=0x7f9a0801c420 comp rx=0 tx=0).stop 2024-09-16T05:14:30.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 -- 172.21.15.114:0/2693234177 shutdown_connections 2024-09-16T05:14:30.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 --2- 172.21.15.114:0/2693234177 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a181370a0 0x7f9a180a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 --2- 172.21.15.114:0/2693234177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a181376e0 0x7f9a18137ac0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 -- 172.21.15.114:0/2693234177 >> 172.21.15.114:0/2693234177 conn(0x7f9a180a9a40 msgr2=0x7f9a180a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:30.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 -- 172.21.15.114:0/2693234177 shutdown_connections 2024-09-16T05:14:30.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 -- 172.21.15.114:0/2693234177 wait complete. 2024-09-16T05:14:30.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient: build_initial_monmap 2024-09-16T05:14:30.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient: monmap: 2024-09-16T05:14:30.855 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:30.855 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:30.855 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:30.856 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:30.856 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:30.856 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:30.856 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:30.856 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:30.856 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:30.856 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:30.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 librados: starting msgr at 2024-09-16T05:14:30.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 librados: starting objecter 2024-09-16T05:14:30.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 Processor -- start 2024-09-16T05:14:30.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 -- start start 2024-09-16T05:14:30.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 librados: setting wanted keys 2024-09-16T05:14:30.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 librados: calling monclient init 2024-09-16T05:14:30.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient: init 2024-09-16T05:14:30.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:30.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:14:30.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a180a5d00 0x7f9a181cf090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:30.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient(hunting): picked mon.noname-c con 0x7f9a180a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:30.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9a181370a0 0x7f9a181361b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:30.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient(hunting): picked mon.noname-b con 0x7f9a181370a0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:30.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a181376e0 0x7f9a1813c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:30.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient(hunting): picked mon.noname-a con 0x7f9a181376e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:30.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:30.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a18141d80 con 0x7f9a181376e0 2024-09-16T05:14:30.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:30.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f9a18141c00 con 0x7f9a181370a0 2024-09-16T05:14:30.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:30.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9a18141f00 con 0x7f9a180a5d00 2024-09-16T05:14:30.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:30.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient(hunting): authenticate will time out at 1355.474691s 2024-09-16T05:14:30.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a180a5d00 0x7f9a181cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:30.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a180a5d00 0x7f9a181cf090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:48710/0 (socket says 172.21.15.114:48710) 2024-09-16T05:14:30.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 1 -- 172.21.15.114:0/1604388489 learned_addr learned my addr 172.21.15.114:0/1604388489 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:30.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 10 monclient(hunting): get_auth_request con 0x7f9a180a5d00 auth_method 0 2024-09-16T05:14:30.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:30.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:30.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:30.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1e1c5640 1 --2- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a181376e0 0x7f9a1813c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:30.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1e1c5640 10 monclient(hunting): get_auth_request con 0x7f9a181376e0 auth_method 0 2024-09-16T05:14:30.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1e1c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:30.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1e1c5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:30.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1e1c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:30.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:30.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:30.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:30.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1e1c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:30.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1e1c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:30.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1e1c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:30.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 10 monclient(hunting): handle_auth_done global_id 54653 payload 824 2024-09-16T05:14:30.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 1 -- 172.21.15.114:0/1604388489 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9a181370a0 msgr2=0x7f9a181361b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:30.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 1 --2- 172.21.15.114:0/1604388489 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9a181370a0 0x7f9a181361b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 1 -- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a181376e0 msgr2=0x7f9a1813c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 1 --2- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a181376e0 0x7f9a1813c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 10 monclient: _finish_hunting 0 2024-09-16T05:14:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 1 monclient: found mon.noname-c 2024-09-16T05:14:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:14:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 1 -- 172.21.15.114:0/1604388489 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a18120b40 con 0x7f9a180a5d00 2024-09-16T05:14:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1d9c4640 1 --2- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a180a5d00 0x7f9a181cf090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9a0801a050 tx=0x7f9a080191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a167fc640 10 client.?.objecter ms_handle_connect 0x7f9a180a5d00 2024-09-16T05:14:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a167fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a167fc640 1 -- 172.21.15.114:0/1604388489 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a080b0040 con 0x7f9a180a5d00 2024-09-16T05:14:30.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:14:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a167fc640 10 monclient: dump: 2024-09-16T05:14:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:30.861 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:30.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a167fc640 10 monclient: _finish_auth 0 2024-09-16T05:14:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a167fc640 10 monclient: _check_auth_tickets 2024-09-16T05:14:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a167fc640 1 -- 172.21.15.114:0/1604388489 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a080b9070 con 0x7f9a180a5d00 2024-09-16T05:14:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 5 monclient: authenticate success, global_id 54653 2024-09-16T05:14:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient: _renew_subs 2024-09-16T05:14:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 -- 172.21.15.114:0/1604388489 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a180a8f90 con 0x7f9a180a5d00 2024-09-16T05:14:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a167fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a167fc640 1 -- 172.21.15.114:0/1604388489 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a080b4660 con 0x7f9a180a5d00 2024-09-16T05:14:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a167fc640 10 monclient: dump: 2024-09-16T05:14:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:30.863 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a17fff640 4 set_mon_vals no callback set 2024-09-16T05:14:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 client.54653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient: _renew_subs 2024-09-16T05:14:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 -- 172.21.15.114:0/1604388489 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a18073d20 con 0x7f9a180a5d00 2024-09-16T05:14:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.840+0000 7f9a1fc28640 1 librados: init done 2024-09-16T05:14:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1fc28640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1fc28640 1 --2- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a181205e0 0x7f9a1813c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1fc28640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1fc28640 1 -- 172.21.15.114:0/1604388489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a181209c0 con 0x7f9a181205e0 2024-09-16T05:14:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 1 -- 172.21.15.114:0/1604388489 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9a080b4800 con 0x7f9a180a5d00 2024-09-16T05:14:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1d1c3640 1 --2- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a181205e0 0x7f9a1813c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1d1c3640 10 monclient: get_auth_request con 0x7f9a181205e0 auth_method 0 2024-09-16T05:14:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1d1c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1d1c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1d1c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1d1c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1d1c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1d1c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 1 --2- 172.21.15.114:0/1604388489 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9a04037810 0x7f9a04039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 1 -- 172.21.15.114:0/1604388489 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(291..291 src has 1..291) ==== 5727+0+0 (secure 0 0 0) 0x7f9a080e2b50 con 0x7f9a180a5d00 2024-09-16T05:14:30.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 10 client.54653.objecter ms_dispatch 0x7f9a18000ba0 osd_map(291..291 src has 1..291) 2024-09-16T05:14:30.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 3 client.54653.objecter handle_osd_map got epochs [291,291] > 0 2024-09-16T05:14:30.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 3 client.54653.objecter handle_osd_map decoding full epoch 291 2024-09-16T05:14:30.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1d1c3640 10 monclient(hunting): handle_auth_done global_id 74166 payload 824 2024-09-16T05:14:30.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1d1c3640 1 -- 172.21.15.114:0/1604388489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a18137ac0 con 0x7f9a181205e0 2024-09-16T05:14:30.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 20 client.54653.objecter dump_active .. 0 homeless 2024-09-16T05:14:30.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 10 client.54653.objecter ms_handle_connect 0x7f9a181205e0 2024-09-16T05:14:30.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 10 client.54653.objecter resend_mon_ops 2024-09-16T05:14:30.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1d1c3640 1 --2- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a181205e0 0x7f9a1813c570 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9a18137480 tx=0x7f9a0c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:30.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1e1c5640 1 --2- 172.21.15.114:0/1604388489 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9a04037810 0x7f9a04039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:30.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 1 -- 172.21.15.114:0/1604388489 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c017070 con 0x7f9a181205e0 2024-09-16T05:14:30.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:30.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:30.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 10 monclient: dump: 2024-09-16T05:14:30.865 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:30.866 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1e1c5640 10 monclient: get_auth_request con 0x7f9a04037810 auth_method 0 2024-09-16T05:14:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 1 -- 172.21.15.114:0/1604388489 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9a0c004020 con 0x7f9a181205e0 2024-09-16T05:14:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:30.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:30.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 1 -- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a181205e0 msgr2=0x7f9a1813c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:30.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 1 --2- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a181205e0 0x7f9a1813c570 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9a18137480 tx=0x7f9a0c00ec50 comp rx=0 tx=0).stop 2024-09-16T05:14:30.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 1 -- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a181205e0 msgr2=0x7f9a1813c570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:30.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 1 --2- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a181205e0 0x7f9a1813c570 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a167fc640 10 client.54653.objecter ms_handle_connect 0x7f9a04037810 2024-09-16T05:14:30.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.844+0000 7f9a1e1c5640 1 --2- 172.21.15.114:0/1604388489 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9a04037810 0x7f9a04039cd0 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7f9a1813dd80 tx=0x7f9a10009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.848+0000 7f9a1fc28640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:14:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.848+0000 7f9a1fc28640 1 --2- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a1813cab0 0x7f9a18138190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.848+0000 7f9a1fc28640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.848+0000 7f9a1fc28640 1 -- 172.21.15.114:0/1604388489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1813cec0 con 0x7f9a1813cab0 2024-09-16T05:14:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.848+0000 7f9a1d1c3640 1 --2- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a1813cab0 0x7f9a18138190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.848+0000 7f9a1d1c3640 10 monclient: get_auth_request con 0x7f9a1813cab0 auth_method 0 2024-09-16T05:14:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.848+0000 7f9a1d1c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.848+0000 7f9a1d1c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.848+0000 7f9a1d1c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.848+0000 7f9a1d1c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.848+0000 7f9a1d1c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.848+0000 7f9a1d1c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.848+0000 7f9a1d1c3640 10 monclient(hunting): handle_auth_done global_id 74169 payload 824 2024-09-16T05:14:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.848+0000 7f9a1d1c3640 1 -- 172.21.15.114:0/1604388489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9a181cff10 con 0x7f9a1813cab0 2024-09-16T05:14:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1d1c3640 1 --2- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a1813cab0 0x7f9a18138190 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9a18137480 tx=0x7f9a0c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a167fc640 10 client.54653.objecter ms_handle_connect 0x7f9a1813cab0 2024-09-16T05:14:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a167fc640 10 client.54653.objecter resend_mon_ops 2024-09-16T05:14:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a167fc640 1 -- 172.21.15.114:0/1604388489 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c017040 con 0x7f9a1813cab0 2024-09-16T05:14:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a167fc640 10 monclient: dump: 2024-09-16T05:14:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:30.871 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a167fc640 1 -- 172.21.15.114:0/1604388489 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7f9a0c005bb0 con 0x7f9a1813cab0 2024-09-16T05:14:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a167fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:14:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a167fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a167fc640 1 -- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a1813cab0 msgr2=0x7f9a18138190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a167fc640 1 --2- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a1813cab0 0x7f9a18138190 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9a18137480 tx=0x7f9a0c0046d0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a167fc640 1 -- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a1813cab0 msgr2=0x7f9a18138190 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:30.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a167fc640 1 --2- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a1813cab0 0x7f9a18138190 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.872 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:14:30.872 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:14:30.872 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:14:30.872 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:14:30.872 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 34, 2024-09-16T05:14:30.872 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 29907, 2024-09-16T05:14:30.872 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:14:30.872 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:14:30.872 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:14:30.872 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:14:30.872 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:30.873 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 4, 2024-09-16T05:14:30.873 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:30.873 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:14:30.873 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:14:30.873 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:30.873 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:30.873 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:30.873 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:30.873 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:30.873 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:30.873 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:30.873 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:30.873 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:30.874 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:30.874 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:30.874 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:30.874 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:14:30.874 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:14:30.874 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:30.874 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:30.874 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:30.874 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:30.874 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:30.874 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:30.874 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:30.874 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:30.875 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:30.875 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:30.875 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:30.875 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:30.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:30.875 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:14:30.875 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:14:30.875 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:14:30.875 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:14:30.875 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:14:30.875 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:14:30.875 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:30.875 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:30.875 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:14:30.876 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:14:30.876 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:14:30.876 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:14:30.876 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:14:30.876 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:14:30.876 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:14:30.876 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:30.876 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:14:30.876 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:30.876 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:30.876 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:30.876 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:30.876 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:30.876 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:30.877 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:30.877 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:30.877 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:30.877 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:30.877 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:30.877 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:30.877 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:14:30.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:30.877 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:14:30.877 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:30.877 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:14:30.877 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:14:30.877 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:30.878 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:30.878 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:30.878 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:30.878 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:14:30.878 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:30.878 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:30.878 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:30.878 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:30.878 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:14:30.878 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:30.878 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:30.878 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:30.878 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:30.878 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:14:30.879 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:14:30.879 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:30.879 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:30.879 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:30.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:30.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:30.879 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:14:30.879 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:14:30.879 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:30.879 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:30.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:30.880 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:30.880 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:14:30.880 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:30.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:30.880 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:30.880 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:30.880 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:14:30.880 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:30.880 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:30.880 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:30.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:30.880 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:14:30.880 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:14:30.881 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:30.881 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:30.881 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:30.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:30.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:30.881 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:14:30.881 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:14:30.881 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:30.881 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:30.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:30.881 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:30.881 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:14:30.881 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:30.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:30.882 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:30.882 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:30.882 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:14:30.882 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:30.882 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:30.882 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:30.882 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:30.882 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:14:30.882 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:14:30.882 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:30.882 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:30.883 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:30.883 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:30.883 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:30.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:30.883 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:14:30.883 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:14:30.883 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:30.883 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:30.883 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:30.883 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:14:30.883 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:30.884 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:30.884 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:14:30.884 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:30.884 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:30.884 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:30.884 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:14:30.884 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:30.884 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:30.884 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:14:30.884 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:30.884 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:30.884 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:30.884 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:14:30.885 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:30.885 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:30.885 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:30.885 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:14:30.885 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:14:30.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1fc28640 10 librados: watch_flush enter 2024-09-16T05:14:30.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1fc28640 10 librados: watch_flush exit 2024-09-16T05:14:30.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1fc28640 20 client.54653.objecter shutdown clearing up homeless session... 2024-09-16T05:14:30.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1fc28640 10 client.54653.objecter successfully canceled tick 2024-09-16T05:14:30.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1fc28640 1 -- 172.21.15.114:0/1604388489 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9a04037810 msgr2=0x7f9a04039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:30.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1fc28640 1 --2- 172.21.15.114:0/1604388489 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9a04037810 0x7f9a04039cd0 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7f9a1813dd80 tx=0x7f9a10009040 comp rx=0 tx=0).stop 2024-09-16T05:14:30.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1fc28640 10 monclient: shutdown 2024-09-16T05:14:30.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1fc28640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:30.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1fc28640 1 -- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a180a5d00 msgr2=0x7f9a181cf090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:30.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1fc28640 1 --2- 172.21.15.114:0/1604388489 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a180a5d00 0x7f9a181cf090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9a0801a050 tx=0x7f9a080191c0 comp rx=0 tx=0).stop 2024-09-16T05:14:30.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1e1c5640 1 -- 172.21.15.114:0/1604388489 reap_dead start 2024-09-16T05:14:30.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1e1c5640 1 -- 172.21.15.114:0/1604388489 reap_dead start 2024-09-16T05:14:30.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1fc28640 1 -- 172.21.15.114:0/1604388489 shutdown_connections 2024-09-16T05:14:30.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1fc28640 1 -- 172.21.15.114:0/1604388489 >> 172.21.15.114:0/1604388489 conn(0x7f9a180a9a40 msgr2=0x7f9a180a7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:30.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1fc28640 1 -- 172.21.15.114:0/1604388489 shutdown_connections 2024-09-16T05:14:30.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1fc28640 1 -- 172.21.15.114:0/1604388489 wait complete. 2024-09-16T05:14:30.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:30.852+0000 7f9a1fc28640 1 librados: shutdown 2024-09-16T05:14:30.888 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:14:30.889 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:14:30.889 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:14:31.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 10 monclient: get_monmap_and_config 2024-09-16T05:14:31.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 10 monclient: build_initial_monmap 2024-09-16T05:14:31.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 10 monclient: monmap: 2024-09-16T05:14:31.016 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:31.016 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.016 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:31.017628+0000 2024-09-16T05:14:31.016 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:31.017628+0000 2024-09-16T05:14:31.016 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:31.016 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.016 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:31.016 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:31.016 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:31.016 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:31.016 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:31.016 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 1 Processor -- start 2024-09-16T05:14:31.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 1 -- start start 2024-09-16T05:14:31.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 10 monclient: init 2024-09-16T05:14:31.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:31.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-16T05:14:31.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa25809b6c0 0x7fa2580a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 10 monclient(hunting): picked mon.noname-b con 0x7fa25809b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:31.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa2580a1400 0x7fa2580a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 10 monclient(hunting): picked mon.noname-a con 0x7fa2580a1400 addr v2:172.21.15.114:3300/0 2024-09-16T05:14:31.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa258139640 addr v1:172.21.15.149:6789/0 2024-09-16T05:14:31.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa25809d4e0 con 0x7fa258139640 2024-09-16T05:14:31.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa25809e390 con 0x7fa2580a1400 2024-09-16T05:14:31.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa25809ece0 con 0x7fa25809b6c0 2024-09-16T05:14:31.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:31.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa260472640 10 monclient(hunting): authenticate will time out at 1355.646802s 2024-09-16T05:14:31.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa25809b6c0 0x7fa2580a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25da0d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa2580a1400 0x7fa2580a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa25809b6c0 0x7fa2580a0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:48726/0 (socket says 172.21.15.114:48726) 2024-09-16T05:14:31.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25da0d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa2580a1400 0x7fa2580a17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36822/0 (socket says 172.21.15.114:36822) 2024-09-16T05:14:31.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 1 -- 172.21.15.114:0/1925096941 learned_addr learned my addr 172.21.15.114:0/1925096941 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:31.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 10 monclient(hunting): get_auth_request con 0x7fa25809b6c0 auth_method 0 2024-09-16T05:14:31.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:31.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:31.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:31.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25da0d640 10 monclient(hunting): get_auth_request con 0x7fa2580a1400 auth_method 0 2024-09-16T05:14:31.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25da0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:31.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25da0d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:31.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25da0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:31.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:31.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:31.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:31.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25da0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:31.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25da0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:31.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25da0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:31.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 10 monclient(hunting): handle_auth_done global_id 54656 payload 274 2024-09-16T05:14:31.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 1 -- 172.21.15.114:0/1925096941 >> v2:172.21.15.114:3300/0 conn(0x7fa2580a1400 msgr2=0x7fa2580a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:31.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 1 --2- 172.21.15.114:0/1925096941 >> v2:172.21.15.114:3300/0 conn(0x7fa2580a1400 0x7fa2580a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 1 -- 172.21.15.114:0/1925096941 >> v1:172.21.15.149:6789/0 conn(0x7fa258139640 legacy=0x7fa25813ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:31.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 10 monclient: _finish_hunting 0 2024-09-16T05:14:31.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 1 monclient: found mon.noname-b 2024-09-16T05:14:31.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:31.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:14:31.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25e20e640 1 -- 172.21.15.114:0/1925096941 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25809f430 con 0x7fa25809b6c0 2024-09-16T05:14:31.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.012+0000 7fa25da0d640 1 --2- 172.21.15.114:0/1925096941 >> v2:172.21.15.114:3300/0 conn(0x7fa2580a1400 0x7fa2580a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:14:31.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25e20e640 1 --2- 172.21.15.114:0/1925096941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa25809b6c0 0x7fa2580a0ec0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa2580a3070 tx=0x7fa24801c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a1bb2ef18945f4b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:31.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25d20c640 1 -- 172.21.15.114:0/1925096941 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2480b0070 con 0x7fa25809b6c0 2024-09-16T05:14:31.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25d20c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25d20c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:14:31.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25d20c640 10 monclient: dump: 2024-09-16T05:14:31.023 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.023 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.023 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.023 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.023 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.023 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.023 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.023 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.023 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.023 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25d20c640 10 monclient: _finish_auth 0 2024-09-16T05:14:31.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25d20c640 10 monclient: _check_auth_tickets 2024-09-16T05:14:31.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25d20c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:31.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25d20c640 1 -- 172.21.15.114:0/1925096941 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2480ab600 con 0x7fa25809b6c0 2024-09-16T05:14:31.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25d20c640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:31.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25d20c640 1 -- 172.21.15.114:0/1925096941 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2480ab9d0 con 0x7fa25809b6c0 2024-09-16T05:14:31.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 5 monclient: authenticate success, global_id 54656 2024-09-16T05:14:31.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:31.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 4 set_mon_vals no callback set 2024-09-16T05:14:31.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25d20c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25d20c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:31.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25d20c640 10 monclient: dump: 2024-09-16T05:14:31.024 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.024 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.025 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.025 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.025 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.025 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.025 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.025 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.025 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.025 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient: shutdown 2024-09-16T05:14:31.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:31.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 -- 172.21.15.114:0/1925096941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa25809b6c0 msgr2=0x7fa2580a0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 --2- 172.21.15.114:0/1925096941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa25809b6c0 0x7fa2580a0ec0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa2580a3070 tx=0x7fa24801c3a0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 -- 172.21.15.114:0/1925096941 shutdown_connections 2024-09-16T05:14:31.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 --2- 172.21.15.114:0/1925096941 >> v2:172.21.15.114:3300/0 conn(0x7fa2580a1400 0x7fa2580a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 --2- 172.21.15.114:0/1925096941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa25809b6c0 0x7fa2580a0ec0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 -- 172.21.15.114:0/1925096941 >> 172.21.15.114:0/1925096941 conn(0x7fa2580a1fb0 msgr2=0x7fa2580a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:31.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 -- 172.21.15.114:0/1925096941 shutdown_connections 2024-09-16T05:14:31.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 -- 172.21.15.114:0/1925096941 wait complete. 2024-09-16T05:14:31.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient: build_initial_monmap 2024-09-16T05:14:31.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient: monmap: 2024-09-16T05:14:31.031 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:31.031 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:31.031 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:31.031 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:31.031 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:31.031 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:31.032 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 librados: starting msgr at 2024-09-16T05:14:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 librados: starting objecter 2024-09-16T05:14:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 Processor -- start 2024-09-16T05:14:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 -- start start 2024-09-16T05:14:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 librados: setting wanted keys 2024-09-16T05:14:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 librados: calling monclient init 2024-09-16T05:14:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient: init 2024-09-16T05:14:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:14:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa25809b6c0 0x7fa25808ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient(hunting): picked mon.noname-c con 0x7fa25809b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa2580a1400 0x7fa2581cf640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient(hunting): picked mon.noname-b con 0x7fa2580a1400 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa258139640 0x7fa2580987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient(hunting): picked mon.noname-a con 0x7fa258139640 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa25813f970 con 0x7fa258139640 2024-09-16T05:14:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fa25809cad0 con 0x7fa2580a1400 2024-09-16T05:14:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa25813faf0 con 0x7fa25809b6c0 2024-09-16T05:14:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient(hunting): authenticate will time out at 1355.654977s 2024-09-16T05:14:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25e20e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa25809b6c0 0x7fa25808ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa258139640 0x7fa2580987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25e20e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa25809b6c0 0x7fa25808ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:48736/0 (socket says 172.21.15.114:48736) 2024-09-16T05:14:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25e20e640 1 -- 172.21.15.114:0/2253396097 learned_addr learned my addr 172.21.15.114:0/2253396097 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25e20e640 10 monclient(hunting): get_auth_request con 0x7fa25809b6c0 auth_method 0 2024-09-16T05:14:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25e20e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25e20e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25e20e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 10 monclient(hunting): get_auth_request con 0x7fa258139640 auth_method 0 2024-09-16T05:14:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25e20e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25e20e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25e20e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 10 monclient(hunting): handle_auth_done global_id 74175 payload 824 2024-09-16T05:14:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25da0d640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa2580a1400 0x7fa2581cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 1 -- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa25809b6c0 msgr2=0x7fa25808ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa25809b6c0 0x7fa25808ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 1 -- 172.21.15.114:0/2253396097 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa2580a1400 msgr2=0x7fa2581cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa2580a1400 0x7fa2581cf640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 10 monclient: _finish_hunting 0 2024-09-16T05:14:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 1 monclient: found mon.noname-a 2024-09-16T05:14:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 1 -- 172.21.15.114:0/2253396097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa258091990 con 0x7fa258139640 2024-09-16T05:14:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25e20e640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa25809b6c0 0x7fa25808ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:14:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ea0f640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa258139640 0x7fa2580987b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa25000c780 tx=0x7fa25000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa24effd640 10 client.?.objecter ms_handle_connect 0x7fa258139640 2024-09-16T05:14:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa24effd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa24effd640 1 -- 172.21.15.114:0/2253396097 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa250016020 con 0x7fa258139640 2024-09-16T05:14:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa24effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa24effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa24effd640 10 monclient: dump: 2024-09-16T05:14:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.037 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa24effd640 10 monclient: _finish_auth 0 2024-09-16T05:14:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa24effd640 10 monclient: _check_auth_tickets 2024-09-16T05:14:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa24effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 5 monclient: authenticate success, global_id 74175 2024-09-16T05:14:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient: _renew_subs 2024-09-16T05:14:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa24effd640 1 -- 172.21.15.114:0/2253396097 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa25000aa50 con 0x7fa258139640 2024-09-16T05:14:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 -- 172.21.15.114:0/2253396097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa258091c70 con 0x7fa258139640 2024-09-16T05:14:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa24effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa24effd640 1 -- 172.21.15.114:0/2253396097 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25000ce30 con 0x7fa258139640 2024-09-16T05:14:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa24effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa24effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa24effd640 10 monclient: dump: 2024-09-16T05:14:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.039 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.039 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa25ca0b640 4 set_mon_vals no callback set 2024-09-16T05:14:31.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 client.74175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:31.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient: _renew_subs 2024-09-16T05:14:31.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:31.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 -- 172.21.15.114:0/2253396097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa258090c10 con 0x7fa258139640 2024-09-16T05:14:31.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.020+0000 7fa260472640 1 librados: init done 2024-09-16T05:14:31.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa260472640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:31.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa260472640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa258098cf0 0x7fa2581cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa260472640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa260472640 1 -- 172.21.15.114:0/2253396097 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa258069390 con 0x7fa258098cf0 2024-09-16T05:14:31.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa25e20e640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa258098cf0 0x7fa2581cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa25e20e640 10 monclient: get_auth_request con 0x7fa258098cf0 auth_method 0 2024-09-16T05:14:31.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa25e20e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:31.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa25e20e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:31.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa25e20e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:31.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa25e20e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:31.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa25e20e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:31.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 1 -- 172.21.15.114:0/2253396097 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa250019020 con 0x7fa258139640 2024-09-16T05:14:31.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa25e20e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:31.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa25e20e640 10 monclient(hunting): handle_auth_done global_id 54662 payload 824 2024-09-16T05:14:31.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 1 --2- 172.21.15.114:0/2253396097 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa23c037810 0x7fa23c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa25e20e640 1 -- 172.21.15.114:0/2253396097 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa2580008e0 con 0x7fa258098cf0 2024-09-16T05:14:31.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 1 -- 172.21.15.114:0/2253396097 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(291..291 src has 1..291) ==== 5727+0+0 (secure 0 0 0) 0x7fa250048ac0 con 0x7fa258139640 2024-09-16T05:14:31.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 10 client.74175.objecter ms_dispatch 0x7fa258000ba0 osd_map(291..291 src has 1..291) 2024-09-16T05:14:31.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 3 client.74175.objecter handle_osd_map got epochs [291,291] > 0 2024-09-16T05:14:31.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 3 client.74175.objecter handle_osd_map decoding full epoch 291 2024-09-16T05:14:31.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 20 client.74175.objecter dump_active .. 0 homeless 2024-09-16T05:14:31.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 10 client.74175.objecter ms_handle_connect 0x7fa258098cf0 2024-09-16T05:14:31.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 10 client.74175.objecter resend_mon_ops 2024-09-16T05:14:31.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa25e20e640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa258098cf0 0x7fa2581cbc60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa24801cc40 tx=0x7fa24801abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:31.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 1 -- 172.21.15.114:0/2253396097 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2480b0050 con 0x7fa258098cf0 2024-09-16T05:14:31.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:31.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 10 monclient: dump: 2024-09-16T05:14:31.041 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.041 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.041 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.041 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.041 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.042 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.042 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.042 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.042 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.042 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa25da0d640 1 --2- 172.21.15.114:0/2253396097 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa23c037810 0x7fa23c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa25da0d640 10 monclient: get_auth_request con 0x7fa23c037810 auth_method 0 2024-09-16T05:14:31.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 1 -- 172.21.15.114:0/2253396097 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa2480bc070 con 0x7fa258098cf0 2024-09-16T05:14:31.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:31.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:31.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 1 -- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa258098cf0 msgr2=0x7fa2581cbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa258098cf0 0x7fa2581cbc60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa24801cc40 tx=0x7fa24801abe0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 1 -- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa258098cf0 msgr2=0x7fa2581cbc60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:31.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa258098cf0 0x7fa2581cbc60 secure :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa24801cc40 tx=0x7fa24801abe0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa25da0d640 1 --2- 172.21.15.114:0/2253396097 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa23c037810 0x7fa23c039cd0 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7fa258099120 tx=0x7fa254008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:31.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.024+0000 7fa24effd640 10 client.74175.objecter ms_handle_connect 0x7fa23c037810 2024-09-16T05:14:31.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.028+0000 7fa260472640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:14:31.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.028+0000 7fa260472640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2581cc1a0 0x7fa2581cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.028+0000 7fa260472640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.028+0000 7fa260472640 1 -- 172.21.15.114:0/2253396097 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa2581ccaf0 con 0x7fa2581cc1a0 2024-09-16T05:14:31.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.028+0000 7fa25e20e640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2581cc1a0 0x7fa2581cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.028+0000 7fa25e20e640 10 monclient: get_auth_request con 0x7fa2581cc1a0 auth_method 0 2024-09-16T05:14:31.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.028+0000 7fa25e20e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:31.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.028+0000 7fa25e20e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:31.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.028+0000 7fa25e20e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:31.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.028+0000 7fa25e20e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:31.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.028+0000 7fa25e20e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:31.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.028+0000 7fa25e20e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:31.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa25e20e640 10 monclient(hunting): handle_auth_done global_id 54665 payload 824 2024-09-16T05:14:31.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa25e20e640 1 -- 172.21.15.114:0/2253396097 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa2580a0dc0 con 0x7fa2581cc1a0 2024-09-16T05:14:31.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa24effd640 10 client.74175.objecter ms_handle_connect 0x7fa2581cc1a0 2024-09-16T05:14:31.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa24effd640 10 client.74175.objecter resend_mon_ops 2024-09-16T05:14:31.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa25e20e640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2581cc1a0 0x7fa2581cc5b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa258076960 tx=0x7fa248019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:31.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa24effd640 1 -- 172.21.15.114:0/2253396097 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2480c7070 con 0x7fa2581cc1a0 2024-09-16T05:14:31.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa24effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa24effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:31.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa24effd640 10 monclient: dump: 2024-09-16T05:14:31.046 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.046 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.046 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.046 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.046 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.046 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.046 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.046 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.046 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.046 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa24effd640 1 -- 172.21.15.114:0/2253396097 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fa2480b1040 con 0x7fa2581cc1a0 2024-09-16T05:14:31.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa24effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:14:31.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa24effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:31.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa24effd640 1 -- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2581cc1a0 msgr2=0x7fa2581cc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa24effd640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2581cc1a0 0x7fa2581cc5b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa258076960 tx=0x7fa248019140 comp rx=0 tx=0).stop 2024-09-16T05:14:31.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa24effd640 1 -- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2581cc1a0 msgr2=0x7fa2581cc5b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:31.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa24effd640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2581cc1a0 0x7fa2581cc5b0 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa258076960 tx=0x7fa248019140 comp rx=0 tx=0).stop 2024-09-16T05:14:31.047 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:14:31.047 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:14:31.047 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:14:31.047 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:14:31.047 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 34, 2024-09-16T05:14:31.047 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 285168, 2024-09-16T05:14:31.048 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:14:31.048 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:14:31.048 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:14:31.048 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:14:31.048 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:31.048 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 4, 2024-09-16T05:14:31.048 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:31.048 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:14:31.048 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:14:31.048 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:31.048 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:31.048 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:31.048 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:31.049 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:31.049 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:31.049 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:31.049 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:31.049 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:31.049 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:31.049 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:31.049 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:31.049 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:14:31.049 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:14:31.049 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:31.049 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:31.049 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:31.049 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:31.049 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:31.050 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:31.050 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:31.050 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:31.050 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:31.050 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:31.050 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:31.050 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:31.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.050 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:14:31.050 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:14:31.050 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:14:31.050 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:14:31.050 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:14:31.051 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:14:31.051 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:31.051 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:31.051 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:14:31.051 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:14:31.051 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:14:31.051 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:14:31.051 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:14:31.051 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:14:31.052 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:14:31.052 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:31.052 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:14:31.052 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:31.052 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:31.052 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:31.052 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:31.052 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:31.052 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:31.052 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:31.052 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:31.052 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:31.052 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:31.052 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:31.053 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:31.053 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:14:31.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.053 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:14:31.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.053 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:14:31.053 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:14:31.053 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:31.053 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:31.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.053 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:31.053 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:14:31.053 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:31.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.054 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:31.054 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:14:31.054 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:31.054 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:31.054 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:31.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.054 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:14:31.054 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:14:31.054 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:31.054 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:31.054 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:31.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.055 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:14:31.055 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:14:31.055 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:31.055 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:31.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.055 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:31.055 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:14:31.055 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:31.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.055 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:31.055 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:14:31.056 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:31.056 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:31.056 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:31.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.056 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:14:31.056 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:14:31.056 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:31.056 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:31.056 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:31.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.056 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:14:31.056 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:14:31.057 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:31.057 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:31.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.057 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:31.057 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:14:31.057 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:31.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.057 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:31.057 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:14:31.057 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:31.057 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:31.057 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:31.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.058 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:14:31.058 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:14:31.058 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:31.058 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:31.058 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:31.058 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:31.058 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:31.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.058 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:14:31.058 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:14:31.058 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.058 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:31.058 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:31.059 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:14:31.059 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:31.059 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:31.059 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:14:31.059 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.059 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:31.059 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:31.059 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:14:31.059 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:31.059 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:31.059 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:14:31.059 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.059 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:31.059 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:31.060 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:14:31.060 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:31.060 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:31.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.060 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:14:31.060 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:14:31.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa260472640 10 librados: watch_flush enter 2024-09-16T05:14:31.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa260472640 10 librados: watch_flush exit 2024-09-16T05:14:31.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa260472640 20 client.74175.objecter shutdown clearing up homeless session... 2024-09-16T05:14:31.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa260472640 10 client.74175.objecter successfully canceled tick 2024-09-16T05:14:31.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa260472640 1 -- 172.21.15.114:0/2253396097 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa23c037810 msgr2=0x7fa23c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa260472640 1 --2- 172.21.15.114:0/2253396097 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa23c037810 0x7fa23c039cd0 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7fa258099120 tx=0x7fa254008040 comp rx=0 tx=0).stop 2024-09-16T05:14:31.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa260472640 10 monclient: shutdown 2024-09-16T05:14:31.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa260472640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:31.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa260472640 1 -- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa258139640 msgr2=0x7fa2580987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa260472640 1 --2- 172.21.15.114:0/2253396097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa258139640 0x7fa2580987b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa25000c780 tx=0x7fa25000cc40 comp rx=0 tx=0).stop 2024-09-16T05:14:31.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa25ea0f640 1 -- 172.21.15.114:0/2253396097 reap_dead start 2024-09-16T05:14:31.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa25ea0f640 1 -- 172.21.15.114:0/2253396097 reap_dead start 2024-09-16T05:14:31.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa260472640 1 -- 172.21.15.114:0/2253396097 shutdown_connections 2024-09-16T05:14:31.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa260472640 1 -- 172.21.15.114:0/2253396097 >> 172.21.15.114:0/2253396097 conn(0x7fa2580a1fb0 msgr2=0x7fa2581389f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:31.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa260472640 1 -- 172.21.15.114:0/2253396097 shutdown_connections 2024-09-16T05:14:31.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa260472640 1 -- 172.21.15.114:0/2253396097 wait complete. 2024-09-16T05:14:31.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.032+0000 7fa260472640 1 librados: shutdown 2024-09-16T05:14:31.063 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:14:31.063 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:14:31.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:14:31.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 10 monclient: get_monmap_and_config 2024-09-16T05:14:31.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 10 monclient: build_initial_monmap 2024-09-16T05:14:31.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 10 monclient: monmap: 2024-09-16T05:14:31.178 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:31.178 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.179 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:31.180490+0000 2024-09-16T05:14:31.179 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:31.180490+0000 2024-09-16T05:14:31.179 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:31.179 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.179 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:31.179 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:31.179 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:31.179 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:31.179 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:31.179 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 1 Processor -- start 2024-09-16T05:14:31.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 1 -- start start 2024-09-16T05:14:31.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 10 monclient: init 2024-09-16T05:14:31.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:31.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-16T05:14:31.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa40006bcc0 0x7fa40009ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 10 monclient(hunting): picked mon.noname-b con 0x7fa40006bcc0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:31.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa40006c0a0 0x7fa40006c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 10 monclient(hunting): picked mon.noname-a con 0x7fa40006c0a0 addr v2:172.21.15.114:3300/0 2024-09-16T05:14:31.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fa4001376b0 0x7fa400137b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 10 monclient(hunting): picked mon.noname-c con 0x7fa4001376b0 addr v2:172.21.15.149:3300/0 2024-09-16T05:14:31.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa40009ec00 con 0x7fa40006c0a0 2024-09-16T05:14:31.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fa40006d700 con 0x7fa4001376b0 2024-09-16T05:14:31.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa40006d9e0 con 0x7fa40006bcc0 2024-09-16T05:14:31.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:31.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa406d25640 10 monclient(hunting): authenticate will time out at 1355.809623s 2024-09-16T05:14:31.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa40006bcc0 0x7fa40009ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa3fffff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa40006c0a0 0x7fa40006c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa40006bcc0 0x7fa40009ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:48772/0 (socket says 172.21.15.114:48772) 2024-09-16T05:14:31.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 1 -- 172.21.15.114:0/3066526745 learned_addr learned my addr 172.21.15.114:0/3066526745 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:31.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 10 monclient(hunting): get_auth_request con 0x7fa40006bcc0 auth_method 0 2024-09-16T05:14:31.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:31.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa3fffff640 10 monclient(hunting): get_auth_request con 0x7fa40006c0a0 auth_method 0 2024-09-16T05:14:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa3fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa3fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa3fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa3fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa3fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa3fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 10 monclient(hunting): handle_auth_done global_id 54668 payload 274 2024-09-16T05:14:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 1 -- 172.21.15.114:0/3066526745 >> v2:172.21.15.149:3300/0 conn(0x7fa4001376b0 msgr2=0x7fa400137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa4052c2640 1 --2- 172.21.15.114:0/3066526745 >> v2:172.21.15.149:3300/0 conn(0x7fa4001376b0 0x7fa400137b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 1 --2- 172.21.15.114:0/3066526745 >> v2:172.21.15.149:3300/0 conn(0x7fa4001376b0 0x7fa400137b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 1 -- 172.21.15.114:0/3066526745 >> v2:172.21.15.114:3300/0 conn(0x7fa40006c0a0 msgr2=0x7fa40006c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 1 --2- 172.21.15.114:0/3066526745 >> v2:172.21.15.114:3300/0 conn(0x7fa40006c0a0 0x7fa40006c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 10 monclient: _finish_hunting 0 2024-09-16T05:14:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 1 monclient: found mon.noname-b 2024-09-16T05:14:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:14:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa404ac1640 1 -- 172.21.15.114:0/3066526745 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa40009f4d0 con 0x7fa40006bcc0 2024-09-16T05:14:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa4052c2640 10 monclient: get_auth_request con 0x7fa4001376b0 auth_method 0 2024-09-16T05:14:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa3fffff640 1 --2- 172.21.15.114:0/3066526745 >> v2:172.21.15.114:3300/0 conn(0x7fa40006c0a0 0x7fa40006c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:14:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.176+0000 7fa4052c2640 1 --2- 172.21.15.114:0/3066526745 >> v2:172.21.15.149:3300/0 conn(0x7fa4001376b0 0x7fa400137b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:14:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 1 --2- 172.21.15.114:0/3066526745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa40006bcc0 0x7fa40009ef90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa400069160 tx=0x7fa3f801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2aefe1dd2a1e536 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3ff7fe640 1 -- 172.21.15.114:0/3066526745 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f80b2070 con 0x7fa40006bcc0 2024-09-16T05:14:31.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3ff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:14:31.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3ff7fe640 10 monclient: dump: 2024-09-16T05:14:31.194 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.194 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.194 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.194 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.194 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.194 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.194 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.194 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.195 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3ff7fe640 10 monclient: _finish_auth 0 2024-09-16T05:14:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3ff7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:14:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3ff7fe640 1 -- 172.21.15.114:0/3066526745 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3f80a9070 con 0x7fa40006bcc0 2024-09-16T05:14:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 5 monclient: authenticate success, global_id 54668 2024-09-16T05:14:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3ff7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 4 set_mon_vals no callback set 2024-09-16T05:14:31.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3ff7fe640 1 -- 172.21.15.114:0/3066526745 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f80ad880 con 0x7fa40006bcc0 2024-09-16T05:14:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3ff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3ff7fe640 10 monclient: dump: 2024-09-16T05:14:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.196 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient: shutdown 2024-09-16T05:14:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 -- 172.21.15.114:0/3066526745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa40006bcc0 msgr2=0x7fa40009ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 --2- 172.21.15.114:0/3066526745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa40006bcc0 0x7fa40009ef90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa400069160 tx=0x7fa3f801c3e0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 -- 172.21.15.114:0/3066526745 shutdown_connections 2024-09-16T05:14:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 --2- 172.21.15.114:0/3066526745 >> v2:172.21.15.149:3300/0 conn(0x7fa4001376b0 0x7fa400137b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 --2- 172.21.15.114:0/3066526745 >> v2:172.21.15.114:3300/0 conn(0x7fa40006c0a0 0x7fa40006c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 --2- 172.21.15.114:0/3066526745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa40006bcc0 0x7fa40009ef90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 -- 172.21.15.114:0/3066526745 >> 172.21.15.114:0/3066526745 conn(0x7fa4000a4210 msgr2=0x7fa4000a4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 -- 172.21.15.114:0/3066526745 shutdown_connections 2024-09-16T05:14:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 -- 172.21.15.114:0/3066526745 wait complete. 2024-09-16T05:14:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient: build_initial_monmap 2024-09-16T05:14:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient: monmap: 2024-09-16T05:14:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:31.198 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 librados: starting msgr at 2024-09-16T05:14:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 librados: starting objecter 2024-09-16T05:14:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 Processor -- start 2024-09-16T05:14:31.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 -- start start 2024-09-16T05:14:31.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 librados: setting wanted keys 2024-09-16T05:14:31.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 librados: calling monclient init 2024-09-16T05:14:31.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient: init 2024-09-16T05:14:31.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:31.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:14:31.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa40006bcc0 0x7fa4001cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient(hunting): picked mon.noname-c con 0x7fa40006bcc0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:31.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa40006c0a0 0x7fa40013d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient(hunting): picked mon.noname-b con 0x7fa40006c0a0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:31.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4001376b0 0x7fa40013d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient(hunting): picked mon.noname-a con 0x7fa4001376b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:31.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa4000a3d30 con 0x7fa4001376b0 2024-09-16T05:14:31.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fa4000a3bb0 con 0x7fa40006c0a0 2024-09-16T05:14:31.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa400141190 con 0x7fa40006bcc0 2024-09-16T05:14:31.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:31.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient(hunting): authenticate will time out at 1355.814746s 2024-09-16T05:14:31.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa40006bcc0 0x7fa4001cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa4052c2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4001376b0 0x7fa40013d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa40006bcc0 0x7fa4001cd5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:48782/0 (socket says 172.21.15.114:48782) 2024-09-16T05:14:31.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 1 -- 172.21.15.114:0/4050528348 learned_addr learned my addr 172.21.15.114:0/4050528348 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:31.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 10 monclient(hunting): get_auth_request con 0x7fa40006bcc0 auth_method 0 2024-09-16T05:14:31.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:31.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:31.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:31.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa4052c2640 10 monclient(hunting): get_auth_request con 0x7fa4001376b0 auth_method 0 2024-09-16T05:14:31.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa4052c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:31.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa4052c2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:31.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa4052c2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:31.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:31.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:31.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:31.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa4052c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:31.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa4052c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:31.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa4052c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:31.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fffff640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa40006c0a0 0x7fa40013d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fffff640 10 monclient(hunting): get_auth_request con 0x7fa40006c0a0 auth_method 0 2024-09-16T05:14:31.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:31.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:31.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:31.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 10 monclient(hunting): handle_auth_done global_id 54671 payload 824 2024-09-16T05:14:31.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 1 -- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa40006c0a0 msgr2=0x7fa40013d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa40006c0a0 0x7fa40013d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 1 -- 172.21.15.114:0/4050528348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4001376b0 msgr2=0x7fa40013d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4001376b0 0x7fa40013d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 10 monclient: _finish_hunting 0 2024-09-16T05:14:31.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 1 monclient: found mon.noname-c 2024-09-16T05:14:31.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:31.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:14:31.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 1 -- 172.21.15.114:0/4050528348 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4001398e0 con 0x7fa40006bcc0 2024-09-16T05:14:31.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa4052c2640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4001376b0 0x7fa40013d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:14:31.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa404ac1640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa40006bcc0 0x7fa4001cd5a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa3f801aa90 tx=0x7fa3f8019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fd7fa640 10 client.?.objecter ms_handle_connect 0x7fa40006bcc0 2024-09-16T05:14:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fd7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fd7fa640 1 -- 172.21.15.114:0/4050528348 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f80bb070 con 0x7fa40006bcc0 2024-09-16T05:14:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:14:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fd7fa640 10 monclient: dump: 2024-09-16T05:14:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.204 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.204 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fd7fa640 10 monclient: _finish_auth 0 2024-09-16T05:14:31.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:14:31.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:31.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fd7fa640 1 -- 172.21.15.114:0/4050528348 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3f80adce0 con 0x7fa40006bcc0 2024-09-16T05:14:31.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 5 monclient: authenticate success, global_id 54671 2024-09-16T05:14:31.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient: _renew_subs 2024-09-16T05:14:31.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:31.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 -- 172.21.15.114:0/4050528348 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4001ce020 con 0x7fa40006bcc0 2024-09-16T05:14:31.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:31.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fd7fa640 1 -- 172.21.15.114:0/4050528348 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f80b2040 con 0x7fa40006bcc0 2024-09-16T05:14:31.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:14:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3fd7fa640 10 monclient: dump: 2024-09-16T05:14:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.205 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 client.54671.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient: _renew_subs 2024-09-16T05:14:31.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa3feffd640 4 set_mon_vals no callback set 2024-09-16T05:14:31.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:14:31.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 -- 172.21.15.114:0/4050528348 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4000a2990 con 0x7fa40006bcc0 2024-09-16T05:14:31.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.180+0000 7fa406d25640 1 librados: init done 2024-09-16T05:14:31.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa406d25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:31.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa406d25640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa4001ce620 0x7fa4001cea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa406d25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa406d25640 1 -- 172.21.15.114:0/4050528348 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fa40013ab40 con 0x7fa4001ce620 2024-09-16T05:14:31.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 1 -- 172.21.15.114:0/4050528348 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa3f80be070 con 0x7fa40006bcc0 2024-09-16T05:14:31.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fffff640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa4001ce620 0x7fa4001cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 1 --2- 172.21.15.114:0/4050528348 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa3ec037810 0x7fa3ec039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fffff640 10 monclient: get_auth_request con 0x7fa4001ce620 auth_method 0 2024-09-16T05:14:31.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:31.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:31.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:31.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 1 -- 172.21.15.114:0/4050528348 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(291..291 src has 1..291) ==== 5727+0+0 (secure 0 0 0) 0x7fa3f80e2090 con 0x7fa40006bcc0 2024-09-16T05:14:31.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 10 client.54671.objecter ms_dispatch 0x7fa400000ba0 osd_map(291..291 src has 1..291) 2024-09-16T05:14:31.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 3 client.54671.objecter handle_osd_map got epochs [291,291] > 0 2024-09-16T05:14:31.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 3 client.54671.objecter handle_osd_map decoding full epoch 291 2024-09-16T05:14:31.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:31.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:31.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 20 client.54671.objecter dump_active .. 0 homeless 2024-09-16T05:14:31.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa4052c2640 1 --2- 172.21.15.114:0/4050528348 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa3ec037810 0x7fa3ec039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa4052c2640 10 monclient: get_auth_request con 0x7fa3ec037810 auth_method 0 2024-09-16T05:14:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fffff640 10 monclient(hunting): handle_auth_done global_id 64381 payload 824 2024-09-16T05:14:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fffff640 1 -- 172.21.15.114:0/4050528348 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa40006c4a0 con 0x7fa4001ce620 2024-09-16T05:14:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 10 client.54671.objecter ms_handle_connect 0x7fa4001ce620 2024-09-16T05:14:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 10 client.54671.objecter resend_mon_ops 2024-09-16T05:14:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fffff640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa4001ce620 0x7fa4001cea00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa400137d70 tx=0x7fa3f00115a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 10 client.54671.objecter ms_handle_connect 0x7fa3ec037810 2024-09-16T05:14:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa4052c2640 1 --2- 172.21.15.114:0/4050528348 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa3ec037810 0x7fa3ec039cd0 secure :-1 s=READY pgs=3504 cs=0 l=1 rev1=1 crypto rx=0x7fa4001395e0 tx=0x7fa3f4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 1 -- 172.21.15.114:0/4050528348 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f001a020 con 0x7fa4001ce620 2024-09-16T05:14:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:14:31.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 10 monclient: dump: 2024-09-16T05:14:31.209 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.209 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.209 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.209 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.209 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.209 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.209 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.209 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.209 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.209 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 1 -- 172.21.15.114:0/4050528348 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa3f0016470 con 0x7fa4001ce620 2024-09-16T05:14:31.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:31.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:31.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 1 -- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa4001ce620 msgr2=0x7fa4001cea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa4001ce620 0x7fa4001cea00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa400137d70 tx=0x7fa3f00115a0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 1 -- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa4001ce620 msgr2=0x7fa4001cea00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:31.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.184+0000 7fa3fd7fa640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa4001ce620 0x7fa4001cea00 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa400137d70 tx=0x7fa3f00115a0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:14:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa4001cef40 0x7fa4001358a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 1 -- 172.21.15.114:0/4050528348 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fa40013a620 con 0x7fa4001cef40 2024-09-16T05:14:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fffff640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa4001cef40 0x7fa4001358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fffff640 10 monclient: get_auth_request con 0x7fa4001cef40 auth_method 0 2024-09-16T05:14:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fffff640 10 monclient(hunting): handle_auth_done global_id 64384 payload 824 2024-09-16T05:14:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fffff640 1 -- 172.21.15.114:0/4050528348 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa4000a36f0 con 0x7fa4001cef40 2024-09-16T05:14:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fffff640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa4001cef40 0x7fa4001358a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa3f001b000 tx=0x7fa3f0004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fd7fa640 10 client.54671.objecter ms_handle_connect 0x7fa4001cef40 2024-09-16T05:14:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fd7fa640 10 client.54671.objecter resend_mon_ops 2024-09-16T05:14:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fd7fa640 1 -- 172.21.15.114:0/4050528348 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f001a070 con 0x7fa4001cef40 2024-09-16T05:14:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:14:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fd7fa640 10 monclient: dump: 2024-09-16T05:14:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.213 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.214 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.214 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.214 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fd7fa640 1 -- 172.21.15.114:0/4050528348 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fa3f001f620 con 0x7fa4001cef40 2024-09-16T05:14:31.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:14:31.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:31.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fd7fa640 1 -- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa4001cef40 msgr2=0x7fa4001358a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fd7fa640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa4001cef40 0x7fa4001358a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa3f001b000 tx=0x7fa3f0004a50 comp rx=0 tx=0).stop 2024-09-16T05:14:31.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fd7fa640 1 -- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa4001cef40 msgr2=0x7fa4001358a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:31.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa3fd7fa640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa4001cef40 0x7fa4001358a0 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa3f001b000 tx=0x7fa3f0004a50 comp rx=0 tx=0).stop 2024-09-16T05:14:31.215 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:14:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:14:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:14:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:14:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 34, 2024-09-16T05:14:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 155329, 2024-09-16T05:14:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:14:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:14:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:14:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:14:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 4, 2024-09-16T05:14:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:14:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:14:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:14:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:14:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:14:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:14:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:14:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:14:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:14:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:14:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:14:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:14:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:14:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:14:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:14:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:14:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:14:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:14:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:14:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:14:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:14:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:14:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:14:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:14:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:14:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:14:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:14:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:14:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:14:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:14:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:14:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:14:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:14:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:14:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:14:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:14:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:14:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:14:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:14:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:14:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:14:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:14:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:14:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:14:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:14:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:14:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:14:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:14:31.228 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:14:31.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 10 librados: watch_flush enter 2024-09-16T05:14:31.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 10 librados: watch_flush exit 2024-09-16T05:14:31.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 20 client.54671.objecter shutdown clearing up homeless session... 2024-09-16T05:14:31.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 10 client.54671.objecter successfully canceled tick 2024-09-16T05:14:31.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 1 -- 172.21.15.114:0/4050528348 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa3ec037810 msgr2=0x7fa3ec039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 1 --2- 172.21.15.114:0/4050528348 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa3ec037810 0x7fa3ec039cd0 secure :-1 s=READY pgs=3504 cs=0 l=1 rev1=1 crypto rx=0x7fa4001395e0 tx=0x7fa3f4008040 comp rx=0 tx=0).stop 2024-09-16T05:14:31.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 10 monclient: shutdown 2024-09-16T05:14:31.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:31.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 1 -- 172.21.15.114:0/4050528348 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa40006bcc0 msgr2=0x7fa4001cd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa40006bcc0 0x7fa4001cd5a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa3f801aa90 tx=0x7fa3f8019180 comp rx=0 tx=0).stop 2024-09-16T05:14:31.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa4052c2640 1 -- 172.21.15.114:0/4050528348 reap_dead start 2024-09-16T05:14:31.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 1 -- 172.21.15.114:0/4050528348 shutdown_connections 2024-09-16T05:14:31.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 1 --2- 172.21.15.114:0/4050528348 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa40006bcc0 0x7fa4001cd5a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 1 -- 172.21.15.114:0/4050528348 >> 172.21.15.114:0/4050528348 conn(0x7fa4000a4210 msgr2=0x7fa4000a05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:31.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 1 -- 172.21.15.114:0/4050528348 shutdown_connections 2024-09-16T05:14:31.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 1 -- 172.21.15.114:0/4050528348 wait complete. 2024-09-16T05:14:31.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.192+0000 7fa406d25640 1 librados: shutdown 2024-09-16T05:14:31.231 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:14:31.231 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T05:14:31.231 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T05:14:31.231 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T05:14:31.231 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T05:14:31.232 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T05:14:31.232 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:14:31.244 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T05:14:31.244 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:14:31.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:14:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 10 monclient: get_monmap_and_config 2024-09-16T05:14:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 10 monclient: build_initial_monmap 2024-09-16T05:14:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 10 monclient: monmap: 2024-09-16T05:14:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:31.405862+0000 2024-09-16T05:14:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:31.405862+0000 2024-09-16T05:14:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:31.406 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 1 Processor -- start 2024-09-16T05:14:31.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 1 -- start start 2024-09-16T05:14:31.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 10 monclient: init 2024-09-16T05:14:31.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:31.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-16T05:14:31.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc58c06bcc0 0x7fc58c09f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 10 monclient(hunting): picked mon.noname-b con 0x7fc58c06bcc0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:31.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc58c0a3580 0x7fc58c06c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 10 monclient(hunting): picked mon.noname-a con 0x7fc58c0a3580 addr v2:172.21.15.114:3300/0 2024-09-16T05:14:31.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc58c06c5e0 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:31.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc58c06d120 con 0x7fc58c06c5e0 2024-09-16T05:14:31.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fc58c06ca50 con 0x7fc58c0a3580 2024-09-16T05:14:31.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc58c09afc0 con 0x7fc58c06bcc0 2024-09-16T05:14:31.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:31.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc5932bb640 10 monclient(hunting): authenticate will time out at 1356.035806s 2024-09-16T05:14:31.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc591057640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc58c06bcc0 msgr2=0x7fc58c09f070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:14:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc591057640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc58c06bcc0 0x7fc58c09f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.400+0000 7fc590856640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc58c0a3580 0x7fc58c06c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc58c0a3580 0x7fc58c06c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36866/0 (socket says 172.21.15.114:36866) 2024-09-16T05:14:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 1 -- 172.21.15.114:0/877612112 learned_addr learned my addr 172.21.15.114:0/877612112 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 10 monclient(hunting): get_auth_request con 0x7fc58c0a3580 auth_method 0 2024-09-16T05:14:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 10 monclient(hunting): handle_auth_done global_id 74184 payload 274 2024-09-16T05:14:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 1 -- 172.21.15.114:0/877612112 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc58c06bcc0 msgr2=0x7fc58c09f070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:14:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 1 --2- 172.21.15.114:0/877612112 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc58c06bcc0 0x7fc58c09f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 1 -- 172.21.15.114:0/877612112 >> v1:172.21.15.114:6789/0 conn(0x7fc58c06c5e0 legacy=0x7fc58c137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 10 monclient: _finish_hunting 0 2024-09-16T05:14:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 1 monclient: found mon.noname-a 2024-09-16T05:14:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 1 -- 172.21.15.114:0/877612112 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58c0a3220 con 0x7fc58c0a3580 2024-09-16T05:14:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc590856640 1 --2- 172.21.15.114:0/877612112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc58c0a3580 0x7fc58c06c0a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc580009870 tx=0x7fc58001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15555386d494eec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc57ffff640 1 -- 172.21.15.114:0/877612112 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5800b0070 con 0x7fc58c0a3580 2024-09-16T05:14:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc57ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc57ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc57ffff640 10 monclient: dump: 2024-09-16T05:14:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.419 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc57ffff640 10 monclient: _finish_auth 0 2024-09-16T05:14:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc57ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:14:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc57ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc57ffff640 1 -- 172.21.15.114:0/877612112 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5800ab600 con 0x7fc58c0a3580 2024-09-16T05:14:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 5 monclient: authenticate success, global_id 74184 2024-09-16T05:14:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc57ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc57ffff640 1 -- 172.21.15.114:0/877612112 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5800ab9d0 con 0x7fc58c0a3580 2024-09-16T05:14:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc57ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 4 set_mon_vals no callback set 2024-09-16T05:14:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc57ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc57ffff640 10 monclient: dump: 2024-09-16T05:14:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.421 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 10 monclient: shutdown 2024-09-16T05:14:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 1 -- 172.21.15.114:0/877612112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc58c0a3580 msgr2=0x7fc58c06c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 1 --2- 172.21.15.114:0/877612112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc58c0a3580 0x7fc58c06c0a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc580009870 tx=0x7fc58001c440 comp rx=0 tx=0).stop 2024-09-16T05:14:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 1 -- 172.21.15.114:0/877612112 shutdown_connections 2024-09-16T05:14:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 1 --2- 172.21.15.114:0/877612112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc58c0a3580 0x7fc58c06c0a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 1 --2- 172.21.15.114:0/877612112 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc58c06bcc0 0x7fc58c09f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 1 -- 172.21.15.114:0/877612112 >> 172.21.15.114:0/877612112 conn(0x7fc58c0a4280 msgr2=0x7fc58c0a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 1 -- 172.21.15.114:0/877612112 shutdown_connections 2024-09-16T05:14:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 1 -- 172.21.15.114:0/877612112 wait complete. 2024-09-16T05:14:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 10 monclient: build_initial_monmap 2024-09-16T05:14:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 10 monclient: monmap: 2024-09-16T05:14:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:31.423 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.404+0000 7fc5932bb640 1 librados: starting msgr at 2024-09-16T05:14:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 1 librados: starting objecter 2024-09-16T05:14:31.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 1 Processor -- start 2024-09-16T05:14:31.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 1 -- start start 2024-09-16T05:14:31.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 1 librados: setting wanted keys 2024-09-16T05:14:31.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 1 librados: calling monclient init 2024-09-16T05:14:31.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient: init 2024-09-16T05:14:31.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:31.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:14:31.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc58c06bcc0 0x7fc58c1cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient(hunting): picked mon.noname-c con 0x7fc58c06bcc0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:31.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc58c06c5e0 0x7fc58c0738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient(hunting): picked mon.noname-a con 0x7fc58c06c5e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:31.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc58c0a3580 0x7fc58c136070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient(hunting): picked mon.noname-b con 0x7fc58c0a3580 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:31.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c142530 con 0x7fc58c06c5e0 2024-09-16T05:14:31.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c1423b0 con 0x7fc58c0a3580 2024-09-16T05:14:31.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:31.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc58c1426b0 con 0x7fc58c06bcc0 2024-09-16T05:14:31.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:31.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient(hunting): authenticate will time out at 1356.040667s 2024-09-16T05:14:31.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc591057640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc58c06bcc0 msgr2=0x7fc58c1cf7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:14:31.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc591057640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc58c06bcc0 0x7fc58c1cf7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:31.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc58c06c5e0 0x7fc58c0738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc58c06c5e0 0x7fc58c0738b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36880/0 (socket says 172.21.15.114:36880) 2024-09-16T05:14:31.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 1 -- 172.21.15.114:0/500569463 learned_addr learned my addr 172.21.15.114:0/500569463 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:31.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 10 monclient(hunting): get_auth_request con 0x7fc58c06c5e0 auth_method 0 2024-09-16T05:14:31.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:31.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:31.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:31.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:31.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:31.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:31.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 10 monclient(hunting): handle_auth_done global_id 74187 payload 824 2024-09-16T05:14:31.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 1 -- 172.21.15.114:0/500569463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc58c06bcc0 msgr2=0x7fc58c1cf7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:14:31.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 1 --2- 172.21.15.114:0/500569463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc58c06bcc0 0x7fc58c1cf7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 1 -- 172.21.15.114:0/500569463 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc58c0a3580 msgr2=0x7fc58c136070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 1 --2- 172.21.15.114:0/500569463 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc58c0a3580 0x7fc58c136070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 10 monclient: _finish_hunting 0 2024-09-16T05:14:31.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 1 monclient: found mon.noname-a 2024-09-16T05:14:31.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:31.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:31.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 1 -- 172.21.15.114:0/500569463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58c0a53b0 con 0x7fc58c06c5e0 2024-09-16T05:14:31.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc590856640 1 --2- 172.21.15.114:0/500569463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc58c06c5e0 0x7fc58c0738b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc5800ae040 tx=0x7fc5800191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:31.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 10 client.?.objecter ms_handle_connect 0x7fc58c06c5e0 2024-09-16T05:14:31.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:31.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 1 -- 172.21.15.114:0/500569463 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5800b0050 con 0x7fc58c06c5e0 2024-09-16T05:14:31.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:31.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 10 monclient: dump: 2024-09-16T05:14:31.428 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.428 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.428 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.428 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.428 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.428 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.428 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.428 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.429 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.429 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 10 monclient: _finish_auth 0 2024-09-16T05:14:31.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:14:31.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:31.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 1 -- 172.21.15.114:0/500569463 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5800bc070 con 0x7fc58c06c5e0 2024-09-16T05:14:31.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 5 monclient: authenticate success, global_id 74187 2024-09-16T05:14:31.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient: _renew_subs 2024-09-16T05:14:31.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:31.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 1 -- 172.21.15.114:0/500569463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc58c072380 con 0x7fc58c06c5e0 2024-09-16T05:14:31.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:31.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 1 -- 172.21.15.114:0/500569463 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5800aac40 con 0x7fc58c06c5e0 2024-09-16T05:14:31.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:31.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:31.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 10 monclient: dump: 2024-09-16T05:14:31.430 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:31.430 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:31.430 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.430 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:31.430 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:31.430 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:31.430 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:31.431 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:31.431 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:31.431 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:31.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57f7fe640 4 set_mon_vals no callback set 2024-09-16T05:14:31.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 client.74187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:31.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient: _renew_subs 2024-09-16T05:14:31.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:31.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 1 -- 172.21.15.114:0/500569463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc58c137b30 con 0x7fc58c06c5e0 2024-09-16T05:14:31.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 1 librados: init done 2024-09-16T05:14:31.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 1 -- 172.21.15.114:0/500569463 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc5800c1460 con 0x7fc58c06c5e0 2024-09-16T05:14:31.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:31.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:31.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:31.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc5932bb640 1 -- 172.21.15.114:0/500569463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc58c09f070 con 0x7fc58c06c5e0 2024-09-16T05:14:31.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 1 --2- 172.21.15.114:0/500569463 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc5700377c0 0x7fc570039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:31.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 1 -- 172.21.15.114:0/500569463 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(291..291 src has 1..291) ==== 5727+0+0 (secure 0 0 0) 0x7fc5800e3a40 con 0x7fc58c06c5e0 2024-09-16T05:14:31.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 10 client.74187.objecter ms_dispatch 0x7fc58c000ba0 osd_map(291..291 src has 1..291) 2024-09-16T05:14:31.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 3 client.74187.objecter handle_osd_map got epochs [291,291] > 0 2024-09-16T05:14:31.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 3 client.74187.objecter handle_osd_map decoding full epoch 291 2024-09-16T05:14:31.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 20 client.74187.objecter dump_active .. 0 homeless 2024-09-16T05:14:31.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc591057640 1 --2- 172.21.15.114:0/500569463 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc5700377c0 0x7fc570039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:31.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc591057640 10 monclient: get_auth_request con 0x7fc5700377c0 auth_method 0 2024-09-16T05:14:31.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc591057640 1 --2- 172.21.15.114:0/500569463 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc5700377c0 0x7fc570039c80 secure :-1 s=READY pgs=3505 cs=0 l=1 rev1=1 crypto rx=0x7fc588003990 tx=0x7fc588006d90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:31.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.408+0000 7fc57dffb640 10 client.74187.objecter ms_handle_connect 0x7fc5700377c0 2024-09-16T05:14:31.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.412+0000 7fc57dffb640 1 -- 172.21.15.114:0/500569463 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc580016150 con 0x7fc58c06c5e0 2024-09-16T05:14:31.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.412+0000 7fc57dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:31.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.412+0000 7fc57dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.624+0000 7fc5932bb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:14:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.624+0000 7fc5932bb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.624+0000 7fc5932bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.624+0000 7fc5932bb640 1 -- 172.21.15.114:0/500569463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc58c09e7a0 con 0x7fc58c06c5e0 2024-09-16T05:14:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.624+0000 7fc57dffb640 1 -- 172.21.15.114:0/500569463 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fc5800bf360 con 0x7fc58c06c5e0 2024-09-16T05:14:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.624+0000 7fc57dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.624+0000 7fc57dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:31.627 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:14:31.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 10 librados: watch_flush enter 2024-09-16T05:14:31.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 10 librados: watch_flush exit 2024-09-16T05:14:31.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 20 client.74187.objecter shutdown clearing up homeless session... 2024-09-16T05:14:31.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 10 client.74187.objecter successfully canceled tick 2024-09-16T05:14:31.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 1 -- 172.21.15.114:0/500569463 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc5700377c0 msgr2=0x7fc570039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 1 --2- 172.21.15.114:0/500569463 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc5700377c0 0x7fc570039c80 secure :-1 s=READY pgs=3505 cs=0 l=1 rev1=1 crypto rx=0x7fc588003990 tx=0x7fc588006d90 comp rx=0 tx=0).stop 2024-09-16T05:14:31.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 10 monclient: shutdown 2024-09-16T05:14:31.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:31.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 1 -- 172.21.15.114:0/500569463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc58c06c5e0 msgr2=0x7fc58c0738b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:31.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 1 --2- 172.21.15.114:0/500569463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc58c06c5e0 0x7fc58c0738b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc5800ae040 tx=0x7fc5800191e0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 1 -- 172.21.15.114:0/500569463 shutdown_connections 2024-09-16T05:14:31.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 1 --2- 172.21.15.114:0/500569463 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc5700377c0 0x7fc570039c80 unknown :-1 s=CLOSED pgs=3505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 1 --2- 172.21.15.114:0/500569463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc58c06c5e0 0x7fc58c0738b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 1 --2- 172.21.15.114:0/500569463 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc58c0a3580 0x7fc58c136070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 1 --2- 172.21.15.114:0/500569463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc58c06bcc0 0x7fc58c1cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:31.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 1 -- 172.21.15.114:0/500569463 >> 172.21.15.114:0/500569463 conn(0x7fc58c0a4280 msgr2=0x7fc58c0a04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:31.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 1 -- 172.21.15.114:0/500569463 shutdown_connections 2024-09-16T05:14:31.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 1 -- 172.21.15.114:0/500569463 wait complete. 2024-09-16T05:14:31.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:31.628+0000 7fc5932bb640 1 librados: shutdown 2024-09-16T05:14:34.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:14:34.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.772+0000 7fc4078f5640 10 monclient: get_monmap_and_config 2024-09-16T05:14:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.772+0000 7fc4078f5640 10 monclient: build_initial_monmap 2024-09-16T05:14:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.772+0000 7fc4078f5640 10 monclient: monmap: 2024-09-16T05:14:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:34.778927+0000 2024-09-16T05:14:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:34.778927+0000 2024-09-16T05:14:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:34.778 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:34.778 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:34.778 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:34.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.772+0000 7fc4078f5640 1 Processor -- start 2024-09-16T05:14:34.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.772+0000 7fc4078f5640 1 -- start start 2024-09-16T05:14:34.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.772+0000 7fc4078f5640 10 monclient: init 2024-09-16T05:14:34.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc4078f5640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:34.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc4078f5640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-16T05:14:34.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc4078f5640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc40009b6c0 0x7fc4000a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:34.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc4078f5640 10 monclient(hunting): picked mon.noname-b con 0x7fc40009b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:34.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc4078f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc4000a1400 addr v1:172.21.15.149:6789/0 2024-09-16T05:14:34.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc4078f5640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc4000a17e0 0x7fc40013cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:34.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc4078f5640 10 monclient(hunting): picked mon.noname-c con 0x7fc4000a17e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:14:34.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc4078f5640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc40009d4e0 con 0x7fc4000a1400 2024-09-16T05:14:34.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc4078f5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:34.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc4078f5640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fc40009e390 con 0x7fc4000a17e0 2024-09-16T05:14:34.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc4078f5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:34.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc4078f5640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc40009ece0 con 0x7fc40009b6c0 2024-09-16T05:14:34.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc4078f5640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:34.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc4078f5640 10 monclient(hunting): authenticate will time out at 1359.408145s 2024-09-16T05:14:34.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405691640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc40009b6c0 msgr2=0x7fc4000a0ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:14:34.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405691640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc40009b6c0 0x7fc4000a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:34.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc4000a17e0 0x7fc40013cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:34.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc404e90640 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7fc4000a1400 0x7fc400139640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.114:35432/0 (socket says 172.21.15.114:35432) 2024-09-16T05:14:34.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc4000a17e0 0x7fc40013cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.114:60474/0 (socket says 172.21.15.114:60474) 2024-09-16T05:14:34.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc404e90640 1 -- 172.21.15.114:0/1924331736 learned_addr learned my addr 172.21.15.114:0/1924331736 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:14:34.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 10 monclient(hunting): get_auth_request con 0x7fc4000a17e0 auth_method 0 2024-09-16T05:14:34.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:34.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:34.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:34.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:34.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:34.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:34.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 10 monclient(hunting): handle_auth_done global_id 64387 payload 274 2024-09-16T05:14:34.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 1 -- 172.21.15.114:0/1924331736 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc40009b6c0 msgr2=0x7fc4000a0ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:14:34.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 1 --2- 172.21.15.114:0/1924331736 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc40009b6c0 0x7fc4000a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:34.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 1 -- 172.21.15.114:0/1924331736 >> v1:172.21.15.149:6789/0 conn(0x7fc4000a1400 legacy=0x7fc400139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:34.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 10 monclient: _finish_hunting 0 2024-09-16T05:14:34.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 1 monclient: found mon.noname-c 2024-09-16T05:14:34.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:34.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:14:34.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 1 -- 172.21.15.114:0/1924331736 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc40009f430 con 0x7fc4000a17e0 2024-09-16T05:14:34.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.776+0000 7fc405e92640 1 --2- 172.21.15.114:0/1924331736 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4000a17e0 0x7fc40013cd70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0097c0 tx=0x7fc3fc01c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e680208e31d8f38 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:34.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f7fff640 1 -- 172.21.15.114:0/1924331736 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc0b0070 con 0x7fc4000a17e0 2024-09-16T05:14:34.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:34.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:14:34.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f7fff640 10 monclient: dump: 2024-09-16T05:14:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:34.789 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f7fff640 10 monclient: _finish_auth 0 2024-09-16T05:14:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:14:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f7fff640 1 -- 172.21.15.114:0/1924331736 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3fc0ab600 con 0x7fc4000a17e0 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f7fff640 1 -- 172.21.15.114:0/1924331736 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc0ab9d0 con 0x7fc4000a17e0 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 5 monclient: authenticate success, global_id 64387 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 4 set_mon_vals no callback set 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f7fff640 10 monclient: dump: 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:34.791 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:34.791 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:34.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient: shutdown 2024-09-16T05:14:34.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:34.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 -- 172.21.15.114:0/1924331736 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4000a17e0 msgr2=0x7fc40013cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:34.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 --2- 172.21.15.114:0/1924331736 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4000a17e0 0x7fc40013cd70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0097c0 tx=0x7fc3fc01c380 comp rx=0 tx=0).stop 2024-09-16T05:14:34.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 -- 172.21.15.114:0/1924331736 shutdown_connections 2024-09-16T05:14:34.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 --2- 172.21.15.114:0/1924331736 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4000a17e0 0x7fc40013cd70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:34.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 --2- 172.21.15.114:0/1924331736 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc40009b6c0 0x7fc4000a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:34.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 -- 172.21.15.114:0/1924331736 >> 172.21.15.114:0/1924331736 conn(0x7fc4000a1fb0 msgr2=0x7fc4000a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:34.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 -- 172.21.15.114:0/1924331736 shutdown_connections 2024-09-16T05:14:34.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 -- 172.21.15.114:0/1924331736 wait complete. 2024-09-16T05:14:34.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient: build_initial_monmap 2024-09-16T05:14:34.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient: monmap: 2024-09-16T05:14:34.791 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:34.792 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:34.792 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:34.792 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:34.792 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:34.792 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:34.792 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:34.792 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:34.792 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:34.792 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:34.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 librados: starting msgr at 2024-09-16T05:14:34.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 librados: starting objecter 2024-09-16T05:14:34.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 Processor -- start 2024-09-16T05:14:34.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 -- start start 2024-09-16T05:14:34.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 librados: setting wanted keys 2024-09-16T05:14:34.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 librados: calling monclient init 2024-09-16T05:14:34.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient: init 2024-09-16T05:14:34.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:34.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:14:34.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc40009b6c0 0x7fc400098730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:34.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient(hunting): picked mon.noname-c con 0x7fc40009b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:34.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4000a1400 0x7fc400091d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:34.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient(hunting): picked mon.noname-b con 0x7fc4000a1400 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:34.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4000a17e0 0x7fc400092290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:34.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient(hunting): picked mon.noname-a con 0x7fc4000a17e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:34.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:34.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc4000a1c50 con 0x7fc4000a17e0 2024-09-16T05:14:34.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:34.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fc40009cad0 con 0x7fc4000a1400 2024-09-16T05:14:34.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:34.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc40013fad0 con 0x7fc40009b6c0 2024-09-16T05:14:34.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:34.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient(hunting): authenticate will time out at 1359.414642s 2024-09-16T05:14:34.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405691640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc40009b6c0 msgr2=0x7fc400098730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:14:34.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405691640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc40009b6c0 0x7fc400098730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:34.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4000a17e0 0x7fc400092290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:34.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4000a17e0 0x7fc400092290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36890/0 (socket says 172.21.15.114:36890) 2024-09-16T05:14:34.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 1 -- 172.21.15.114:0/217261711 learned_addr learned my addr 172.21.15.114:0/217261711 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:34.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 10 monclient(hunting): get_auth_request con 0x7fc4000a17e0 auth_method 0 2024-09-16T05:14:34.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:34.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:34.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:34.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:34.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:34.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:34.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc404e90640 1 --2- 172.21.15.114:0/217261711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4000a1400 0x7fc400091d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:34.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 10 monclient(hunting): handle_auth_done global_id 74190 payload 824 2024-09-16T05:14:34.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 1 -- 172.21.15.114:0/217261711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc40009b6c0 msgr2=0x7fc400098730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:14:34.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 1 --2- 172.21.15.114:0/217261711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc40009b6c0 0x7fc400098730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:34.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 1 -- 172.21.15.114:0/217261711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4000a1400 msgr2=0x7fc400091d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:34.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 1 --2- 172.21.15.114:0/217261711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4000a1400 0x7fc400091d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:34.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 10 monclient: _finish_hunting 0 2024-09-16T05:14:34.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 1 monclient: found mon.noname-a 2024-09-16T05:14:34.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:34.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 1 -- 172.21.15.114:0/217261711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc400072dc0 con 0x7fc4000a17e0 2024-09-16T05:14:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc404e90640 10 monclient: get_auth_request con 0x7fc4000a1400 auth_method 0 2024-09-16T05:14:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc404e90640 1 --2- 172.21.15.114:0/217261711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4000a1400 0x7fc400091d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:14:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc405e92640 1 --2- 172.21.15.114:0/217261711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4000a17e0 0x7fc400092290 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0ae040 tx=0x7fc3fc01abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f5ffb640 10 client.?.objecter ms_handle_connect 0x7fc4000a17e0 2024-09-16T05:14:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f5ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f5ffb640 1 -- 172.21.15.114:0/217261711 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc0b0050 con 0x7fc4000a17e0 2024-09-16T05:14:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f5ffb640 10 monclient: dump: 2024-09-16T05:14:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:34.797 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:34.797 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:34.797 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:34.797 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:34.797 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:34.797 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:34.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:14:34.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:14:34.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:34.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f5ffb640 1 -- 172.21.15.114:0/217261711 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3fc0bc070 con 0x7fc4000a17e0 2024-09-16T05:14:34.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 5 monclient: authenticate success, global_id 74190 2024-09-16T05:14:34.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient: _renew_subs 2024-09-16T05:14:34.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:34.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 -- 172.21.15.114:0/217261711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc400094510 con 0x7fc4000a17e0 2024-09-16T05:14:34.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:34.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f5ffb640 1 -- 172.21.15.114:0/217261711 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc0b5810 con 0x7fc4000a17e0 2024-09-16T05:14:34.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:34.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:34.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f5ffb640 10 monclient: dump: 2024-09-16T05:14:34.798 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:34.798 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:34.798 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:34.798 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:34.798 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:34.798 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:34.798 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:34.798 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:34.798 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:34.798 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:34.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc3f77fe640 4 set_mon_vals no callback set 2024-09-16T05:14:34.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 client.74190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:34.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient: _renew_subs 2024-09-16T05:14:34.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:34.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 -- 172.21.15.114:0/217261711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc400095400 con 0x7fc4000a17e0 2024-09-16T05:14:34.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.780+0000 7fc4078f5640 1 librados: init done 2024-09-16T05:14:34.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.784+0000 7fc3f5ffb640 1 -- 172.21.15.114:0/217261711 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc3fc0b59b0 con 0x7fc4000a17e0 2024-09-16T05:14:34.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.784+0000 7fc4078f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:34.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.784+0000 7fc4078f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:34.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.784+0000 7fc4078f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:34.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.784+0000 7fc4078f5640 1 -- 172.21.15.114:0/217261711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4000a0ec0 con 0x7fc4000a17e0 2024-09-16T05:14:34.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.784+0000 7fc3f5ffb640 1 --2- 172.21.15.114:0/217261711 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc3e40377c0 0x7fc3e4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:34.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.784+0000 7fc3f5ffb640 1 -- 172.21.15.114:0/217261711 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(291..291 src has 1..291) ==== 5727+0+0 (secure 0 0 0) 0x7fc3fc0e3940 con 0x7fc4000a17e0 2024-09-16T05:14:34.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.784+0000 7fc3f5ffb640 10 client.74190.objecter ms_dispatch 0x7fc400000ba0 osd_map(291..291 src has 1..291) 2024-09-16T05:14:34.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.784+0000 7fc3f5ffb640 3 client.74190.objecter handle_osd_map got epochs [291,291] > 0 2024-09-16T05:14:34.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.784+0000 7fc3f5ffb640 3 client.74190.objecter handle_osd_map decoding full epoch 291 2024-09-16T05:14:34.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.784+0000 7fc3f5ffb640 20 client.74190.objecter dump_active .. 0 homeless 2024-09-16T05:14:34.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.784+0000 7fc405691640 1 --2- 172.21.15.114:0/217261711 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc3e40377c0 0x7fc3e4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:34.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.784+0000 7fc405691640 10 monclient: get_auth_request con 0x7fc3e40377c0 auth_method 0 2024-09-16T05:14:34.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.784+0000 7fc405691640 1 --2- 172.21.15.114:0/217261711 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc3e40377c0 0x7fc3e4039c80 secure :-1 s=READY pgs=3506 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0003990 tx=0x7fc3f0006d90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:34.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.784+0000 7fc3f5ffb640 10 client.74190.objecter ms_handle_connect 0x7fc3e40377c0 2024-09-16T05:14:34.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.788+0000 7fc3f5ffb640 1 -- 172.21.15.114:0/217261711 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc3fc016090 con 0x7fc4000a17e0 2024-09-16T05:14:34.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.788+0000 7fc3f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:34.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:34.788+0000 7fc3f5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 1 -- 172.21.15.114:0/217261711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc40008e710 con 0x7fc4000a17e0 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc3f5ffb640 1 -- 172.21.15.114:0/217261711 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fc3fc0aae10 con 0x7fc4000a17e0 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc3f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc3f5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 10 librados: watch_flush enter 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 10 librados: watch_flush exit 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 20 client.74190.objecter shutdown clearing up homeless session... 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 10 client.74190.objecter successfully canceled tick 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 1 -- 172.21.15.114:0/217261711 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc3e40377c0 msgr2=0x7fc3e4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 1 --2- 172.21.15.114:0/217261711 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc3e40377c0 0x7fc3e4039c80 secure :-1 s=READY pgs=3506 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0003990 tx=0x7fc3f0006d90 comp rx=0 tx=0).stop 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 10 monclient: shutdown 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 1 -- 172.21.15.114:0/217261711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4000a17e0 msgr2=0x7fc400092290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 1 --2- 172.21.15.114:0/217261711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4000a17e0 0x7fc400092290 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0ae040 tx=0x7fc3fc01abc0 comp rx=0 tx=0).stop 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 1 -- 172.21.15.114:0/217261711 shutdown_connections 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 1 --2- 172.21.15.114:0/217261711 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc3e40377c0 0x7fc3e4039c80 unknown :-1 s=CLOSED pgs=3506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 1 --2- 172.21.15.114:0/217261711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4000a17e0 0x7fc400092290 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 1 --2- 172.21.15.114:0/217261711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4000a1400 0x7fc400091d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 1 --2- 172.21.15.114:0/217261711 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc40009b6c0 0x7fc400098730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 1 -- 172.21.15.114:0/217261711 >> 172.21.15.114:0/217261711 conn(0x7fc4000a1fb0 msgr2=0x7fc4000a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 1 -- 172.21.15.114:0/217261711 shutdown_connections 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 1 -- 172.21.15.114:0/217261711 wait complete. 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:35.004+0000 7fc4078f5640 1 librados: shutdown 2024-09-16T05:14:38.025 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:14:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 10 monclient: get_monmap_and_config 2024-09-16T05:14:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 10 monclient: build_initial_monmap 2024-09-16T05:14:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 10 monclient: monmap: 2024-09-16T05:14:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:38.154364+0000 2024-09-16T05:14:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:38.154364+0000 2024-09-16T05:14:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:38.153 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 1 Processor -- start 2024-09-16T05:14:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 1 -- start start 2024-09-16T05:14:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 10 monclient: init 2024-09-16T05:14:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-16T05:14:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f601c1376e0 addr v1:172.21.15.149:6789/0 2024-09-16T05:14:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f601c1370a0 0x7f601c137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 10 monclient(hunting): picked mon.noname-b con 0x7f601c1370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f601c0a5d00 0x7f601c0a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 10 monclient(hunting): picked mon.noname-a con 0x7f601c0a5d00 addr v2:172.21.15.114:3300/0 2024-09-16T05:14:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601c09bd20 con 0x7f601c1376e0 2024-09-16T05:14:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f601c138000 con 0x7f601c0a5d00 2024-09-16T05:14:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f601c09c5c0 con 0x7f601c1370a0 2024-09-16T05:14:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6024622640 10 monclient(hunting): authenticate will time out at 1362.783606s 2024-09-16T05:14:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6021bbd640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f601c1370a0 msgr2=0x7f601c137ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:14:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6021bbd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f601c1370a0 0x7f601c137ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6022bbf640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f601c0a5d00 0x7f601c0a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6022bbf640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f601c0a5d00 0x7f601c0a4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41330/0 (socket says 172.21.15.114:41330) 2024-09-16T05:14:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6022bbf640 1 -- 172.21.15.114:0/2488029158 learned_addr learned my addr 172.21.15.114:0/2488029158 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6022bbf640 10 monclient(hunting): get_auth_request con 0x7f601c0a5d00 auth_method 0 2024-09-16T05:14:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6022bbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6022bbf640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.148+0000 7f6022bbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.152+0000 7f6022bbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.152+0000 7f6022bbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.152+0000 7f6022bbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.152+0000 7f6022bbf640 10 monclient(hunting): handle_auth_done global_id 74193 payload 274 2024-09-16T05:14:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.152+0000 7f6022bbf640 1 -- 172.21.15.114:0/2488029158 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f601c1370a0 msgr2=0x7f601c137ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:14:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.152+0000 7f6022bbf640 1 --2- 172.21.15.114:0/2488029158 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f601c1370a0 0x7f601c137ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.152+0000 7f6022bbf640 1 -- 172.21.15.114:0/2488029158 >> v1:172.21.15.149:6789/0 conn(0x7f601c1376e0 legacy=0x7f601c0a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.152+0000 7f6022bbf640 10 monclient: _finish_hunting 0 2024-09-16T05:14:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.152+0000 7f6022bbf640 1 monclient: found mon.noname-a 2024-09-16T05:14:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.152+0000 7f6022bbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.152+0000 7f6022bbf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.152+0000 7f6022bbf640 1 -- 172.21.15.114:0/2488029158 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601c09c000 con 0x7f601c0a5d00 2024-09-16T05:14:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6022bbf640 1 --2- 172.21.15.114:0/2488029158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601c0a5d00 0x7f601c0a4980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6014009870 tx=0x7f601401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34c95bdc4c6ec27 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f60213bc640 1 -- 172.21.15.114:0/2488029158 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60140b0070 con 0x7f601c0a5d00 2024-09-16T05:14:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f60213bc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f60213bc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f60213bc640 10 monclient: dump: 2024-09-16T05:14:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:38.162 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f60213bc640 10 monclient: _finish_auth 0 2024-09-16T05:14:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f60213bc640 10 monclient: _check_auth_tickets 2024-09-16T05:14:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f60213bc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 5 monclient: authenticate success, global_id 74193 2024-09-16T05:14:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f60213bc640 1 -- 172.21.15.114:0/2488029158 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6014004230 con 0x7f601c0a5d00 2024-09-16T05:14:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f60213bc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 4 set_mon_vals no callback set 2024-09-16T05:14:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f60213bc640 1 -- 172.21.15.114:0/2488029158 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6014003620 con 0x7f601c0a5d00 2024-09-16T05:14:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f60213bc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f60213bc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f60213bc640 10 monclient: dump: 2024-09-16T05:14:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:38.164 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 10 monclient: shutdown 2024-09-16T05:14:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 -- 172.21.15.114:0/2488029158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601c0a5d00 msgr2=0x7f601c0a4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 --2- 172.21.15.114:0/2488029158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601c0a5d00 0x7f601c0a4980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6014009870 tx=0x7f601401c440 comp rx=0 tx=0).stop 2024-09-16T05:14:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 -- 172.21.15.114:0/2488029158 shutdown_connections 2024-09-16T05:14:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 --2- 172.21.15.114:0/2488029158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601c0a5d00 0x7f601c0a4980 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 --2- 172.21.15.114:0/2488029158 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f601c1370a0 0x7f601c137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 -- 172.21.15.114:0/2488029158 >> 172.21.15.114:0/2488029158 conn(0x7f601c0a9a40 msgr2=0x7f601c0a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 -- 172.21.15.114:0/2488029158 shutdown_connections 2024-09-16T05:14:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 -- 172.21.15.114:0/2488029158 wait complete. 2024-09-16T05:14:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 10 monclient: build_initial_monmap 2024-09-16T05:14:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 10 monclient: monmap: 2024-09-16T05:14:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:38.166 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 librados: starting msgr at 2024-09-16T05:14:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 librados: starting objecter 2024-09-16T05:14:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 Processor -- start 2024-09-16T05:14:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 -- start start 2024-09-16T05:14:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 librados: setting wanted keys 2024-09-16T05:14:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 librados: calling monclient init 2024-09-16T05:14:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 10 monclient: init 2024-09-16T05:14:38.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:38.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:14:38.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f601c0a5d00 0x7f601c1cf190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:38.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 10 monclient(hunting): picked mon.noname-b con 0x7f601c0a5d00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:38.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f601c1370a0 0x7f601c136290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 10 monclient(hunting): picked mon.noname-c con 0x7f601c1370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601c1376e0 0x7f601c13c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 10 monclient(hunting): picked mon.noname-a con 0x7f601c1376e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f601c141d80 con 0x7f601c1376e0 2024-09-16T05:14:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f601c141c00 con 0x7f601c0a5d00 2024-09-16T05:14:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f601c141f00 con 0x7f601c1370a0 2024-09-16T05:14:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6024622640 10 monclient(hunting): authenticate will time out at 1362.791750s 2024-09-16T05:14:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6021bbd640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f601c1370a0 msgr2=0x7f601c136290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:14:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6021bbd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f601c1370a0 0x7f601c136290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6022bbf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601c1376e0 0x7f601c13c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6022bbf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601c1376e0 0x7f601c13c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41338/0 (socket says 172.21.15.114:41338) 2024-09-16T05:14:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6022bbf640 1 -- 172.21.15.114:0/2158594763 learned_addr learned my addr 172.21.15.114:0/2158594763 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6022bbf640 10 monclient(hunting): get_auth_request con 0x7f601c1376e0 auth_method 0 2024-09-16T05:14:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6022bbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6022bbf640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6022bbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6022bbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6022bbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.156+0000 7f6022bbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f60223be640 1 --2- 172.21.15.114:0/2158594763 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f601c0a5d00 0x7f601c1cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f60223be640 10 monclient(hunting): get_auth_request con 0x7f601c0a5d00 auth_method 0 2024-09-16T05:14:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f60223be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f60223be640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f60223be640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6022bbf640 10 monclient(hunting): handle_auth_done global_id 74196 payload 824 2024-09-16T05:14:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6022bbf640 1 -- 172.21.15.114:0/2158594763 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f601c1370a0 msgr2=0x7f601c136290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:14:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6022bbf640 1 --2- 172.21.15.114:0/2158594763 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f601c1370a0 0x7f601c136290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6022bbf640 1 -- 172.21.15.114:0/2158594763 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f601c0a5d00 msgr2=0x7f601c1cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6022bbf640 1 --2- 172.21.15.114:0/2158594763 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f601c0a5d00 0x7f601c1cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6022bbf640 10 monclient: _finish_hunting 0 2024-09-16T05:14:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6022bbf640 1 monclient: found mon.noname-a 2024-09-16T05:14:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6022bbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6022bbf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6022bbf640 1 -- 172.21.15.114:0/2158594763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601c120940 con 0x7f601c1376e0 2024-09-16T05:14:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6022bbf640 1 --2- 172.21.15.114:0/2158594763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601c1376e0 0x7f601c13c110 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f60140ae040 tx=0x7f60140191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:38.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 10 client.?.objecter ms_handle_connect 0x7f601c1376e0 2024-09-16T05:14:38.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:38.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 1 -- 172.21.15.114:0/2158594763 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60140b9020 con 0x7f601c1376e0 2024-09-16T05:14:38.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:38.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:38.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 10 monclient: dump: 2024-09-16T05:14:38.175 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:38.175 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:38.175 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:38.175 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:38.175 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:38.175 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:38.175 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:38.176 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:38.176 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:38.176 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:38.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 10 monclient: _finish_auth 0 2024-09-16T05:14:38.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 10 monclient: _check_auth_tickets 2024-09-16T05:14:38.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:38.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6024622640 5 monclient: authenticate success, global_id 74196 2024-09-16T05:14:38.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 1 -- 172.21.15.114:0/2158594763 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60140b1040 con 0x7f601c1376e0 2024-09-16T05:14:38.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:38.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 1 -- 172.21.15.114:0/2158594763 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60140b0050 con 0x7f601c1376e0 2024-09-16T05:14:38.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6024622640 10 monclient: _renew_subs 2024-09-16T05:14:38.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6024622640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:38.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6024622640 1 -- 172.21.15.114:0/2158594763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f601c120c20 con 0x7f601c1376e0 2024-09-16T05:14:38.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600ffff640 4 set_mon_vals no callback set 2024-09-16T05:14:38.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:38.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:38.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 10 monclient: dump: 2024-09-16T05:14:38.177 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:38.177 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:38.177 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:38.177 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:38.177 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:38.178 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:38.178 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:38.178 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:38.178 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:38.178 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:38.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6024622640 10 client.74196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:38.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6024622640 10 monclient: _renew_subs 2024-09-16T05:14:38.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6024622640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:38.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6024622640 1 -- 172.21.15.114:0/2158594763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f601c0a2650 con 0x7f601c1376e0 2024-09-16T05:14:38.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6024622640 1 librados: init done 2024-09-16T05:14:38.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 1 -- 172.21.15.114:0/2158594763 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f60140bc020 con 0x7f601c1376e0 2024-09-16T05:14:38.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6024622640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:38.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6024622640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:38.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6024622640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:38.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f6024622640 1 -- 172.21.15.114:0/2158594763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f601c137ac0 con 0x7f601c1376e0 2024-09-16T05:14:38.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 1 --2- 172.21.15.114:0/2158594763 >> v2:172.21.15.149:6812/1185531135 conn(0x7f60000377c0 0x7f6000039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:38.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 1 -- 172.21.15.114:0/2158594763 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(291..291 src has 1..291) ==== 5727+0+0 (secure 0 0 0) 0x7f60140e2ec0 con 0x7f601c1376e0 2024-09-16T05:14:38.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 10 client.74196.objecter ms_dispatch 0x7f601c000ba0 osd_map(291..291 src has 1..291) 2024-09-16T05:14:38.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 3 client.74196.objecter handle_osd_map got epochs [291,291] > 0 2024-09-16T05:14:38.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 3 client.74196.objecter handle_osd_map decoding full epoch 291 2024-09-16T05:14:38.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 20 client.74196.objecter dump_active .. 0 homeless 2024-09-16T05:14:38.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f60223be640 1 --2- 172.21.15.114:0/2158594763 >> v2:172.21.15.149:6812/1185531135 conn(0x7f60000377c0 0x7f6000039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:38.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f60223be640 10 monclient: get_auth_request con 0x7f60000377c0 auth_method 0 2024-09-16T05:14:38.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f60223be640 1 --2- 172.21.15.114:0/2158594763 >> v2:172.21.15.149:6812/1185531135 conn(0x7f60000377c0 0x7f6000039c80 secure :-1 s=READY pgs=3507 cs=0 l=1 rev1=1 crypto rx=0x7f6010001650 tx=0x7f6010009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:38.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.160+0000 7f600effd640 10 client.74196.objecter ms_handle_connect 0x7f60000377c0 2024-09-16T05:14:38.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.164+0000 7f600effd640 1 -- 172.21.15.114:0/2158594763 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f60140c3030 con 0x7f601c1376e0 2024-09-16T05:14:38.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.164+0000 7f600effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:38.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.164+0000 7f600effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:38.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.380+0000 7f6024622640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:14:38.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.380+0000 7f6024622640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:38.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:38.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 1 -- 172.21.15.114:0/2158594763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f601c073d20 con 0x7f601c1376e0 2024-09-16T05:14:38.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f600effd640 1 -- 172.21.15.114:0/2158594763 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f6014016150 con 0x7f601c1376e0 2024-09-16T05:14:38.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f600effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:38.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f600effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:38.387 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:14:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 10 librados: watch_flush enter 2024-09-16T05:14:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 10 librados: watch_flush exit 2024-09-16T05:14:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 20 client.74196.objecter shutdown clearing up homeless session... 2024-09-16T05:14:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 10 client.74196.objecter successfully canceled tick 2024-09-16T05:14:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 1 -- 172.21.15.114:0/2158594763 >> v2:172.21.15.149:6812/1185531135 conn(0x7f60000377c0 msgr2=0x7f6000039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 1 --2- 172.21.15.114:0/2158594763 >> v2:172.21.15.149:6812/1185531135 conn(0x7f60000377c0 0x7f6000039c80 secure :-1 s=READY pgs=3507 cs=0 l=1 rev1=1 crypto rx=0x7f6010001650 tx=0x7f6010009040 comp rx=0 tx=0).stop 2024-09-16T05:14:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 10 monclient: shutdown 2024-09-16T05:14:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 1 -- 172.21.15.114:0/2158594763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601c1376e0 msgr2=0x7f601c13c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 1 --2- 172.21.15.114:0/2158594763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601c1376e0 0x7f601c13c110 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f60140ae040 tx=0x7f60140191e0 comp rx=0 tx=0).stop 2024-09-16T05:14:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 1 -- 172.21.15.114:0/2158594763 shutdown_connections 2024-09-16T05:14:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 1 --2- 172.21.15.114:0/2158594763 >> v2:172.21.15.149:6812/1185531135 conn(0x7f60000377c0 0x7f6000039c80 unknown :-1 s=CLOSED pgs=3507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 1 --2- 172.21.15.114:0/2158594763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601c1376e0 0x7f601c13c110 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 1 --2- 172.21.15.114:0/2158594763 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f601c1370a0 0x7f601c136290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 1 --2- 172.21.15.114:0/2158594763 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f601c0a5d00 0x7f601c1cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 1 -- 172.21.15.114:0/2158594763 >> 172.21.15.114:0/2158594763 conn(0x7f601c0a9a40 msgr2=0x7f601c0a7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 1 -- 172.21.15.114:0/2158594763 shutdown_connections 2024-09-16T05:14:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 1 -- 172.21.15.114:0/2158594763 wait complete. 2024-09-16T05:14:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:38.384+0000 7f6024622640 1 librados: shutdown 2024-09-16T05:14:41.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:14:41.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 10 monclient: get_monmap_and_config 2024-09-16T05:14:41.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 10 monclient: build_initial_monmap 2024-09-16T05:14:41.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 10 monclient: monmap: 2024-09-16T05:14:41.535 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:41.535 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:41.536 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:41.537434+0000 2024-09-16T05:14:41.536 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:41.537434+0000 2024-09-16T05:14:41.536 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:41.536 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:41.536 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:41.536 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:41.536 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:41.536 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:41.536 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:41.536 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:41.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 1 Processor -- start 2024-09-16T05:14:41.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 1 -- start start 2024-09-16T05:14:41.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 10 monclient: init 2024-09-16T05:14:41.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:41.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T05:14:41.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f827c09b6c0 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:41.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827c0a15d0 0x7f827c13a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:41.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 10 monclient(hunting): picked mon.noname-b con 0x7f827c0a15d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:41.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f827c0a19e0 addr v1:172.21.15.149:6789/0 2024-09-16T05:14:41.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f827c09d4e0 con 0x7f827c09b6c0 2024-09-16T05:14:41.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f827c09f430 con 0x7f827c0a19e0 2024-09-16T05:14:41.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:41.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f827c09e390 con 0x7f827c0a15d0 2024-09-16T05:14:41.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8279d9b640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827c0a15d0 msgr2=0x7f827c13a650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:14:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8280f0b640 10 monclient(hunting): authenticate will time out at 1366.166679s 2024-09-16T05:14:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f8279d9b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827c0a15d0 0x7f827c13a650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f827a59c640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f827c09b6c0 0x7f827c0a0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40628/0 (socket says 172.21.15.114:40628) 2024-09-16T05:14:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.536+0000 7f827a59c640 1 -- 172.21.15.114:0/681018715 learned_addr learned my addr 172.21.15.114:0/681018715 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:14:41.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.740+0000 7f8279d9b640 1 -- 172.21.15.114:0/681018715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827c0a15d0 msgr2=0x7f827c13a650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:14:41.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:41.740+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827c0a15d0 0x7f827c13a650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:14:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:42.144+0000 7f8279d9b640 1 -- 172.21.15.114:0/681018715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827c0a15d0 msgr2=0x7f827c13a650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:14:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:42.144+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827c0a15d0 0x7f827c13a650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:14:42.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:42.948+0000 7f8279d9b640 1 -- 172.21.15.114:0/681018715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827c0a15d0 msgr2=0x7f827c13a650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:14:42.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:42.948+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827c0a15d0 0x7f827c13a650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:14:44.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 10 monclient(hunting): tick 2024-09-16T05:14:44.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:14:44.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 1 monclient(hunting): continuing hunt 2024-09-16T05:14:44.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:14:44.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 1 -- 172.21.15.114:0/681018715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827c0a15d0 msgr2=0x7f827c13a650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:14:44.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 1 --2- 172.21.15.114:0/681018715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827c0a15d0 0x7f827c13a650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:44.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 1 -- 172.21.15.114:0/681018715 >> v1:172.21.15.149:6789/0 conn(0x7f827c0a19e0 legacy=0x7f827c13cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:44.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 1 -- 172.21.15.114:0/681018715 >> v1:172.21.15.114:6789/0 conn(0x7f827c09b6c0 legacy=0x7f827c0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:44.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 10 monclient: _add_conns ranks=[3,4] 2024-09-16T05:14:44.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 0x7f825c005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:44.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f825c003720 addr v2:172.21.15.114:3300/0 2024-09-16T05:14:44.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 0x7f825c008730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:44.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f825c006290 addr v2:172.21.15.149:3300/0 2024-09-16T05:14:44.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:44.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 1 -- 172.21.15.114:0/681018715 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f827c09e390 con 0x7f825c003720 2024-09-16T05:14:44.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:44.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 1 -- 172.21.15.114:0/681018715 --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f825c008dd0 con 0x7f825c006290 2024-09-16T05:14:44.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f826bfff640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:44.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f827a59c640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 0x7f825c005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:44.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f827a59c640 10 monclient(hunting): get_auth_request con 0x7f825c003720 auth_method 0 2024-09-16T05:14:44.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f827a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:44.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f827a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:44.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f827a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:44.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 0x7f825c008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:44.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f8279d9b640 10 monclient(hunting): get_auth_request con 0x7f825c006290 auth_method 0 2024-09-16T05:14:44.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f8279d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:44.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f8279d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:44.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f827a59c640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 msgr2=0x7f825c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:14:44.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f8279d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:44.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f827a59c640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 msgr2=0x7f825c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:44.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f827a59c640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 0x7f825c005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:44.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.536+0000 7f827a59c640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 0x7f825c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:44.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.540+0000 7f8279d9b640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 msgr2=0x7f825c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:14:44.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.540+0000 7f8279d9b640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 msgr2=0x7f825c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:44.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.540+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 0x7f825c008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:44.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.540+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 0x7f825c008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:44.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.740+0000 7f827a59c640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 0x7f825c005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:44.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.740+0000 7f827a59c640 10 monclient(hunting): get_auth_request con 0x7f825c003720 auth_method 0 2024-09-16T05:14:44.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.740+0000 7f827a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:44.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.740+0000 7f827a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:44.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.740+0000 7f827a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:44.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.740+0000 7f827a59c640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 msgr2=0x7f825c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:14:44.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.740+0000 7f827a59c640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 msgr2=0x7f825c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:44.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.740+0000 7f827a59c640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 0x7f825c005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:44.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.740+0000 7f827a59c640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 0x7f825c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:14:44.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.744+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 0x7f825c008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:44.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.744+0000 7f8279d9b640 10 monclient(hunting): get_auth_request con 0x7f825c006290 auth_method 0 2024-09-16T05:14:44.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.744+0000 7f8279d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:44.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.744+0000 7f8279d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:44.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.744+0000 7f8279d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:44.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.744+0000 7f8279d9b640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 msgr2=0x7f825c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:14:44.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.744+0000 7f8279d9b640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 msgr2=0x7f825c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:44.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.744+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 0x7f825c008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:44.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:44.744+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 0x7f825c008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:14:45.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.144+0000 7f827a59c640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 0x7f825c005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:45.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.144+0000 7f827a59c640 10 monclient(hunting): get_auth_request con 0x7f825c003720 auth_method 0 2024-09-16T05:14:45.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.144+0000 7f827a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:45.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.144+0000 7f827a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:45.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.144+0000 7f827a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:45.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.144+0000 7f827a59c640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 msgr2=0x7f825c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:14:45.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.144+0000 7f827a59c640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 msgr2=0x7f825c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:45.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.144+0000 7f827a59c640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 0x7f825c005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:45.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.144+0000 7f827a59c640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 0x7f825c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:14:45.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.148+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 0x7f825c008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:45.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.148+0000 7f8279d9b640 10 monclient(hunting): get_auth_request con 0x7f825c006290 auth_method 0 2024-09-16T05:14:45.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.148+0000 7f8279d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:45.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.148+0000 7f8279d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:45.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.148+0000 7f8279d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:45.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.148+0000 7f8279d9b640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 msgr2=0x7f825c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:14:45.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.148+0000 7f8279d9b640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 msgr2=0x7f825c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:45.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.148+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 0x7f825c008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:45.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.148+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 0x7f825c008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:14:45.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.948+0000 7f827a59c640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 0x7f825c005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:45.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.948+0000 7f827a59c640 10 monclient(hunting): get_auth_request con 0x7f825c003720 auth_method 0 2024-09-16T05:14:45.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.948+0000 7f827a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:45.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.948+0000 7f827a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:45.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.948+0000 7f827a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:45.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.948+0000 7f827a59c640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 msgr2=0x7f825c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:14:45.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.948+0000 7f827a59c640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 msgr2=0x7f825c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:45.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.948+0000 7f827a59c640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 0x7f825c005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:45.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.948+0000 7f827a59c640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 0x7f825c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:14:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.952+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 0x7f825c008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.952+0000 7f8279d9b640 10 monclient(hunting): get_auth_request con 0x7f825c006290 auth_method 0 2024-09-16T05:14:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.952+0000 7f8279d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:45.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.952+0000 7f8279d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:45.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.952+0000 7f8279d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:14:45.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.952+0000 7f8279d9b640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 msgr2=0x7f825c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:14:45.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.952+0000 7f8279d9b640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 msgr2=0x7f825c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:14:45.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.952+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 0x7f825c008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:14:45.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:45.952+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 0x7f825c008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:14:47.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 10 monclient(hunting): tick 2024-09-16T05:14:47.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:14:47.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 1 monclient(hunting): continuing hunt 2024-09-16T05:14:47.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:14:47.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 msgr2=0x7f825c008730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:14:47.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006290 0x7f825c008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 msgr2=0x7f825c005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:14:47.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003720 0x7f825c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T05:14:47.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f827ad9d640 1 -- 172.21.15.114:0/681018715 reap_dead start 2024-09-16T05:14:47.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f827c09b6c0 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:47.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003b90 0x7f825c006720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:47.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f825c003b90 addr v2:172.21.15.114:3300/0 2024-09-16T05:14:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006c60 0x7f825c007110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f825c006c60 addr v2:172.21.15.149:3300/0 2024-09-16T05:14:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 1 -- 172.21.15.114:0/681018715 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f827c09c2f0 con 0x7f827c09b6c0 2024-09-16T05:14:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 1 -- 172.21.15.114:0/681018715 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f827c09e390 con 0x7f825c003b90 2024-09-16T05:14:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 1 -- 172.21.15.114:0/681018715 --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f825c008dd0 con 0x7f825c006c60 2024-09-16T05:14:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f826bfff640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.114:3300/0 conn(0x7f825c003b90 0x7f825c006720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f8279d9b640 10 monclient(hunting): get_auth_request con 0x7f825c003b90 auth_method 0 2024-09-16T05:14:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f8279d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f8279d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f8279d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.536+0000 7f827ad9d640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006c60 0x7f825c007110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f827ad9d640 10 monclient(hunting): get_auth_request con 0x7f825c006c60 auth_method 0 2024-09-16T05:14:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f827ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f827ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f827ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f8279d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f8279d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f8279d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f827ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f827ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f827ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f8279d9b640 10 monclient(hunting): handle_auth_done global_id 74199 payload 274 2024-09-16T05:14:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f8279d9b640 1 -- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006c60 msgr2=0x7f825c007110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006c60 0x7f825c007110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f8279d9b640 1 -- 172.21.15.114:0/681018715 >> v1:172.21.15.114:6789/0 conn(0x7f827c09b6c0 legacy=0x7f825c00b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f8279d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:14:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f8279d9b640 1 monclient: found mon.noname-a 2024-09-16T05:14:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f8279d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f8279d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.540+0000 7f8279d9b640 1 -- 172.21.15.114:0/681018715 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c0087b0 con 0x7f825c003b90 2024-09-16T05:14:47.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f8279d9b640 1 --2- 172.21.15.114:0/681018715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f825c003b90 0x7f825c006720 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f825c006020 tx=0x7f826c01d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9810ef321f4e21fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:47.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f827959a640 1 -- 172.21.15.114:0/681018715 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c0b3070 con 0x7f825c003b90 2024-09-16T05:14:47.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f827959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:47.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f827959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:47.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f827959a640 10 monclient: dump: 2024-09-16T05:14:47.545 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:47.545 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:47.545 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.545 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.545 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:47.545 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:47.545 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:47.546 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:47.546 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:47.546 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:47.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f827959a640 10 monclient: _finish_auth 0 2024-09-16T05:14:47.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f827959a640 10 monclient: _check_auth_tickets 2024-09-16T05:14:47.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f827959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:47.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f827959a640 1 -- 172.21.15.114:0/681018715 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f826c0ae410 con 0x7f825c003b90 2024-09-16T05:14:47.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f827959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:47.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f8280f0b640 5 monclient: authenticate success, global_id 74199 2024-09-16T05:14:47.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f827959a640 1 -- 172.21.15.114:0/681018715 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c0ae5e0 con 0x7f825c003b90 2024-09-16T05:14:47.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f827959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:47.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f827959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:47.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f827959a640 10 monclient: dump: 2024-09-16T05:14:47.546 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:47.547 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:47.547 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.547 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.547 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:47.547 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:47.547 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:47.547 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:47.547 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:47.547 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:47.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f8280f0b640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:47.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f8280f0b640 4 set_mon_vals no callback set 2024-09-16T05:14:47.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f8280f0b640 10 monclient: shutdown 2024-09-16T05:14:47.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f8280f0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:47.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f8280f0b640 1 -- 172.21.15.114:0/681018715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f825c003b90 msgr2=0x7f825c006720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:47.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.544+0000 7f8280f0b640 1 --2- 172.21.15.114:0/681018715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f825c003b90 0x7f825c006720 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f825c006020 tx=0x7f826c01d490 comp rx=0 tx=0).stop 2024-09-16T05:14:47.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 -- 172.21.15.114:0/681018715 shutdown_connections 2024-09-16T05:14:47.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 --2- 172.21.15.114:0/681018715 >> v2:172.21.15.149:3300/0 conn(0x7f825c006c60 0x7f825c007110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 --2- 172.21.15.114:0/681018715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f825c003b90 0x7f825c006720 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 -- 172.21.15.114:0/681018715 >> 172.21.15.114:0/681018715 conn(0x7f827c0a1fb0 msgr2=0x7f827c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:47.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 -- 172.21.15.114:0/681018715 shutdown_connections 2024-09-16T05:14:47.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 -- 172.21.15.114:0/681018715 wait complete. 2024-09-16T05:14:47.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 10 monclient: build_initial_monmap 2024-09-16T05:14:47.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 10 monclient: monmap: 2024-09-16T05:14:47.553 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:47.553 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:47.553 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:47.553 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:47.553 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:47.553 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:47.553 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:47.553 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:47.553 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:47.553 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:47.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 librados: starting msgr at 2024-09-16T05:14:47.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 librados: starting objecter 2024-09-16T05:14:47.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 Processor -- start 2024-09-16T05:14:47.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 -- start start 2024-09-16T05:14:47.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 librados: setting wanted keys 2024-09-16T05:14:47.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 librados: calling monclient init 2024-09-16T05:14:47.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 10 monclient: init 2024-09-16T05:14:47.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:47.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:14:47.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f825c003b90 0x7f827c1cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:47.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 10 monclient(hunting): picked mon.noname-a con 0x7f825c003b90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:47.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f825c006c60 0x7f827c0967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:47.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 10 monclient(hunting): picked mon.noname-c con 0x7f825c006c60 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:47.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f827c09b6c0 0x7f827c1c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:47.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 10 monclient(hunting): picked mon.noname-b con 0x7f827c09b6c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:47.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:47.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f827c0a12d0 con 0x7f825c003b90 2024-09-16T05:14:47.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:47.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f827c09baa0 con 0x7f827c09b6c0 2024-09-16T05:14:47.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:47.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f827c0a1450 con 0x7f825c006c60 2024-09-16T05:14:47.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:47.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 10 monclient(hunting): authenticate will time out at 1372.178557s 2024-09-16T05:14:47.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8279d9b640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f825c006c60 msgr2=0x7f827c0967a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:14:47.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8279d9b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f825c006c60 0x7f827c0967a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:47.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f825c003b90 0x7f827c1cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:47.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f825c003b90 0x7f827c1cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60946/0 (socket says 172.21.15.114:60946) 2024-09-16T05:14:47.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 1 -- 172.21.15.114:0/1897877708 learned_addr learned my addr 172.21.15.114:0/1897877708 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:47.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 10 monclient(hunting): get_auth_request con 0x7f825c003b90 auth_method 0 2024-09-16T05:14:47.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:47.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:47.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:47.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827ad9d640 1 --2- 172.21.15.114:0/1897877708 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f827c09b6c0 0x7f827c1c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:47.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:47.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:47.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:47.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827ad9d640 10 monclient(hunting): get_auth_request con 0x7f827c09b6c0 auth_method 0 2024-09-16T05:14:47.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:47.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:47.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:47.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:47.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:47.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:47.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 10 monclient(hunting): handle_auth_done global_id 74205 payload 824 2024-09-16T05:14:47.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 1 -- 172.21.15.114:0/1897877708 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f825c006c60 msgr2=0x7f827c0967a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:14:47.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 1 --2- 172.21.15.114:0/1897877708 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f825c006c60 0x7f827c0967a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 1 -- 172.21.15.114:0/1897877708 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f827c09b6c0 msgr2=0x7f827c1c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:47.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 1 --2- 172.21.15.114:0/1897877708 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f827c09b6c0 0x7f827c1c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 10 monclient: _finish_hunting 0 2024-09-16T05:14:47.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 1 monclient: found mon.noname-a 2024-09-16T05:14:47.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:47.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:47.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 1 -- 172.21.15.114:0/1897877708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827c09c6a0 con 0x7f825c003b90 2024-09-16T05:14:47.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f827a59c640 1 --2- 172.21.15.114:0/1897877708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f825c003b90 0x7f827c1cb220 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8270007b20 tx=0x7f8270001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f826affd640 10 client.?.objecter ms_handle_connect 0x7f825c003b90 2024-09-16T05:14:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f826affd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f826affd640 1 -- 172.21.15.114:0/1897877708 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827001b020 con 0x7f825c003b90 2024-09-16T05:14:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f826affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f826affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f826affd640 10 monclient: dump: 2024-09-16T05:14:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:47.559 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:47.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f826affd640 10 monclient: _finish_auth 0 2024-09-16T05:14:47.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f826affd640 10 monclient: _check_auth_tickets 2024-09-16T05:14:47.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f826affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:47.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f826affd640 1 -- 172.21.15.114:0/1897877708 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8270001d20 con 0x7f825c003b90 2024-09-16T05:14:47.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 5 monclient: authenticate success, global_id 74205 2024-09-16T05:14:47.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 10 monclient: _renew_subs 2024-09-16T05:14:47.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:47.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8280f0b640 1 -- 172.21.15.114:0/1897877708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f827c09b290 con 0x7f825c003b90 2024-09-16T05:14:47.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f826affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:47.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f826affd640 1 -- 172.21.15.114:0/1897877708 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8270017870 con 0x7f825c003b90 2024-09-16T05:14:47.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f826affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:47.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f826affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:47.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f826affd640 10 monclient: dump: 2024-09-16T05:14:47.559 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:47.560 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:47.560 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.560 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.560 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:47.560 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:47.560 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:47.560 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:47.560 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:47.560 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:47.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.548+0000 7f8278d99640 4 set_mon_vals no callback set 2024-09-16T05:14:47.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f8280f0b640 10 client.74205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:47.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f8280f0b640 10 monclient: _renew_subs 2024-09-16T05:14:47.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f8280f0b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:47.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f8280f0b640 1 -- 172.21.15.114:0/1897877708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f827c091f80 con 0x7f825c003b90 2024-09-16T05:14:47.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f8280f0b640 1 librados: init done 2024-09-16T05:14:47.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f826affd640 1 -- 172.21.15.114:0/1897877708 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f827001e020 con 0x7f825c003b90 2024-09-16T05:14:47.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f8280f0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:47.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f8280f0b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:47.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f8280f0b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:47.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f8280f0b640 1 -- 172.21.15.114:0/1897877708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f827c091c00 con 0x7f825c003b90 2024-09-16T05:14:47.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f826affd640 1 --2- 172.21.15.114:0/1897877708 >> v2:172.21.15.149:6812/1185531135 conn(0x7f825c039070 0x7f825c03b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:47.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f826affd640 1 -- 172.21.15.114:0/1897877708 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(293..293 src has 1..293) ==== 6282+0+0 (secure 0 0 0) 0x7f8270049a80 con 0x7f825c003b90 2024-09-16T05:14:47.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f826affd640 10 client.74205.objecter ms_dispatch 0x7f827c000ba0 osd_map(293..293 src has 1..293) 2024-09-16T05:14:47.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f826affd640 3 client.74205.objecter handle_osd_map got epochs [293,293] > 0 2024-09-16T05:14:47.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f826affd640 3 client.74205.objecter handle_osd_map decoding full epoch 293 2024-09-16T05:14:47.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f826affd640 20 client.74205.objecter dump_active .. 0 homeless 2024-09-16T05:14:47.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f8279d9b640 1 --2- 172.21.15.114:0/1897877708 >> v2:172.21.15.149:6812/1185531135 conn(0x7f825c039070 0x7f825c03b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:47.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f8279d9b640 10 monclient: get_auth_request con 0x7f825c039070 auth_method 0 2024-09-16T05:14:47.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f826affd640 10 client.74205.objecter ms_handle_connect 0x7f825c039070 2024-09-16T05:14:47.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.552+0000 7f8279d9b640 1 --2- 172.21.15.114:0/1897877708 >> v2:172.21.15.149:6812/1185531135 conn(0x7f825c039070 0x7f825c03b530 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7f826c0b0000 tx=0x7f826c0a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:47.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.556+0000 7f826affd640 1 -- 172.21.15.114:0/1897877708 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f8270025020 con 0x7f825c003b90 2024-09-16T05:14:47.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.556+0000 7f826affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:47.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.556+0000 7f826affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:47.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.780+0000 7f8280f0b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:14:47.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.780+0000 7f8280f0b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:47.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.780+0000 7f8280f0b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.780+0000 7f8280f0b640 1 -- 172.21.15.114:0/1897877708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f827c099620 con 0x7f825c003b90 2024-09-16T05:14:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.780+0000 7f826affd640 1 -- 172.21.15.114:0/1897877708 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f827c09c6a0 con 0x7f825c003b90 2024-09-16T05:14:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.780+0000 7f826affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:14:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.780+0000 7f826affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:47.781 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":38,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:14:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 10 librados: watch_flush enter 2024-09-16T05:14:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 10 librados: watch_flush exit 2024-09-16T05:14:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 20 client.74205.objecter shutdown clearing up homeless session... 2024-09-16T05:14:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 10 client.74205.objecter successfully canceled tick 2024-09-16T05:14:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 1 -- 172.21.15.114:0/1897877708 >> v2:172.21.15.149:6812/1185531135 conn(0x7f825c039070 msgr2=0x7f825c03b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 1 --2- 172.21.15.114:0/1897877708 >> v2:172.21.15.149:6812/1185531135 conn(0x7f825c039070 0x7f825c03b530 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7f826c0b0000 tx=0x7f826c0a7000 comp rx=0 tx=0).stop 2024-09-16T05:14:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 10 monclient: shutdown 2024-09-16T05:14:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 1 -- 172.21.15.114:0/1897877708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f825c003b90 msgr2=0x7f827c1cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 1 --2- 172.21.15.114:0/1897877708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f825c003b90 0x7f827c1cb220 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8270007b20 tx=0x7f8270001640 comp rx=0 tx=0).stop 2024-09-16T05:14:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 1 -- 172.21.15.114:0/1897877708 shutdown_connections 2024-09-16T05:14:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 1 --2- 172.21.15.114:0/1897877708 >> v2:172.21.15.149:6812/1185531135 conn(0x7f825c039070 0x7f825c03b530 unknown :-1 s=CLOSED pgs=3508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 1 --2- 172.21.15.114:0/1897877708 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f827c09b6c0 0x7f827c1c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 1 --2- 172.21.15.114:0/1897877708 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f825c006c60 0x7f827c0967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 1 --2- 172.21.15.114:0/1897877708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f825c003b90 0x7f827c1cb220 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 1 -- 172.21.15.114:0/1897877708 >> 172.21.15.114:0/1897877708 conn(0x7f827c0a1fb0 msgr2=0x7f827c098550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:47.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 1 -- 172.21.15.114:0/1897877708 shutdown_connections 2024-09-16T05:14:47.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 1 -- 172.21.15.114:0/1897877708 wait complete. 2024-09-16T05:14:47.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.784+0000 7f8280f0b640 1 librados: shutdown 2024-09-16T05:14:47.799 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:14:47.799 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T05:14:47.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:14:47.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 10 monclient: get_monmap_and_config 2024-09-16T05:14:47.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 10 monclient: build_initial_monmap 2024-09-16T05:14:47.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 10 monclient: monmap: 2024-09-16T05:14:47.922 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:47.922 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:47.922 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:47.924346+0000 2024-09-16T05:14:47.922 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:47.924346+0000 2024-09-16T05:14:47.922 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:47.922 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:47.922 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:47.923 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:47.923 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:47.923 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:47.923 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:47.923 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 1 Processor -- start 2024-09-16T05:14:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 1 -- start start 2024-09-16T05:14:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 10 monclient: init 2024-09-16T05:14:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-16T05:14:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44580a0ec0 addr v1:172.21.15.149:6789/0 2024-09-16T05:14:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f445809e1d0 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f445809e5e0 0x7f445813cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 10 monclient(hunting): picked mon.noname-a con 0x7f445809e5e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:14:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f445809a760 con 0x7f445809e1d0 2024-09-16T05:14:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44580a1870 con 0x7f44580a0ec0 2024-09-16T05:14:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f44580a12a0 con 0x7f445809e5e0 2024-09-16T05:14:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f445e0c0640 10 monclient(hunting): authenticate will time out at 1372.553507s 2024-09-16T05:14:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f445809e5e0 0x7f445813cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f445809e5e0 0x7f445813cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60952/0 (socket says 172.21.15.114:60952) 2024-09-16T05:14:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 1 -- 172.21.15.114:0/3301719164 learned_addr learned my addr 172.21.15.114:0/3301719164 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 10 monclient(hunting): get_auth_request con 0x7f445809e5e0 auth_method 0 2024-09-16T05:14:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:47.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:47.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4456ffd640 1 --1- 172.21.15.114:0/3301719164 >> v1:172.21.15.114:6789/0 conn(0x7f445809e1d0 0x7f4458139640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33776/0 (socket says 172.21.15.114:33776) 2024-09-16T05:14:47.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:47.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:47.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:47.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:47.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 10 monclient(hunting): handle_auth_done global_id 74208 payload 274 2024-09-16T05:14:47.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 1 -- 172.21.15.114:0/3301719164 >> v1:172.21.15.149:6789/0 conn(0x7f44580a0ec0 legacy=0x7f445809dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:14:47.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 1 -- 172.21.15.114:0/3301719164 >> v1:172.21.15.114:6789/0 conn(0x7f445809e1d0 legacy=0x7f4458139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:47.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 10 monclient: _finish_hunting 0 2024-09-16T05:14:47.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 1 monclient: found mon.noname-a 2024-09-16T05:14:47.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:47.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:47.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.924+0000 7f4457fff640 1 -- 172.21.15.114:0/3301719164 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4458099790 con 0x7f445809e5e0 2024-09-16T05:14:47.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4457fff640 1 --2- 172.21.15.114:0/3301719164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445809e5e0 0x7f445813cd70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4458074140 tx=0x7f445001eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1fa0ecfa7e1a9f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:47.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f44567fc640 1 -- 172.21.15.114:0/3301719164 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445000b2d0 con 0x7f445809e5e0 2024-09-16T05:14:47.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f44567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:47.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f44567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:47.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f44567fc640 10 monclient: dump: 2024-09-16T05:14:47.936 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:47.936 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:47.936 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.936 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.936 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:47.936 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:47.936 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:47.936 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:47.936 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:47.936 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:47.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f44567fc640 10 monclient: _finish_auth 0 2024-09-16T05:14:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f44567fc640 10 monclient: _check_auth_tickets 2024-09-16T05:14:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f44567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f44567fc640 1 -- 172.21.15.114:0/3301719164 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f445000b470 con 0x7f445809e5e0 2024-09-16T05:14:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 5 monclient: authenticate success, global_id 74208 2024-09-16T05:14:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f44567fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f44567fc640 1 -- 172.21.15.114:0/3301719164 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44500a7a00 con 0x7f445809e5e0 2024-09-16T05:14:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 4 set_mon_vals no callback set 2024-09-16T05:14:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f44567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f44567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f44567fc640 10 monclient: dump: 2024-09-16T05:14:47.938 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:47.938 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:47.938 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.938 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.938 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:47.938 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:47.938 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:47.938 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:47.938 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:47.938 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:47.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient: shutdown 2024-09-16T05:14:47.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:47.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 -- 172.21.15.114:0/3301719164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445809e5e0 msgr2=0x7f445813cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 --2- 172.21.15.114:0/3301719164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445809e5e0 0x7f445813cd70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4458074140 tx=0x7f445001eb00 comp rx=0 tx=0).stop 2024-09-16T05:14:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 -- 172.21.15.114:0/3301719164 shutdown_connections 2024-09-16T05:14:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 --2- 172.21.15.114:0/3301719164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445809e5e0 0x7f445813cd70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 -- 172.21.15.114:0/3301719164 >> 172.21.15.114:0/3301719164 conn(0x7f44580a1fb0 msgr2=0x7f44580a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 -- 172.21.15.114:0/3301719164 shutdown_connections 2024-09-16T05:14:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 -- 172.21.15.114:0/3301719164 wait complete. 2024-09-16T05:14:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient: build_initial_monmap 2024-09-16T05:14:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient: monmap: 2024-09-16T05:14:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:47.940 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 librados: starting msgr at 2024-09-16T05:14:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 librados: starting objecter 2024-09-16T05:14:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 Processor -- start 2024-09-16T05:14:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 -- start start 2024-09-16T05:14:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 librados: setting wanted keys 2024-09-16T05:14:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 librados: calling monclient init 2024-09-16T05:14:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient: init 2024-09-16T05:14:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:14:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f445809e1d0 0x7f4458133b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient(hunting): picked mon.noname-c con 0x7f445809e1d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445809e5e0 0x7f4458134090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient(hunting): picked mon.noname-a con 0x7f445809e5e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f44580a0ec0 0x7f44581cf550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient(hunting): picked mon.noname-b con 0x7f44580a0ec0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f445809d340 con 0x7f445809e5e0 2024-09-16T05:14:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f445809d1c0 con 0x7f44580a0ec0 2024-09-16T05:14:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f445809d4c0 con 0x7f445809e1d0 2024-09-16T05:14:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient(hunting): authenticate will time out at 1372.558644s 2024-09-16T05:14:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f44577fe640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f445809e1d0 msgr2=0x7f4458133b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:14:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f44577fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f445809e1d0 0x7f4458133b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:47.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445809e5e0 0x7f4458134090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:47.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445809e5e0 0x7f4458134090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60964/0 (socket says 172.21.15.114:60964) 2024-09-16T05:14:47.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 1 -- 172.21.15.114:0/430624402 learned_addr learned my addr 172.21.15.114:0/430624402 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:47.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 10 monclient(hunting): get_auth_request con 0x7f445809e5e0 auth_method 0 2024-09-16T05:14:47.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:47.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:47.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:47.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:47.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:47.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:47.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 10 monclient(hunting): handle_auth_done global_id 74214 payload 824 2024-09-16T05:14:47.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 1 -- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f445809e1d0 msgr2=0x7f4458133b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:14:47.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 1 --2- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f445809e1d0 0x7f4458133b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 1 -- 172.21.15.114:0/430624402 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f44580a0ec0 msgr2=0x7f44581cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 1 --2- 172.21.15.114:0/430624402 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f44580a0ec0 0x7f44581cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:14:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 1 monclient: found mon.noname-a 2024-09-16T05:14:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 1 -- 172.21.15.114:0/430624402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445812d920 con 0x7f445809e5e0 2024-09-16T05:14:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4456ffd640 1 --2- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445809e5e0 0x7f4458134090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f444800c390 tx=0x7f444800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f444ffff640 10 client.?.objecter ms_handle_connect 0x7f445809e5e0 2024-09-16T05:14:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f444ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f444ffff640 1 -- 172.21.15.114:0/430624402 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44480074e0 con 0x7f445809e5e0 2024-09-16T05:14:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f444ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f444ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:47.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f444ffff640 10 monclient: dump: 2024-09-16T05:14:47.944 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:47.944 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:47.944 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.944 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.944 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:47.944 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:47.944 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:47.944 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:47.944 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:47.944 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:47.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f444ffff640 10 monclient: _finish_auth 0 2024-09-16T05:14:47.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f444ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:14:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f444ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f444ffff640 1 -- 172.21.15.114:0/430624402 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4448007680 con 0x7f445809e5e0 2024-09-16T05:14:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 5 monclient: authenticate success, global_id 74214 2024-09-16T05:14:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient: _renew_subs 2024-09-16T05:14:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 -- 172.21.15.114:0/430624402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f445812cd30 con 0x7f445809e5e0 2024-09-16T05:14:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f444ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f444ffff640 1 -- 172.21.15.114:0/430624402 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4448004a30 con 0x7f445809e5e0 2024-09-16T05:14:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f444ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f444ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f444ffff640 10 monclient: dump: 2024-09-16T05:14:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:47.946 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f4455ffb640 4 set_mon_vals no callback set 2024-09-16T05:14:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 client.74214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient: _renew_subs 2024-09-16T05:14:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 -- 172.21.15.114:0/430624402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44581345d0 con 0x7f445809e5e0 2024-09-16T05:14:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.928+0000 7f445e0c0640 1 librados: init done 2024-09-16T05:14:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f445e0c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f445e0c0640 1 --2- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445812aa50 0x7f44581cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f445e0c0640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f445e0c0640 1 -- 172.21.15.114:0/430624402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f445812cf50 con 0x7f445812aa50 2024-09-16T05:14:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f44577fe640 1 --2- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445812aa50 0x7f44581cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f44577fe640 10 monclient: get_auth_request con 0x7f445812aa50 auth_method 0 2024-09-16T05:14:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f44577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f44577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f44577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 1 -- 172.21.15.114:0/430624402 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f444801a020 con 0x7f445809e5e0 2024-09-16T05:14:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f44577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f44577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f44577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f44577fe640 10 monclient(hunting): handle_auth_done global_id 74217 payload 824 2024-09-16T05:14:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f44577fe640 1 -- 172.21.15.114:0/430624402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f445809dac0 con 0x7f445812aa50 2024-09-16T05:14:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 1 --2- 172.21.15.114:0/430624402 >> v2:172.21.15.149:6812/1185531135 conn(0x7f44380377c0 0x7f4438039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 1 -- 172.21.15.114:0/430624402 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(293..293 src has 1..293) ==== 6282+0+0 (secure 0 0 0) 0x7f4448048940 con 0x7f445809e5e0 2024-09-16T05:14:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 10 client.74214.objecter ms_dispatch 0x7f4458000ba0 osd_map(293..293 src has 1..293) 2024-09-16T05:14:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 3 client.74214.objecter handle_osd_map got epochs [293,293] > 0 2024-09-16T05:14:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 3 client.74214.objecter handle_osd_map decoding full epoch 293 2024-09-16T05:14:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f44577fe640 1 --2- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445812aa50 0x7f44581cfa90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4440007fd0 tx=0x7f444000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 20 client.74214.objecter dump_active .. 0 homeless 2024-09-16T05:14:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 10 client.74214.objecter ms_handle_connect 0x7f445812aa50 2024-09-16T05:14:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 10 client.74214.objecter resend_mon_ops 2024-09-16T05:14:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 1 -- 172.21.15.114:0/430624402 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4440017070 con 0x7f445812aa50 2024-09-16T05:14:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 10 monclient: dump: 2024-09-16T05:14:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:47.949 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f4457fff640 1 --2- 172.21.15.114:0/430624402 >> v2:172.21.15.149:6812/1185531135 conn(0x7f44380377c0 0x7f4438039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:47.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f4457fff640 10 monclient: get_auth_request con 0x7f44380377c0 auth_method 0 2024-09-16T05:14:47.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 1 -- 172.21.15.114:0/430624402 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f44400074e0 con 0x7f445812aa50 2024-09-16T05:14:47.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:47.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:47.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 1 -- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445812aa50 msgr2=0x7f44581cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:47.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 1 --2- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445812aa50 0x7f44581cfa90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4440007fd0 tx=0x7f444000c810 comp rx=0 tx=0).stop 2024-09-16T05:14:47.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 1 -- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445812aa50 msgr2=0x7f44581cfa90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:47.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 1 --2- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445812aa50 0x7f44581cfa90 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4440007fd0 tx=0x7f444000c810 comp rx=0 tx=0).stop 2024-09-16T05:14:47.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f444ffff640 10 client.74214.objecter ms_handle_connect 0x7f44380377c0 2024-09-16T05:14:47.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.932+0000 7f4457fff640 1 --2- 172.21.15.114:0/430624402 >> v2:172.21.15.149:6812/1185531135 conn(0x7f44380377c0 0x7f4438039c80 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7f44500b0000 tx=0x7f4450004ac0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f445e0c0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:14:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f445e0c0640 1 --2- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44581cffd0 0x7f4458138b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f445e0c0640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f445e0c0640 1 -- 172.21.15.114:0/430624402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f44581d03e0 con 0x7f44581cffd0 2024-09-16T05:14:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f44577fe640 1 --2- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44581cffd0 0x7f4458138b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f44577fe640 10 monclient: get_auth_request con 0x7f44581cffd0 auth_method 0 2024-09-16T05:14:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f44577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f44577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f44577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f44577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f44577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f44577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f44577fe640 10 monclient(hunting): handle_auth_done global_id 74220 payload 824 2024-09-16T05:14:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f44577fe640 1 -- 172.21.15.114:0/430624402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f44580686f0 con 0x7f44581cffd0 2024-09-16T05:14:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f44577fe640 1 --2- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44581cffd0 0x7f4458138b10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4440007fd0 tx=0x7f44400037f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f444ffff640 10 client.74214.objecter ms_handle_connect 0x7f44581cffd0 2024-09-16T05:14:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.936+0000 7f444ffff640 10 client.74214.objecter resend_mon_ops 2024-09-16T05:14:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f444ffff640 1 -- 172.21.15.114:0/430624402 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4440017040 con 0x7f44581cffd0 2024-09-16T05:14:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f444ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f444ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f444ffff640 10 monclient: dump: 2024-09-16T05:14:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:47.954 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f444ffff640 1 -- 172.21.15.114:0/430624402 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7f4440022400 con 0x7f44581cffd0 2024-09-16T05:14:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f444ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:14:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f444ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f444ffff640 1 -- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44581cffd0 msgr2=0x7f4458138b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f444ffff640 1 --2- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44581cffd0 0x7f4458138b10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4440007fd0 tx=0x7f44400037f0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f444ffff640 1 -- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44581cffd0 msgr2=0x7f4458138b10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:47.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f444ffff640 1 --2- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44581cffd0 0x7f4458138b10 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4440007fd0 tx=0x7f44400037f0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.955 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:14:47.955 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:14:47.955 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:14:47.955 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:14:47.955 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 38, 2024-09-16T05:14:47.955 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 46995, 2024-09-16T05:14:47.955 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:14:47.955 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:14:47.955 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-16T05:14:47.956 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:47.956 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 1, 2024-09-16T05:14:47.956 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:47.956 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:14:47.956 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:14:47.956 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:47.956 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:47.956 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:47.956 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:47.956 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:47.956 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:47.956 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:47.956 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:47.956 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:47.957 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:47.957 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:47.957 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:47.957 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:14:47.957 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:14:47.957 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:47.957 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:47.957 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:47.957 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:47.957 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:47.957 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:47.957 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:47.959 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:47.959 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:47.959 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:47.959 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:47.959 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:47.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:47.959 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:14:47.959 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:14:47.959 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:14:47.959 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:14:47.959 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:14:47.960 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:14:47.960 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:47.960 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:47.960 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:14:47.960 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:14:47.960 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:14:47.960 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:14:47.960 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:14:47.960 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:14:47.960 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:14:47.960 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:47.961 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:14:47.961 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:47.961 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:47.961 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:47.961 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:47.961 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:47.961 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:47.961 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:47.961 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:47.961 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:47.961 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:47.961 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:47.961 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:47.962 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:14:47.962 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:47.962 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:14:47.962 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:47.962 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:14:47.962 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:14:47.962 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:47.962 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:47.962 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:47.962 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:47.962 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:14:47.962 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:47.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:47.963 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:47.963 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:47.963 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:14:47.963 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:47.963 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:47.963 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:47.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:47.963 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:14:47.963 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:14:47.963 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:47.963 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:47.963 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:47.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:47.963 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:47.964 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:14:47.964 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:14:47.964 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:47.964 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:47.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:47.964 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:47.964 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:14:47.964 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:47.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:47.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:47.964 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:47.964 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:14:47.965 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:47.965 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:47.965 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:47.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:47.965 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:14:47.965 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:14:47.965 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:47.965 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:47.965 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:47.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:47.965 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:47.965 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:14:47.965 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:14:47.965 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:47.965 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:47.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:47.966 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:47.966 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:14:47.966 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:47.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:47.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:47.966 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:47.966 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:14:47.966 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:47.966 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:47.966 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:47.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:47.966 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:14:47.966 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:14:47.967 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:47.967 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:47.967 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:47.967 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:47.967 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:47.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:47.967 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:14:47.967 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:14:47.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:47.967 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:47.967 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:47.967 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:14:47.967 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:47.968 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:47.968 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:14:47.968 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:47.968 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:47.968 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:47.968 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:14:47.968 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:47.968 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:47.968 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:14:47.968 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:47.968 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:47.968 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:47.968 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:14:47.969 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:47.969 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:47.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:47.969 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:14:47.969 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:14:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 10 librados: watch_flush enter 2024-09-16T05:14:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 10 librados: watch_flush exit 2024-09-16T05:14:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 20 client.74214.objecter shutdown clearing up homeless session... 2024-09-16T05:14:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 10 client.74214.objecter successfully canceled tick 2024-09-16T05:14:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 1 -- 172.21.15.114:0/430624402 >> v2:172.21.15.149:6812/1185531135 conn(0x7f44380377c0 msgr2=0x7f4438039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 1 --2- 172.21.15.114:0/430624402 >> v2:172.21.15.149:6812/1185531135 conn(0x7f44380377c0 0x7f4438039c80 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7f44500b0000 tx=0x7f4450004ac0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 10 monclient: shutdown 2024-09-16T05:14:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 1 -- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445809e5e0 msgr2=0x7f4458134090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 1 --2- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445809e5e0 0x7f4458134090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f444800c390 tx=0x7f444800c850 comp rx=0 tx=0).stop 2024-09-16T05:14:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f4457fff640 1 -- 172.21.15.114:0/430624402 reap_dead start 2024-09-16T05:14:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 1 -- 172.21.15.114:0/430624402 shutdown_connections 2024-09-16T05:14:47.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 1 --2- 172.21.15.114:0/430624402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f445809e5e0 0x7f4458134090 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:47.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 1 -- 172.21.15.114:0/430624402 >> 172.21.15.114:0/430624402 conn(0x7f44580a1fb0 msgr2=0x7f4458136850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:47.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 1 -- 172.21.15.114:0/430624402 shutdown_connections 2024-09-16T05:14:47.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 1 -- 172.21.15.114:0/430624402 wait complete. 2024-09-16T05:14:47.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:47.940+0000 7f445e0c0640 1 librados: shutdown 2024-09-16T05:14:47.972 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:14:47.972 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T05:14:47.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:14:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.096+0000 7f8f08a79640 10 monclient: get_monmap_and_config 2024-09-16T05:14:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.096+0000 7f8f08a79640 10 monclient: build_initial_monmap 2024-09-16T05:14:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.096+0000 7f8f08a79640 10 monclient: monmap: 2024-09-16T05:14:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:14:48.099893+0000 2024-09-16T05:14:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:14:48.099893+0000 2024-09-16T05:14:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:14:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:14:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:14:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:14:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:14:48.098 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:48.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 1 Processor -- start 2024-09-16T05:14:48.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 1 -- start start 2024-09-16T05:14:48.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 10 monclient: init 2024-09-16T05:14:48.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:48.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-16T05:14:48.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8f04137730 0x7f8f04137b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:48.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 10 monclient(hunting): picked mon.noname-a con 0x7f8f04137730 addr v2:172.21.15.114:3300/0 2024-09-16T05:14:48.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f040a5d00 0x7f8f040a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:48.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 10 monclient(hunting): picked mon.noname-b con 0x7f8f040a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:48.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f040a47b0 addr v1:172.21.15.114:6789/0 2024-09-16T05:14:48.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f040a55e0 con 0x7f8f040a47b0 2024-09-16T05:14:48.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:48.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f8f0409c450 con 0x7f8f04137730 2024-09-16T05:14:48.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:48.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8f040a3710 con 0x7f8f040a5d00 2024-09-16T05:14:48.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:48.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f08a79640 10 monclient(hunting): authenticate will time out at 1372.729038s 2024-09-16T05:14:48.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8efbfff640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f040a5d00 msgr2=0x7f8f040a4270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:14:48.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8efbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f040a5d00 0x7f8f040a4270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:48.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8f04137730 0x7f8f04137b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:48.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8f04137730 0x7f8f04137b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60984/0 (socket says 172.21.15.114:60984) 2024-09-16T05:14:48.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 1 -- 172.21.15.114:0/789334740 learned_addr learned my addr 172.21.15.114:0/789334740 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:48.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 10 monclient(hunting): get_auth_request con 0x7f8f04137730 auth_method 0 2024-09-16T05:14:48.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:48.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:48.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:48.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:48.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:48.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:48.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 10 monclient(hunting): handle_auth_done global_id 74223 payload 274 2024-09-16T05:14:48.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 1 -- 172.21.15.114:0/789334740 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f040a5d00 msgr2=0x7f8f040a4270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:14:48.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 1 --2- 172.21.15.114:0/789334740 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f040a5d00 0x7f8f040a4270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:48.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 1 -- 172.21.15.114:0/789334740 >> v1:172.21.15.114:6789/0 conn(0x7f8f040a47b0 legacy=0x7f8f0413e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:48.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 10 monclient: _finish_hunting 0 2024-09-16T05:14:48.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 1 monclient: found mon.noname-a 2024-09-16T05:14:48.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:48.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:48.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.100+0000 7f8f0259c640 1 -- 172.21.15.114:0/789334740 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0409be00 con 0x7f8f04137730 2024-09-16T05:14:48.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 1 --2- 172.21.15.114:0/789334740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f04137730 0x7f8f04137b10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4009870 tx=0x7f8ef401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60bc01dc2b4e8e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:48.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f01d9b640 1 -- 172.21.15.114:0/789334740 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef40b0070 con 0x7f8f04137730 2024-09-16T05:14:48.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f01d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:48.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f01d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:48.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f01d9b640 10 monclient: dump: 2024-09-16T05:14:48.110 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:48.110 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:48.110 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:48.110 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:48.110 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:48.110 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:48.111 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:48.111 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:48.111 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:48.111 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:48.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f01d9b640 10 monclient: _finish_auth 0 2024-09-16T05:14:48.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f01d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:14:48.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f01d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:48.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f01d9b640 1 -- 172.21.15.114:0/789334740 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ef40ab600 con 0x7f8f04137730 2024-09-16T05:14:48.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 5 monclient: authenticate success, global_id 74223 2024-09-16T05:14:48.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:14:48.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f01d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:48.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f01d9b640 1 -- 172.21.15.114:0/789334740 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef40ab9d0 con 0x7f8f04137730 2024-09-16T05:14:48.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient: get_monmap_and_config success 2024-09-16T05:14:48.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 4 set_mon_vals no callback set 2024-09-16T05:14:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f01d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f01d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f01d9b640 10 monclient: dump: 2024-09-16T05:14:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:48.112 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient: shutdown 2024-09-16T05:14:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 -- 172.21.15.114:0/789334740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f04137730 msgr2=0x7f8f04137b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 --2- 172.21.15.114:0/789334740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f04137730 0x7f8f04137b10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4009870 tx=0x7f8ef401c4a0 comp rx=0 tx=0).stop 2024-09-16T05:14:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 -- 172.21.15.114:0/789334740 shutdown_connections 2024-09-16T05:14:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 --2- 172.21.15.114:0/789334740 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f040a5d00 0x7f8f040a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 --2- 172.21.15.114:0/789334740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f04137730 0x7f8f04137b10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 -- 172.21.15.114:0/789334740 >> 172.21.15.114:0/789334740 conn(0x7f8f040a9c70 msgr2=0x7f8f040aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 -- 172.21.15.114:0/789334740 shutdown_connections 2024-09-16T05:14:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 -- 172.21.15.114:0/789334740 wait complete. 2024-09-16T05:14:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient: build_initial_monmap 2024-09-16T05:14:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient: monmap: 2024-09-16T05:14:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:14:48.114 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:14:48.114 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:14:48.114 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:14:48.114 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:14:48.114 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:48.114 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:14:48.114 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:14:48.114 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:14:48.114 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:48.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 librados: starting msgr at 2024-09-16T05:14:48.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 librados: starting objecter 2024-09-16T05:14:48.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 Processor -- start 2024-09-16T05:14:48.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 -- start start 2024-09-16T05:14:48.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 librados: setting wanted keys 2024-09-16T05:14:48.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 librados: calling monclient init 2024-09-16T05:14:48.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient: init 2024-09-16T05:14:48.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient: _reopen_session rank -1 2024-09-16T05:14:48.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:14:48.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f040a47b0 0x7f8f041cf680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:48.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient(hunting): picked mon.noname-a con 0x7f8f040a47b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:14:48.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f040a5d00 0x7f8f0413e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:48.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient(hunting): picked mon.noname-c con 0x7f8f040a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:14:48.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04137730 0x7f8f0413ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:48.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient(hunting): picked mon.noname-b con 0x7f8f04137730 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:14:48.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:48.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8f04072a20 con 0x7f8f040a47b0 2024-09-16T05:14:48.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:48.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f8f040728a0 con 0x7f8f04137730 2024-09-16T05:14:48.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:48.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8f041371a0 con 0x7f8f040a5d00 2024-09-16T05:14:48.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient(hunting): _renew_subs 2024-09-16T05:14:48.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient(hunting): authenticate will time out at 1372.734719s 2024-09-16T05:14:48.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efbfff640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f040a5d00 msgr2=0x7f8f0413e7b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:14:48.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f040a5d00 0x7f8f0413e7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:14:48.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f040a47b0 0x7f8f041cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:48.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f040a47b0 0x7f8f041cf680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60996/0 (socket says 172.21.15.114:60996) 2024-09-16T05:14:48.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 1 -- 172.21.15.114:0/2107496567 learned_addr learned my addr 172.21.15.114:0/2107496567 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:14:48.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 10 monclient(hunting): get_auth_request con 0x7f8f040a47b0 auth_method 0 2024-09-16T05:14:48.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:48.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:48.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:48.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:48.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:48.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:48.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 10 monclient(hunting): handle_auth_done global_id 74229 payload 824 2024-09-16T05:14:48.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 1 -- 172.21.15.114:0/2107496567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f040a5d00 msgr2=0x7f8f0413e7b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:14:48.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 1 --2- 172.21.15.114:0/2107496567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f040a5d00 0x7f8f0413e7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:48.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 1 -- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04137730 msgr2=0x7f8f0413ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:48.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 1 --2- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04137730 0x7f8f0413ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:14:48.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 10 monclient: _finish_hunting 0 2024-09-16T05:14:48.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 1 monclient: found mon.noname-a 2024-09-16T05:14:48.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:14:48.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:14:48.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 1 -- 172.21.15.114:0/2107496567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0413f230 con 0x7f8f040a47b0 2024-09-16T05:14:48.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0259c640 1 --2- 172.21.15.114:0/2107496567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f040a47b0 0x7f8f041cf680 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40ae040 tx=0x7f8ef401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:48.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efaffd640 10 client.?.objecter ms_handle_connect 0x7f8f040a47b0 2024-09-16T05:14:48.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efaffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:14:48.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efaffd640 1 -- 172.21.15.114:0/2107496567 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef40ba020 con 0x7f8f040a47b0 2024-09-16T05:14:48.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:48.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:14:48.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efaffd640 10 monclient: dump: 2024-09-16T05:14:48.118 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:48.119 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:48.119 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:48.119 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:48.119 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:48.119 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:48.119 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:48.119 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:48.119 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:48.119 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:48.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efaffd640 10 monclient: _finish_auth 0 2024-09-16T05:14:48.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efaffd640 10 monclient: _check_auth_tickets 2024-09-16T05:14:48.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:14:48.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efaffd640 1 -- 172.21.15.114:0/2107496567 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ef40b6620 con 0x7f8f040a47b0 2024-09-16T05:14:48.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 5 monclient: authenticate success, global_id 74229 2024-09-16T05:14:48.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient: _renew_subs 2024-09-16T05:14:48.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:48.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 -- 172.21.15.114:0/2107496567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f0413cfa0 con 0x7f8f040a47b0 2024-09-16T05:14:48.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:14:48.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efaffd640 1 -- 172.21.15.114:0/2107496567 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef40b0040 con 0x7f8f040a47b0 2024-09-16T05:14:48.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:48.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:14:48.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8efaffd640 10 monclient: dump: 2024-09-16T05:14:48.120 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:48.120 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:48.120 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:48.120 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:48.120 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:48.120 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:48.121 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:48.121 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:48.121 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:48.121 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:48.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 client.74229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:14:48.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient: _renew_subs 2024-09-16T05:14:48.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:14:48.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f0159a640 4 set_mon_vals no callback set 2024-09-16T05:14:48.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 -- 172.21.15.114:0/2107496567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f0413f940 con 0x7f8f040a47b0 2024-09-16T05:14:48.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.104+0000 7f8f08a79640 1 librados: init done 2024-09-16T05:14:48.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8f08a79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:48.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8f08a79640 1 --2- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04139480 0x7f8f04139860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:48.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8f08a79640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:48.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8f08a79640 1 -- 172.21.15.114:0/2107496567 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f8f040a4270 con 0x7f8f04139480 2024-09-16T05:14:48.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 1 -- 172.21.15.114:0/2107496567 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8ef40bd020 con 0x7f8f040a47b0 2024-09-16T05:14:48.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 1 --2- 172.21.15.114:0/2107496567 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8ee40377c0 0x7f8ee4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:48.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 1 -- 172.21.15.114:0/2107496567 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(294..294 src has 1..294) ==== 6308+0+0 (secure 0 0 0) 0x7f8ef40e3520 con 0x7f8f040a47b0 2024-09-16T05:14:48.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 10 client.74229.objecter ms_dispatch 0x7f8f04000ba0 osd_map(294..294 src has 1..294) 2024-09-16T05:14:48.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 3 client.74229.objecter handle_osd_map got epochs [294,294] > 0 2024-09-16T05:14:48.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 3 client.74229.objecter handle_osd_map decoding full epoch 294 2024-09-16T05:14:48.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efbfff640 1 --2- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04139480 0x7f8f04139860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:48.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 20 client.74229.objecter dump_active .. 0 homeless 2024-09-16T05:14:48.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efbfff640 10 monclient: get_auth_request con 0x7f8f04139480 auth_method 0 2024-09-16T05:14:48.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:48.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:48.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:48.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:48.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:48.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:48.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8f02d9d640 1 --2- 172.21.15.114:0/2107496567 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8ee40377c0 0x7f8ee4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:48.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8f02d9d640 10 monclient: get_auth_request con 0x7f8ee40377c0 auth_method 0 2024-09-16T05:14:48.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efbfff640 10 monclient(hunting): handle_auth_done global_id 64399 payload 824 2024-09-16T05:14:48.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efbfff640 1 -- 172.21.15.114:0/2107496567 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f040008e0 con 0x7f8f04139480 2024-09-16T05:14:48.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 10 client.74229.objecter ms_handle_connect 0x7f8ee40377c0 2024-09-16T05:14:48.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8f02d9d640 1 --2- 172.21.15.114:0/2107496567 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8ee40377c0 0x7f8ee4039c80 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7f8efc0012f0 tx=0x7f8efc007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:48.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 10 client.74229.objecter ms_handle_connect 0x7f8f04139480 2024-09-16T05:14:48.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 10 client.74229.objecter resend_mon_ops 2024-09-16T05:14:48.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efbfff640 1 --2- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04139480 0x7f8f04139860 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8f0409de70 tx=0x7f8eec00b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:48.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 1 -- 172.21.15.114:0/2107496567 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eec003e20 con 0x7f8f04139480 2024-09-16T05:14:48.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:48.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:14:48.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 10 monclient: dump: 2024-09-16T05:14:48.123 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:48.124 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:48.124 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:48.124 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:48.124 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:48.124 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:48.124 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:48.124 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:48.124 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:48.124 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:48.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 1 -- 172.21.15.114:0/2107496567 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8eec015070 con 0x7f8f04139480 2024-09-16T05:14:48.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:14:48.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:14:48.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 1 -- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04139480 msgr2=0x7f8f04139860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:48.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 1 --2- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04139480 0x7f8f04139860 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8f0409de70 tx=0x7f8eec00b8e0 comp rx=0 tx=0).stop 2024-09-16T05:14:48.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 1 -- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04139480 msgr2=0x7f8f04139860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:48.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.108+0000 7f8efaffd640 1 --2- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04139480 0x7f8f04139860 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8f0409de70 tx=0x7f8eec00b8e0 comp rx=0 tx=0).stop 2024-09-16T05:14:48.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:14:48.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 1 --2- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04139da0 0x7f8f0409ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:14:48.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 10 monclient(hunting): start opening mon connection 2024-09-16T05:14:48.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 1 -- 172.21.15.114:0/2107496567 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0413a1b0 con 0x7f8f04139da0 2024-09-16T05:14:48.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efbfff640 1 --2- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04139da0 0x7f8f0409ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:14:48.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efbfff640 10 monclient: get_auth_request con 0x7f8f04139da0 auth_method 0 2024-09-16T05:14:48.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:14:48.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:14:48.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:14:48.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:14:48.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:14:48.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:14:48.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efbfff640 10 monclient(hunting): handle_auth_done global_id 64402 payload 824 2024-09-16T05:14:48.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efbfff640 1 -- 172.21.15.114:0/2107496567 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8f0413cc20 con 0x7f8f04139da0 2024-09-16T05:14:48.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efbfff640 1 --2- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04139da0 0x7f8f0409ecb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00bfd0 tx=0x7f8eec005a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:14:48.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efaffd640 10 client.74229.objecter ms_handle_connect 0x7f8f04139da0 2024-09-16T05:14:48.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efaffd640 10 client.74229.objecter resend_mon_ops 2024-09-16T05:14:48.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efaffd640 1 -- 172.21.15.114:0/2107496567 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eec01ae00 con 0x7f8f04139da0 2024-09-16T05:14:48.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:14:48.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:14:48.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efaffd640 10 monclient: dump: 2024-09-16T05:14:48.128 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:14:48.128 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:14:48.128 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:48.128 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:14:48.128 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:14:48.128 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:14:48.129 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:14:48.129 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:14:48.129 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:14:48.129 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:14:48.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efaffd640 1 -- 172.21.15.114:0/2107496567 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f8eec005800 con 0x7f8f04139da0 2024-09-16T05:14:48.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:14:48.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:14:48.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efaffd640 1 -- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04139da0 msgr2=0x7f8f0409ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:48.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efaffd640 1 --2- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04139da0 0x7f8f0409ecb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00bfd0 tx=0x7f8eec005a30 comp rx=0 tx=0).stop 2024-09-16T05:14:48.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efaffd640 1 -- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04139da0 msgr2=0x7f8f0409ecb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:14:48.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8efaffd640 1 --2- 172.21.15.114:0/2107496567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f04139da0 0x7f8f0409ecb0 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00bfd0 tx=0x7f8eec005a30 comp rx=0 tx=0).stop 2024-09-16T05:14:48.130 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:14:48.130 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:14:48.130 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:14:48.130 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:14:48.130 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 38, 2024-09-16T05:14:48.130 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 172249, 2024-09-16T05:14:48.130 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:14:48.131 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:14:48.131 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-16T05:14:48.132 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:48.132 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:14:48.132 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:48.132 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:14:48.132 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:14:48.132 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:48.132 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:48.132 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:48.132 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:48.132 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:48.132 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:48.132 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:48.133 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:48.133 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:48.133 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:48.133 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:48.133 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:48.133 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:14:48.133 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:14:48.133 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:48.133 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:48.133 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:48.133 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:48.133 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:48.133 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:48.133 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:48.133 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:48.134 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:48.134 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:48.134 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:48.134 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:48.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:48.134 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:14:48.134 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:14:48.134 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:14:48.134 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:14:48.134 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:14:48.134 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:14:48.134 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:48.134 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:14:48.134 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:14:48.134 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:14:48.135 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:14:48.135 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:14:48.135 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:14:48.135 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:14:48.135 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:14:48.135 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:14:48.135 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:14:48.135 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:14:48.135 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:14:48.135 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:14:48.135 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:14:48.135 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:14:48.135 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:14:48.136 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:14:48.136 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:14:48.136 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:14:48.136 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:14:48.136 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:14:48.136 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:48.136 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:14:48.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:48.136 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:14:48.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:48.136 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:14:48.136 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:14:48.136 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:48.137 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:48.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:48.137 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:48.137 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:14:48.137 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:48.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:48.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:48.137 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:48.137 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:14:48.137 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:48.137 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:48.137 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:48.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:48.137 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:14:48.137 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:14:48.138 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:48.138 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:48.138 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:48.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:48.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:48.138 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:14:48.138 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:14:48.138 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:48.138 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:48.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:48.138 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:48.138 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:14:48.138 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:48.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:48.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:48.139 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:48.139 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:14:48.139 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:48.139 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:48.139 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:48.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:48.139 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:14:48.139 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:14:48.139 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:48.139 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:48.139 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:48.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:48.140 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:14:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:48.142 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:48.142 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:14:48.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:48.142 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:48.142 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:48.142 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:14:48.142 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:48.142 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:48.142 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:14:48.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:48.142 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:48.142 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:48.142 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:14:48.142 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:48.143 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:14:48.143 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:14:48.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:14:48.143 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:14:48.143 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:14:48.143 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:14:48.143 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:14:48.143 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:14:48.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:14:48.143 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:14:48.143 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:14:48.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 10 librados: watch_flush enter 2024-09-16T05:14:48.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 10 librados: watch_flush exit 2024-09-16T05:14:48.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 20 client.74229.objecter shutdown clearing up homeless session... 2024-09-16T05:14:48.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 10 client.74229.objecter successfully canceled tick 2024-09-16T05:14:48.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 1 -- 172.21.15.114:0/2107496567 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8ee40377c0 msgr2=0x7f8ee4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:48.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 1 --2- 172.21.15.114:0/2107496567 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8ee40377c0 0x7f8ee4039c80 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7f8efc0012f0 tx=0x7f8efc007d40 comp rx=0 tx=0).stop 2024-09-16T05:14:48.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 10 monclient: shutdown 2024-09-16T05:14:48.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:14:48.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 1 -- 172.21.15.114:0/2107496567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f040a47b0 msgr2=0x7f8f041cf680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:14:48.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 1 --2- 172.21.15.114:0/2107496567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f040a47b0 0x7f8f041cf680 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40ae040 tx=0x7f8ef401ace0 comp rx=0 tx=0).stop 2024-09-16T05:14:48.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f02d9d640 1 -- 172.21.15.114:0/2107496567 reap_dead start 2024-09-16T05:14:48.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f02d9d640 1 -- 172.21.15.114:0/2107496567 reap_dead start 2024-09-16T05:14:48.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 1 -- 172.21.15.114:0/2107496567 shutdown_connections 2024-09-16T05:14:48.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 1 -- 172.21.15.114:0/2107496567 >> 172.21.15.114:0/2107496567 conn(0x7f8f040a9c70 msgr2=0x7f8f04136220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:14:48.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 1 -- 172.21.15.114:0/2107496567 shutdown_connections 2024-09-16T05:14:48.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 1 -- 172.21.15.114:0/2107496567 wait complete. 2024-09-16T05:14:48.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:14:48.116+0000 7f8f08a79640 1 librados: shutdown 2024-09-16T05:14:48.146 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:14:48.146 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:15:05.040 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ 2 2024-09-16T05:15:05.040 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3041 ms) 2024-09-16T05:15:05.040 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-16T05:15:05.040 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6951 ms) 2024-09-16T05:15:05.040 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-16T05:15:05.040 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6989 ms) 2024-09-16T05:15:05.041 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-16T05:15:05.041 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (18631 ms) 2024-09-16T05:15:05.041 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-16T05:15:05.041 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (21408 ms) 2024-09-16T05:15:05.041 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-16T05:15:05.041 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7048 ms) 2024-09-16T05:15:05.041 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-16T05:15:05.041 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7003 ms) 2024-09-16T05:15:05.041 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (193840 ms total) 2024-09-16T05:15:05.041 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: 2024-09-16T05:15:05.041 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-16T05:15:05.041 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (375729 ms total) 2024-09-16T05:15:05.041 INFO:tasks.workunit.client.0.smithi149.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-16T05:16:18.148 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T05:16:18.148 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T05:16:18.148 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T05:16:18.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T05:16:18.153 INFO:tasks.ceph.mon.c:Started 2024-09-16T05:16:18.153 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:16:18.153 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:16:18.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 10 monclient: get_monmap_and_config 2024-09-16T05:16:18.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 10 monclient: build_initial_monmap 2024-09-16T05:16:18.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 10 monclient: monmap: 2024-09-16T05:16:18.270 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:18.270 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:18.270 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:18.273313+0000 2024-09-16T05:16:18.270 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:18.273313+0000 2024-09-16T05:16:18.270 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:18.271 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:18.271 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:18.271 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:18.271 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:18.271 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:18.271 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:18.271 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:18.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 1 Processor -- start 2024-09-16T05:16:18.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 1 -- start start 2024-09-16T05:16:18.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 10 monclient: init 2024-09-16T05:16:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-16T05:16:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc87c1376e0 0x7fc87c137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 10 monclient(hunting): picked mon.noname-c con 0x7fc87c1376e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:16:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 0x7fc87c0a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 10 monclient(hunting): picked mon.noname-b con 0x7fc87c1370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc87c0a5d00 addr v1:172.21.15.149:6789/0 2024-09-16T05:16:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc87c09bd20 con 0x7fc87c0a5d00 2024-09-16T05:16:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fc87c138000 con 0x7fc87c1376e0 2024-09-16T05:16:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc87c09c5c0 con 0x7fc87c1370a0 2024-09-16T05:16:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc884029640 10 monclient(hunting): authenticate will time out at 1462.902153s 2024-09-16T05:16:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc8815c4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 0x7fc87c0a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc8815c4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 0x7fc87c0a4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:46166/0 (socket says 172.21.15.114:46166) 2024-09-16T05:16:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc8815c4640 1 -- 172.21.15.114:0/893544571 learned_addr learned my addr 172.21.15.114:0/893544571 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc8815c4640 10 monclient(hunting): get_auth_request con 0x7fc87c1370a0 auth_method 0 2024-09-16T05:16:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc8815c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc8815c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc8815c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc8815c4640 1 -- 172.21.15.114:0/893544571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 msgr2=0x7fc87c0a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:16:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc8815c4640 1 -- 172.21.15.114:0/893544571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 msgr2=0x7fc87c0a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:16:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc8815c4640 1 --2- 172.21.15.114:0/893544571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 0x7fc87c0a4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:16:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc8815c4640 1 --2- 172.21.15.114:0/893544571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 0x7fc87c0a4270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 1 --2- 172.21.15.114:0/893544571 >> v2:172.21.15.149:3300/0 conn(0x7fc87c1376e0 0x7fc87c137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 10 monclient(hunting): get_auth_request con 0x7fc87c1376e0 auth_method 0 2024-09-16T05:16:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc880dc3640 1 -- 172.21.15.114:0/893544571 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1691667794 0 0) 0x7fc87c09bd20 con 0x7fc87c0a5d00 2024-09-16T05:16:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc880dc3640 10 monclient(hunting): _negotiate 2024-09-16T05:16:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc880dc3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc880dc3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:18.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc880dc3640 10 monclient(hunting): my global_id is 64420 2024-09-16T05:16:18.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc880dc3640 1 -- 172.21.15.114:0/893544571 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc868003530 con 0x7fc87c0a5d00 2024-09-16T05:16:18.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 10 monclient(hunting): handle_auth_done global_id 64417 payload 274 2024-09-16T05:16:18.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 1 -- 172.21.15.114:0/893544571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 msgr2=0x7fc87c0a4270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:16:18.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 1 --2- 172.21.15.114:0/893544571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 0x7fc87c0a4270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:18.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 1 -- 172.21.15.114:0/893544571 >> v1:172.21.15.149:6789/0 conn(0x7fc87c0a5d00 legacy=0x7fc87c0a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:18.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 10 monclient: _finish_hunting 0 2024-09-16T05:16:18.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 1 monclient: found mon.noname-c 2024-09-16T05:16:18.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:18.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:16:18.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.271+0000 7fc881dc5640 1 -- 172.21.15.114:0/893544571 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87c09c000 con 0x7fc87c1376e0 2024-09-16T05:16:18.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc881dc5640 1 --2- 172.21.15.114:0/893544571 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc87c1376e0 0x7fc87c137ac0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc8780097c0 tx=0x7fc87801c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fbe6ca0dea394d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:18.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc880dc3640 1 -- 172.21.15.114:0/893544571 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8780b0070 con 0x7fc87c1376e0 2024-09-16T05:16:18.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc880dc3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc880dc3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:16:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc880dc3640 10 monclient: dump: 2024-09-16T05:16:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:18.279 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc880dc3640 10 monclient: _finish_auth 0 2024-09-16T05:16:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc880dc3640 10 monclient: _check_auth_tickets 2024-09-16T05:16:18.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc880dc3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:18.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc880dc3640 1 -- 172.21.15.114:0/893544571 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8780ab600 con 0x7fc87c1376e0 2024-09-16T05:16:18.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc880dc3640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:18.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc880dc3640 1 -- 172.21.15.114:0/893544571 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8780ab9d0 con 0x7fc87c1376e0 2024-09-16T05:16:18.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc880dc3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:18.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc880dc3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:16:18.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc880dc3640 10 monclient: dump: 2024-09-16T05:16:18.280 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:18.280 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:18.280 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:18.280 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:18.280 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:18.280 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:18.281 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 5 monclient: authenticate success, global_id 64417 2024-09-16T05:16:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 4 set_mon_vals no callback set 2024-09-16T05:16:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 10 monclient: shutdown 2024-09-16T05:16:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 -- 172.21.15.114:0/893544571 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc87c1376e0 msgr2=0x7fc87c137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 --2- 172.21.15.114:0/893544571 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc87c1376e0 0x7fc87c137ac0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc8780097c0 tx=0x7fc87801c360 comp rx=0 tx=0).stop 2024-09-16T05:16:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 -- 172.21.15.114:0/893544571 shutdown_connections 2024-09-16T05:16:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 --2- 172.21.15.114:0/893544571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 0x7fc87c0a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 --2- 172.21.15.114:0/893544571 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc87c1376e0 0x7fc87c137ac0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 -- 172.21.15.114:0/893544571 >> 172.21.15.114:0/893544571 conn(0x7fc87c0a9a40 msgr2=0x7fc87c0a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 -- 172.21.15.114:0/893544571 shutdown_connections 2024-09-16T05:16:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 -- 172.21.15.114:0/893544571 wait complete. 2024-09-16T05:16:18.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 10 monclient: build_initial_monmap 2024-09-16T05:16:18.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 10 monclient: monmap: 2024-09-16T05:16:18.287 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:18.287 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:18.287 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:18.287 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:18.287 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:18.288 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:18.288 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:18.288 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:18.288 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:18.288 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:18.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 librados: starting msgr at 2024-09-16T05:16:18.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 librados: starting objecter 2024-09-16T05:16:18.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 Processor -- start 2024-09-16T05:16:18.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 -- start start 2024-09-16T05:16:18.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 librados: setting wanted keys 2024-09-16T05:16:18.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 librados: calling monclient init 2024-09-16T05:16:18.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 10 monclient: init 2024-09-16T05:16:18.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:18.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:16:18.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc87c0a5d00 0x7fc87c135d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:18.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 10 monclient(hunting): picked mon.noname-a con 0x7fc87c0a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:18.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 0x7fc87c1362d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:18.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 10 monclient(hunting): picked mon.noname-c con 0x7fc87c1370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:18.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc87c1376e0 0x7fc87c13bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:18.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 10 monclient(hunting): picked mon.noname-b con 0x7fc87c1376e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:18.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:18.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c141d80 con 0x7fc87c0a5d00 2024-09-16T05:16:18.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:18.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c141c00 con 0x7fc87c1376e0 2024-09-16T05:16:18.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:18.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc87c141f00 con 0x7fc87c1370a0 2024-09-16T05:16:18.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:18.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc884029640 10 monclient(hunting): authenticate will time out at 1462.907762s 2024-09-16T05:16:18.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc8815c4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 0x7fc87c1362d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:18.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc881dc5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc87c0a5d00 0x7fc87c135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:18.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc8815c4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 0x7fc87c1362d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:46172/0 (socket says 172.21.15.114:46172) 2024-09-16T05:16:18.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc881dc5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc87c0a5d00 0x7fc87c135d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52888/0 (socket says 172.21.15.114:52888) 2024-09-16T05:16:18.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc8815c4640 1 -- 172.21.15.114:0/3673211564 learned_addr learned my addr 172.21.15.114:0/3673211564 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:18.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc8815c4640 10 monclient(hunting): get_auth_request con 0x7fc87c1370a0 auth_method 0 2024-09-16T05:16:18.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc8815c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:18.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc8815c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:18.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc8815c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:18.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc881dc5640 10 monclient(hunting): get_auth_request con 0x7fc87c0a5d00 auth_method 0 2024-09-16T05:16:18.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc881dc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:18.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc881dc5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:18.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc881dc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:18.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc881dc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:18.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.275+0000 7fc881dc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:18.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc881dc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:18.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8815c4640 1 -- 172.21.15.114:0/3673211564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 msgr2=0x7fc87c1362d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:16:18.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8815c4640 1 -- 172.21.15.114:0/3673211564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 msgr2=0x7fc87c1362d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:16:18.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8815c4640 1 --2- 172.21.15.114:0/3673211564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 0x7fc87c1362d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:16:18.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8815c4640 1 --2- 172.21.15.114:0/3673211564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 0x7fc87c1362d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:18.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc881dc5640 10 monclient(hunting): handle_auth_done global_id 74244 payload 824 2024-09-16T05:16:18.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8825c6640 1 --2- 172.21.15.114:0/3673211564 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc87c1376e0 0x7fc87c13bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:18.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc881dc5640 1 -- 172.21.15.114:0/3673211564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 msgr2=0x7fc87c1362d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:16:18.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc881dc5640 1 --2- 172.21.15.114:0/3673211564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 0x7fc87c1362d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:18.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc881dc5640 1 -- 172.21.15.114:0/3673211564 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc87c1376e0 msgr2=0x7fc87c13bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:18.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc881dc5640 1 --2- 172.21.15.114:0/3673211564 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc87c1376e0 0x7fc87c13bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:18.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc881dc5640 10 monclient: _finish_hunting 0 2024-09-16T05:16:18.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc881dc5640 1 monclient: found mon.noname-a 2024-09-16T05:16:18.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc881dc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:18.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc881dc5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:18.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc881dc5640 1 -- 172.21.15.114:0/3673211564 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87c13e330 con 0x7fc87c0a5d00 2024-09-16T05:16:18.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8825c6640 10 monclient: get_auth_request con 0x7fc87c1376e0 auth_method 0 2024-09-16T05:16:18.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8825c6640 1 --2- 172.21.15.114:0/3673211564 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc87c1376e0 0x7fc87c13bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:16:18.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc881dc5640 1 --2- 172.21.15.114:0/3673211564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc87c0a5d00 0x7fc87c135d90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc8780ab000 tx=0x7fc878019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:18.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 10 client.?.objecter ms_handle_connect 0x7fc87c0a5d00 2024-09-16T05:16:18.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:18.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 1 -- 172.21.15.114:0/3673211564 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8780b8020 con 0x7fc87c0a5d00 2024-09-16T05:16:18.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:18.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:18.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 10 monclient: dump: 2024-09-16T05:16:18.293 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:18.293 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:18.293 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:18.293 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:18.293 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:18.293 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:18.293 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:18.293 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:18.293 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:18.293 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:18.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 10 monclient: _finish_auth 0 2024-09-16T05:16:18.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 10 monclient: _check_auth_tickets 2024-09-16T05:16:18.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:18.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 1 -- 172.21.15.114:0/3673211564 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8780b4410 con 0x7fc87c0a5d00 2024-09-16T05:16:18.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc884029640 5 monclient: authenticate success, global_id 74244 2024-09-16T05:16:18.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc884029640 10 monclient: _renew_subs 2024-09-16T05:16:18.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc884029640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:18.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc884029640 1 -- 172.21.15.114:0/3673211564 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc87c120890 con 0x7fc87c0a5d00 2024-09-16T05:16:18.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:18.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 1 -- 172.21.15.114:0/3673211564 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8780b0040 con 0x7fc87c0a5d00 2024-09-16T05:16:18.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:18.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:18.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 10 monclient: dump: 2024-09-16T05:16:18.294 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:18.294 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:18.295 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:18.295 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:18.295 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:18.295 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:18.295 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:18.295 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:18.295 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:18.295 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:18.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc884029640 10 client.74244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:18.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc884029640 10 monclient: _renew_subs 2024-09-16T05:16:18.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc873fff640 4 set_mon_vals no callback set 2024-09-16T05:16:18.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc884029640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:18.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc884029640 1 -- 172.21.15.114:0/3673211564 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc87c0a25a0 con 0x7fc87c0a5d00 2024-09-16T05:16:18.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc884029640 1 librados: init done 2024-09-16T05:16:18.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc884029640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:18.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc884029640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:18.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc884029640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:18.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc884029640 1 -- 172.21.15.114:0/3673211564 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc87c073d20 con 0x7fc87c0a5d00 2024-09-16T05:16:18.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 1 -- 172.21.15.114:0/3673211564 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc8780bb020 con 0x7fc87c0a5d00 2024-09-16T05:16:18.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 1 --2- 172.21.15.114:0/3673211564 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc860037810 0x7fc860039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:18.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 1 -- 172.21.15.114:0/3673211564 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(349..349 src has 1..349) ==== 5406+0+0 (secure 0 0 0) 0x7fc8780e2920 con 0x7fc87c0a5d00 2024-09-16T05:16:18.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 10 client.74244.objecter ms_dispatch 0x7fc87c000ba0 osd_map(349..349 src has 1..349) 2024-09-16T05:16:18.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 3 client.74244.objecter handle_osd_map got epochs [349,349] > 0 2024-09-16T05:16:18.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 3 client.74244.objecter handle_osd_map decoding full epoch 349 2024-09-16T05:16:18.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 20 client.74244.objecter dump_active .. 0 homeless 2024-09-16T05:16:18.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8815c4640 1 --2- 172.21.15.114:0/3673211564 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc860037810 0x7fc860039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:18.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8815c4640 10 monclient: get_auth_request con 0x7fc860037810 auth_method 0 2024-09-16T05:16:18.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8727fc640 10 client.74244.objecter ms_handle_connect 0x7fc860037810 2024-09-16T05:16:18.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.279+0000 7fc8815c4640 1 --2- 172.21.15.114:0/3673211564 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc860037810 0x7fc860039cd0 secure :-1 s=READY pgs=3515 cs=0 l=1 rev1=1 crypto rx=0x7fc87c13d650 tx=0x7fc86c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:18.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.283+0000 7fc8727fc640 1 -- 172.21.15.114:0/3673211564 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc8780c2030 con 0x7fc87c0a5d00 2024-09-16T05:16:18.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.283+0000 7fc8727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:18.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.283+0000 7fc8727fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:18.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.507+0000 7fc884029640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:16:18.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.507+0000 7fc884029640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:18.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.507+0000 7fc884029640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.507+0000 7fc884029640 1 -- 172.21.15.114:0/3673211564 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc87c137ac0 con 0x7fc87c0a5d00 2024-09-16T05:16:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.507+0000 7fc8727fc640 1 -- 172.21.15.114:0/3673211564 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fc878016070 con 0x7fc87c0a5d00 2024-09-16T05:16:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.507+0000 7fc8727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.507+0000 7fc8727fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:18.509 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":38,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":92,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:16:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 10 librados: watch_flush enter 2024-09-16T05:16:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 10 librados: watch_flush exit 2024-09-16T05:16:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 20 client.74244.objecter shutdown clearing up homeless session... 2024-09-16T05:16:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 10 client.74244.objecter successfully canceled tick 2024-09-16T05:16:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 1 -- 172.21.15.114:0/3673211564 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc860037810 msgr2=0x7fc860039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 1 --2- 172.21.15.114:0/3673211564 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc860037810 0x7fc860039cd0 secure :-1 s=READY pgs=3515 cs=0 l=1 rev1=1 crypto rx=0x7fc87c13d650 tx=0x7fc86c008040 comp rx=0 tx=0).stop 2024-09-16T05:16:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 10 monclient: shutdown 2024-09-16T05:16:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 1 -- 172.21.15.114:0/3673211564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc87c0a5d00 msgr2=0x7fc87c135d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 1 --2- 172.21.15.114:0/3673211564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc87c0a5d00 0x7fc87c135d90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc8780ab000 tx=0x7fc878019100 comp rx=0 tx=0).stop 2024-09-16T05:16:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 1 -- 172.21.15.114:0/3673211564 shutdown_connections 2024-09-16T05:16:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 1 --2- 172.21.15.114:0/3673211564 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc860037810 0x7fc860039cd0 unknown :-1 s=CLOSED pgs=3515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 1 --2- 172.21.15.114:0/3673211564 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc87c1376e0 0x7fc87c13bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 1 --2- 172.21.15.114:0/3673211564 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc87c1370a0 0x7fc87c1362d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 1 --2- 172.21.15.114:0/3673211564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc87c0a5d00 0x7fc87c135d90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 1 -- 172.21.15.114:0/3673211564 >> 172.21.15.114:0/3673211564 conn(0x7fc87c0a9a40 msgr2=0x7fc87c0a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 1 -- 172.21.15.114:0/3673211564 shutdown_connections 2024-09-16T05:16:18.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 1 -- 172.21.15.114:0/3673211564 wait complete. 2024-09-16T05:16:18.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:18.511+0000 7fc884029640 1 librados: shutdown 2024-09-16T05:16:21.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:16:21.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 10 monclient: get_monmap_and_config 2024-09-16T05:16:21.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 10 monclient: build_initial_monmap 2024-09-16T05:16:21.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 10 monclient: monmap: 2024-09-16T05:16:21.658 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:21.658 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:21.658 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:21.661627+0000 2024-09-16T05:16:21.658 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:21.661627+0000 2024-09-16T05:16:21.658 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:21.658 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:21.659 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:21.659 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:21.659 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:21.659 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:21.659 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:21.659 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:21.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 1 Processor -- start 2024-09-16T05:16:21.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 1 -- start start 2024-09-16T05:16:21.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 10 monclient: init 2024-09-16T05:16:21.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:21.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-16T05:16:21.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe57c09b6c0 0x7fe57c0a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:21.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 10 monclient(hunting): picked mon.noname-c con 0x7fe57c09b6c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:16:21.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe57c0a1400 addr v1:172.21.15.114:6789/0 2024-09-16T05:16:21.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe57c0a1810 addr v1:172.21.15.149:6789/0 2024-09-16T05:16:21.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe57c09d4e0 con 0x7fe57c0a1400 2024-09-16T05:16:21.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe57c09f430 con 0x7fe57c0a1810 2024-09-16T05:16:21.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:21.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fe57c09e390 con 0x7fe57c09b6c0 2024-09-16T05:16:21.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:21.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 10 monclient(hunting): authenticate will time out at 1466.290807s 2024-09-16T05:16:21.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581b36640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe57c0a1400 0x7fe57c139640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37092/0 (socket says 172.21.15.114:37092) 2024-09-16T05:16:21.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581b36640 1 -- 172.21.15.114:0/412171738 learned_addr learned my addr 172.21.15.114:0/412171738 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:16:21.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe582337640 1 --2- 172.21.15.114:0/412171738 >> v2:172.21.15.149:3300/0 conn(0x7fe57c09b6c0 0x7fe57c0a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:21.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe582337640 10 monclient(hunting): get_auth_request con 0x7fe57c09b6c0 auth_method 0 2024-09-16T05:16:21.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe582337640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:21.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe582337640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:21.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe582337640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:21.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 -- 172.21.15.114:0/412171738 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2311910050 0 0) 0x7fe57c09d4e0 con 0x7fe57c0a1400 2024-09-16T05:16:21.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient(hunting): _negotiate 2024-09-16T05:16:21.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:21.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:21.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient(hunting): my global_id is 74247 2024-09-16T05:16:21.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 -- 172.21.15.114:0/412171738 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe564003530 con 0x7fe57c0a1400 2024-09-16T05:16:21.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe582337640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:21.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe582337640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:21.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe582337640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:21.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 -- 172.21.15.114:0/412171738 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2607738241 0 0) 0x7fe564003530 con 0x7fe57c0a1400 2024-09-16T05:16:21.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 -- 172.21.15.114:0/412171738 >> v2:172.21.15.149:3300/0 conn(0x7fe57c09b6c0 msgr2=0x7fe57c0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:21.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 --2- 172.21.15.114:0/412171738 >> v2:172.21.15.149:3300/0 conn(0x7fe57c09b6c0 0x7fe57c0a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:21.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 -- 172.21.15.114:0/412171738 >> v1:172.21.15.149:6789/0 conn(0x7fe57c0a1810 legacy=0x7fe57c13cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:21.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient: _finish_hunting 0 2024-09-16T05:16:21.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:16:21.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:21.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:16:21.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 -- 172.21.15.114:0/412171738 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe57c0a1cc0 con 0x7fe57c0a1400 2024-09-16T05:16:21.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient: _finish_auth 0 2024-09-16T05:16:21.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient: _check_auth_tickets 2024-09-16T05:16:21.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:21.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 5 monclient: authenticate success, global_id 74247 2024-09-16T05:16:21.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:16:21.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 -- 172.21.15.114:0/412171738 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7fe570001ba0 con 0x7fe57c0a1400 2024-09-16T05:16:21.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:21.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:16:21.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient: dump: 2024-09-16T05:16:21.672 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:21.672 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:21.672 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:21.672 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:21.672 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:21.672 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:21.673 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:21.673 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:21.673 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:21.673 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:21.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:16:21.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:21.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 -- 172.21.15.114:0/412171738 >> v1:172.21.15.114:6789/0 conn(0x7fe57c0a1400 legacy=0x7fe57c139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:21.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:16:21.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 --2- 172.21.15.114:0/412171738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5640039c0 0x7fe564003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:21.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient(hunting): picked mon.c con 0x7fe5640039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:21.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 --2- 172.21.15.114:0/412171738 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe564004810 0x7fe564004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:21.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient(hunting): picked mon.b con 0x7fe564004810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:21.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 --2- 172.21.15.114:0/412171738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5640052a0 0x7fe56400fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:21.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient(hunting): picked mon.a con 0x7fe5640052a0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:21.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:21.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 -- 172.21.15.114:0/412171738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c09e390 con 0x7fe5640052a0 2024-09-16T05:16:21.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:21.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 -- 172.21.15.114:0/412171738 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fe5640105d0 con 0x7fe564004810 2024-09-16T05:16:21.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:21.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 1 -- 172.21.15.114:0/412171738 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe5640107d0 con 0x7fe5640039c0 2024-09-16T05:16:21.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:21.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe581335640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:16:21.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe58459b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:16:21.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe582b38640 1 --2- 172.21.15.114:0/412171738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5640052a0 0x7fe56400fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:21.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe582b38640 10 monclient(hunting): get_auth_request con 0x7fe5640052a0 auth_method 0 2024-09-16T05:16:21.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe582b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:21.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe582b38640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:21.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe582b38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:16:21.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.659+0000 7fe582337640 1 --2- 172.21.15.114:0/412171738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5640039c0 0x7fe564003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:21.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582337640 10 monclient(hunting): get_auth_request con 0x7fe5640039c0 auth_method 0 2024-09-16T05:16:21.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582337640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:21.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582337640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:21.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582337640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:16:21.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:21.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:21.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581b36640 1 --2- 172.21.15.114:0/412171738 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe564004810 0x7fe564004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:21.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:16:21.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581b36640 10 monclient(hunting): get_auth_request con 0x7fe564004810 auth_method 0 2024-09-16T05:16:21.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581b36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:21.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581b36640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:21.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581b36640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:16:21.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582337640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:21.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582337640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:21.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582337640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:16:21.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 10 monclient(hunting): handle_auth_done global_id 74247 payload 293 2024-09-16T05:16:21.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 1 -- 172.21.15.114:0/412171738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5640039c0 msgr2=0x7fe564003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:21.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 1 --2- 172.21.15.114:0/412171738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5640039c0 0x7fe564003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:21.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 1 -- 172.21.15.114:0/412171738 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe564004810 msgr2=0x7fe564004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:21.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 1 --2- 172.21.15.114:0/412171738 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe564004810 0x7fe564004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:21.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 10 monclient: _finish_hunting 0 2024-09-16T05:16:21.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 1 monclient: found mon.a 2024-09-16T05:16:21.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:21.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:21.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 1 -- 172.21.15.114:0/412171738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe564003530 con 0x7fe5640052a0 2024-09-16T05:16:21.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 10 monclient: _finish_auth 0 2024-09-16T05:16:21.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 10 monclient: _check_auth_tickets 2024-09-16T05:16:21.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:21.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581b36640 1 --2- 172.21.15.114:0/412171738 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe564004810 0x7fe564004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:16:21.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 1 -- 172.21.15.114:0/412171738 reap_dead start 2024-09-16T05:16:21.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe582b38640 1 --2- 172.21.15.114:0/412171738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5640052a0 0x7fe56400fde0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe574001780 tx=0x7fe57401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af4c075093ce4b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:21.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581335640 1 -- 172.21.15.114:0/412171738 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5740013a0 con 0x7fe5640052a0 2024-09-16T05:16:21.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581335640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:21.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581335640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:21.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581335640 10 monclient: dump: 2024-09-16T05:16:21.679 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:21.679 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:21.679 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:21.679 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:21.679 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:21.679 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:21.679 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:21.680 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:21.680 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:21.680 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:21.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581335640 1 -- 172.21.15.114:0/412171738 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5740050d0 con 0x7fe5640052a0 2024-09-16T05:16:21.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581335640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:21.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581335640 1 -- 172.21.15.114:0/412171738 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5740054a0 con 0x7fe5640052a0 2024-09-16T05:16:21.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe58459b640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:21.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe58459b640 4 set_mon_vals no callback set 2024-09-16T05:16:21.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581335640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:21.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581335640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:21.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe581335640 10 monclient: dump: 2024-09-16T05:16:21.680 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:21.680 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:21.681 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:21.681 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:21.681 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:21.681 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:21.681 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:21.681 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:21.681 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:21.681 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:21.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe58459b640 10 monclient: shutdown 2024-09-16T05:16:21.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe58459b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:21.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe58459b640 1 -- 172.21.15.114:0/412171738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5640052a0 msgr2=0x7fe56400fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:21.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe58459b640 1 --2- 172.21.15.114:0/412171738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5640052a0 0x7fe56400fde0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe574001780 tx=0x7fe57401c400 comp rx=0 tx=0).stop 2024-09-16T05:16:21.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe58459b640 1 -- 172.21.15.114:0/412171738 shutdown_connections 2024-09-16T05:16:21.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe58459b640 1 --2- 172.21.15.114:0/412171738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5640052a0 0x7fe56400fde0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:21.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe58459b640 1 -- 172.21.15.114:0/412171738 >> 172.21.15.114:0/412171738 conn(0x7fe57c0a1fb0 msgr2=0x7fe57c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:21.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe58459b640 1 -- 172.21.15.114:0/412171738 shutdown_connections 2024-09-16T05:16:21.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe58459b640 1 -- 172.21.15.114:0/412171738 wait complete. 2024-09-16T05:16:21.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe58459b640 10 monclient: build_initial_monmap 2024-09-16T05:16:21.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.663+0000 7fe58459b640 10 monclient: monmap: 2024-09-16T05:16:21.682 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:21.682 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:21.682 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:21.683 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:21.683 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:21.683 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:21.683 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:21.683 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:21.683 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:21.683 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:21.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 librados: starting msgr at 2024-09-16T05:16:21.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 librados: starting objecter 2024-09-16T05:16:21.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 Processor -- start 2024-09-16T05:16:21.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 -- start start 2024-09-16T05:16:21.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 librados: setting wanted keys 2024-09-16T05:16:21.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 librados: calling monclient init 2024-09-16T05:16:21.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient: init 2024-09-16T05:16:21.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:21.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:16:21.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5640052a0 0x7fe57c0983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:21.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient(hunting): picked mon.noname-b con 0x7fe5640052a0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:21.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe57c0988e0 0x7fe57c139520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:21.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient(hunting): picked mon.noname-a con 0x7fe57c0988e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:21.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe57c098d40 0x7fe57c139a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:21.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient(hunting): picked mon.noname-c con 0x7fe57c098d40 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:21.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:21.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c0a0ec0 con 0x7fe57c0988e0 2024-09-16T05:16:21.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:21.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c09cad0 con 0x7fe5640052a0 2024-09-16T05:16:21.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:21.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe57c0a1040 con 0x7fe57c098d40 2024-09-16T05:16:21.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:21.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient(hunting): authenticate will time out at 1466.296765s 2024-09-16T05:16:21.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582b38640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe57c098d40 0x7fe57c139a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:21.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe57c0988e0 0x7fe57c139520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:21.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582b38640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe57c098d40 0x7fe57c139a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:46194/0 (socket says 172.21.15.114:46194) 2024-09-16T05:16:21.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582b38640 1 -- 172.21.15.114:0/2867156097 learned_addr learned my addr 172.21.15.114:0/2867156097 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:21.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582b38640 10 monclient(hunting): get_auth_request con 0x7fe57c098d40 auth_method 0 2024-09-16T05:16:21.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:21.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582b38640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:21.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:21.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe57c0988e0 0x7fe57c139520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52938/0 (socket says 172.21.15.114:52938) 2024-09-16T05:16:21.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 10 monclient(hunting): get_auth_request con 0x7fe57c0988e0 auth_method 0 2024-09-16T05:16:21.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:21.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:21.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:21.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582b38640 1 -- 172.21.15.114:0/2867156097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe57c098d40 msgr2=0x7fe57c139a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:16:21.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582b38640 1 -- 172.21.15.114:0/2867156097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe57c098d40 msgr2=0x7fe57c139a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:16:21.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582b38640 1 --2- 172.21.15.114:0/2867156097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe57c098d40 0x7fe57c139a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:16:21.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:21.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:21.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:21.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582b38640 1 --2- 172.21.15.114:0/2867156097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe57c098d40 0x7fe57c139a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:21.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582337640 1 --2- 172.21.15.114:0/2867156097 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5640052a0 0x7fe57c0983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:21.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582337640 10 monclient(hunting): get_auth_request con 0x7fe5640052a0 auth_method 0 2024-09-16T05:16:21.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582337640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:21.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582337640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:21.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582337640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:21.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 10 monclient(hunting): handle_auth_done global_id 74250 payload 824 2024-09-16T05:16:21.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 1 -- 172.21.15.114:0/2867156097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe57c098d40 msgr2=0x7fe57c139a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:16:21.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 1 --2- 172.21.15.114:0/2867156097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe57c098d40 0x7fe57c139a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:21.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 1 -- 172.21.15.114:0/2867156097 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5640052a0 msgr2=0x7fe57c0983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:21.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 1 --2- 172.21.15.114:0/2867156097 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5640052a0 0x7fe57c0983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:21.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 10 monclient: _finish_hunting 0 2024-09-16T05:16:21.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 1 monclient: found mon.noname-a 2024-09-16T05:16:21.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:21.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:21.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 1 -- 172.21.15.114:0/2867156097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe57c13d0c0 con 0x7fe57c0988e0 2024-09-16T05:16:21.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe581b36640 1 --2- 172.21.15.114:0/2867156097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe57c0988e0 0x7fe57c139520 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe57c09d6b0 tx=0x7fe57000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:21.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 10 client.?.objecter ms_handle_connect 0x7fe57c0988e0 2024-09-16T05:16:21.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:21.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 1 -- 172.21.15.114:0/2867156097 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe570005d40 con 0x7fe57c0988e0 2024-09-16T05:16:21.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:21.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:21.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 10 monclient: dump: 2024-09-16T05:16:21.688 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:21.688 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:21.688 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:21.688 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:21.689 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:21.689 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:21.689 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:21.689 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:21.689 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:21.689 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:21.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 10 monclient: _finish_auth 0 2024-09-16T05:16:21.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 10 monclient: _check_auth_tickets 2024-09-16T05:16:21.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:21.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 1 -- 172.21.15.114:0/2867156097 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe570018070 con 0x7fe57c0988e0 2024-09-16T05:16:21.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:21.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 1 -- 172.21.15.114:0/2867156097 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57000c8e0 con 0x7fe57c0988e0 2024-09-16T05:16:21.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 5 monclient: authenticate success, global_id 74250 2024-09-16T05:16:21.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient: _renew_subs 2024-09-16T05:16:21.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:21.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 -- 172.21.15.114:0/2867156097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe57c09c6a0 con 0x7fe57c0988e0 2024-09-16T05:16:21.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe580b34640 4 set_mon_vals no callback set 2024-09-16T05:16:21.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:21.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:21.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 10 monclient: dump: 2024-09-16T05:16:21.691 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:21.691 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:21.691 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:21.691 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:21.691 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:21.691 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:21.691 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:21.691 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:21.691 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:21.691 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:21.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 client.74250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:21.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient: _renew_subs 2024-09-16T05:16:21.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:21.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 -- 172.21.15.114:0/2867156097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe57c13bba0 con 0x7fe57c0988e0 2024-09-16T05:16:21.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 librados: init done 2024-09-16T05:16:21.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:21.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 1 -- 172.21.15.114:0/2867156097 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe57000ca80 con 0x7fe57c0988e0 2024-09-16T05:16:21.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:21.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe58459b640 1 -- 172.21.15.114:0/2867156097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe57c13d830 con 0x7fe57c0988e0 2024-09-16T05:16:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 1 --2- 172.21.15.114:0/2867156097 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe568037810 0x7fe568039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 1 -- 172.21.15.114:0/2867156097 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(352..352 src has 1..352) ==== 5406+0+0 (secure 0 0 0) 0x7fe570048ff0 con 0x7fe57c0988e0 2024-09-16T05:16:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 10 client.74250.objecter ms_dispatch 0x7fe57c000ba0 osd_map(352..352 src has 1..352) 2024-09-16T05:16:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 3 client.74250.objecter handle_osd_map got epochs [352,352] > 0 2024-09-16T05:16:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 3 client.74250.objecter handle_osd_map decoding full epoch 352 2024-09-16T05:16:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe57affd640 20 client.74250.objecter dump_active .. 0 homeless 2024-09-16T05:16:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582337640 1 --2- 172.21.15.114:0/2867156097 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe568037810 0x7fe568039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.667+0000 7fe582337640 10 monclient: get_auth_request con 0x7fe568037810 auth_method 0 2024-09-16T05:16:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.671+0000 7fe57affd640 10 client.74250.objecter ms_handle_connect 0x7fe568037810 2024-09-16T05:16:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.671+0000 7fe582337640 1 --2- 172.21.15.114:0/2867156097 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe568037810 0x7fe568039cd0 secure :-1 s=READY pgs=3516 cs=0 l=1 rev1=1 crypto rx=0x7fe56c009aa0 tx=0x7fe56c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.671+0000 7fe57affd640 1 -- 172.21.15.114:0/2867156097 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fe57001a310 con 0x7fe57c0988e0 2024-09-16T05:16:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.671+0000 7fe57affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.671+0000 7fe57affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:21.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.899+0000 7fe58459b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:16:21.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.899+0000 7fe58459b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:21.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.899+0000 7fe58459b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:21.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.899+0000 7fe58459b640 1 -- 172.21.15.114:0/2867156097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe57c06de60 con 0x7fe57c0988e0 2024-09-16T05:16:21.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.899+0000 7fe57affd640 1 -- 172.21.15.114:0/2867156097 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fe5700477e0 con 0x7fe57c0988e0 2024-09-16T05:16:21.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.899+0000 7fe57affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:21.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.899+0000 7fe57affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:21.899 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":38,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":95,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:16:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 10 librados: watch_flush enter 2024-09-16T05:16:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 10 librados: watch_flush exit 2024-09-16T05:16:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 20 client.74250.objecter shutdown clearing up homeless session... 2024-09-16T05:16:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 10 client.74250.objecter successfully canceled tick 2024-09-16T05:16:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 1 -- 172.21.15.114:0/2867156097 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe568037810 msgr2=0x7fe568039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 1 --2- 172.21.15.114:0/2867156097 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe568037810 0x7fe568039cd0 secure :-1 s=READY pgs=3516 cs=0 l=1 rev1=1 crypto rx=0x7fe56c009aa0 tx=0x7fe56c008040 comp rx=0 tx=0).stop 2024-09-16T05:16:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 10 monclient: shutdown 2024-09-16T05:16:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 1 -- 172.21.15.114:0/2867156097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe57c0988e0 msgr2=0x7fe57c139520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 1 --2- 172.21.15.114:0/2867156097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe57c0988e0 0x7fe57c139520 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe57c09d6b0 tx=0x7fe57000ed50 comp rx=0 tx=0).stop 2024-09-16T05:16:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 1 -- 172.21.15.114:0/2867156097 shutdown_connections 2024-09-16T05:16:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 1 --2- 172.21.15.114:0/2867156097 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe568037810 0x7fe568039cd0 unknown :-1 s=CLOSED pgs=3516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 1 --2- 172.21.15.114:0/2867156097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe57c0988e0 0x7fe57c139520 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 1 --2- 172.21.15.114:0/2867156097 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe57c098d40 0x7fe57c139a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 1 --2- 172.21.15.114:0/2867156097 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5640052a0 0x7fe57c0983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 1 -- 172.21.15.114:0/2867156097 >> 172.21.15.114:0/2867156097 conn(0x7fe57c0a1fb0 msgr2=0x7fe57c0717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 1 -- 172.21.15.114:0/2867156097 shutdown_connections 2024-09-16T05:16:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 1 -- 172.21.15.114:0/2867156097 wait complete. 2024-09-16T05:16:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:21.903+0000 7fe58459b640 1 librados: shutdown 2024-09-16T05:16:24.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:16:25.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 10 monclient: get_monmap_and_config 2024-09-16T05:16:25.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 10 monclient: build_initial_monmap 2024-09-16T05:16:25.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 10 monclient: monmap: 2024-09-16T05:16:25.050 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:25.050 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:25.050 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:25.053585+0000 2024-09-16T05:16:25.050 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:25.053585+0000 2024-09-16T05:16:25.050 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:25.050 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:25.050 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:25.051 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:25.051 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:25.051 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:25.051 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:25.051 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:25.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 1 Processor -- start 2024-09-16T05:16:25.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 1 -- start start 2024-09-16T05:16:25.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 10 monclient: init 2024-09-16T05:16:25.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:25.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T05:16:25.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fbcd80994b0 0x7fbcd8099890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:25.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 10 monclient(hunting): picked mon.noname-c con 0x7fbcd80994b0 addr v2:172.21.15.149:3300/0 2024-09-16T05:16:25.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd809c1f0 0x7fbcd8099dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:25.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 10 monclient(hunting): picked mon.noname-b con 0x7fbcd809c1f0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:25.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbcd80a1ab0 0x7fbcd813b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:25.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 10 monclient(hunting): picked mon.noname-a con 0x7fbcd80a1ab0 addr v2:172.21.15.114:3300/0 2024-09-16T05:16:25.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:25.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fbcd809a310 con 0x7fbcd80a1ab0 2024-09-16T05:16:25.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:25.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fbcd809bdd0 con 0x7fbcd80994b0 2024-09-16T05:16:25.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:25.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd809b530 con 0x7fbcd809c1f0 2024-09-16T05:16:25.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:25.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcdcb5a640 10 monclient(hunting): authenticate will time out at 1469.683538s 2024-09-16T05:16:25.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcd6d9d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbcd80a1ab0 0x7fbcd813b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:25.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbccdd9b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd809c1f0 0x7fbcd8099dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:25.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcd6d9d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbcd80a1ab0 0x7fbcd813b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52958/0 (socket says 172.21.15.114:52958) 2024-09-16T05:16:25.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcd6d9d640 1 -- 172.21.15.114:0/3501198190 learned_addr learned my addr 172.21.15.114:0/3501198190 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:25.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcd6d9d640 10 monclient(hunting): get_auth_request con 0x7fbcd80a1ab0 auth_method 0 2024-09-16T05:16:25.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcd6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:25.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcd6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:25.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcd6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:25.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbccdd9b640 10 monclient(hunting): get_auth_request con 0x7fbcd809c1f0 auth_method 0 2024-09-16T05:16:25.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbccdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:25.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbccdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:25.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbccdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:25.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcd659c640 1 --2- 172.21.15.114:0/3501198190 >> v2:172.21.15.149:3300/0 conn(0x7fbcd80994b0 0x7fbcd8099890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:25.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.051+0000 7fbcd659c640 10 monclient(hunting): get_auth_request con 0x7fbcd80994b0 auth_method 0 2024-09-16T05:16:25.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:25.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:25.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:25.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:25.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:25.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:25.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbccdd9b640 1 -- 172.21.15.114:0/3501198190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd809c1f0 msgr2=0x7fbcd8099dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:16:25.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbccdd9b640 1 -- 172.21.15.114:0/3501198190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd809c1f0 msgr2=0x7fbcd8099dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:16:25.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbccdd9b640 1 --2- 172.21.15.114:0/3501198190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd809c1f0 0x7fbcd8099dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:16:25.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbccdd9b640 1 --2- 172.21.15.114:0/3501198190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd809c1f0 0x7fbcd8099dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:25.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:25.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:25.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:25.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd6d9d640 10 monclient(hunting): handle_auth_done global_id 74253 payload 274 2024-09-16T05:16:25.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd6d9d640 1 -- 172.21.15.114:0/3501198190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd809c1f0 msgr2=0x7fbcd8099dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:16:25.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd6d9d640 1 --2- 172.21.15.114:0/3501198190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd809c1f0 0x7fbcd8099dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:25.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd6d9d640 1 -- 172.21.15.114:0/3501198190 >> v2:172.21.15.149:3300/0 conn(0x7fbcd80994b0 msgr2=0x7fbcd8099890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:25.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd6d9d640 1 --2- 172.21.15.114:0/3501198190 >> v2:172.21.15.149:3300/0 conn(0x7fbcd80994b0 0x7fbcd8099890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:25.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd6d9d640 10 monclient: _finish_hunting 0 2024-09-16T05:16:25.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd6d9d640 1 monclient: found mon.noname-a 2024-09-16T05:16:25.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:25.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:25.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd6d9d640 1 -- 172.21.15.114:0/3501198190 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd80a1810 con 0x7fbcd80a1ab0 2024-09-16T05:16:25.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd6d9d640 1 --2- 172.21.15.114:0/3501198190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcd80a1ab0 0x7fbcd813b760 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80a3050 tx=0x7fbcd001c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e29b54d8fd6b9bc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:25.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd5d9b640 1 -- 172.21.15.114:0/3501198190 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd00a5520 con 0x7fbcd80a1ab0 2024-09-16T05:16:25.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:25.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:25.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd5d9b640 10 monclient: dump: 2024-09-16T05:16:25.064 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:25.064 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:25.064 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:25.064 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:25.064 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:25.064 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:25.065 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:25.065 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:25.065 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:25.065 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:25.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd5d9b640 10 monclient: _finish_auth 0 2024-09-16T05:16:25.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd5d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:16:25.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:25.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd5d9b640 1 -- 172.21.15.114:0/3501198190 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcd00a56c0 con 0x7fbcd80a1ab0 2024-09-16T05:16:25.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 5 monclient: authenticate success, global_id 74253 2024-09-16T05:16:25.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:16:25.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd5d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:25.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:25.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 4 set_mon_vals no callback set 2024-09-16T05:16:25.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd5d9b640 1 -- 172.21.15.114:0/3501198190 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd00a5a90 con 0x7fbcd80a1ab0 2024-09-16T05:16:25.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:25.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:25.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcd5d9b640 10 monclient: dump: 2024-09-16T05:16:25.066 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:25.066 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:25.066 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:25.066 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:25.066 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:25.066 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:25.066 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:25.066 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:25.066 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:25.066 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 10 monclient: shutdown 2024-09-16T05:16:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 1 -- 172.21.15.114:0/3501198190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcd80a1ab0 msgr2=0x7fbcd813b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 1 --2- 172.21.15.114:0/3501198190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcd80a1ab0 0x7fbcd813b760 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80a3050 tx=0x7fbcd001c680 comp rx=0 tx=0).stop 2024-09-16T05:16:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 1 -- 172.21.15.114:0/3501198190 shutdown_connections 2024-09-16T05:16:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 1 --2- 172.21.15.114:0/3501198190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcd80a1ab0 0x7fbcd813b760 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 1 --2- 172.21.15.114:0/3501198190 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd809c1f0 0x7fbcd8099dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 1 --2- 172.21.15.114:0/3501198190 >> v2:172.21.15.149:3300/0 conn(0x7fbcd80994b0 0x7fbcd8099890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 1 -- 172.21.15.114:0/3501198190 >> 172.21.15.114:0/3501198190 conn(0x7fbcd80a1fb0 msgr2=0x7fbcd80a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 1 -- 172.21.15.114:0/3501198190 shutdown_connections 2024-09-16T05:16:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 1 -- 172.21.15.114:0/3501198190 wait complete. 2024-09-16T05:16:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 10 monclient: build_initial_monmap 2024-09-16T05:16:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.055+0000 7fbcdcb5a640 10 monclient: monmap: 2024-09-16T05:16:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:25.068 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 librados: starting msgr at 2024-09-16T05:16:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 librados: starting objecter 2024-09-16T05:16:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 Processor -- start 2024-09-16T05:16:25.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 -- start start 2024-09-16T05:16:25.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 librados: setting wanted keys 2024-09-16T05:16:25.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 librados: calling monclient init 2024-09-16T05:16:25.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient: init 2024-09-16T05:16:25.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:25.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:16:25.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd80994b0 0x7fbcd81d3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:25.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient(hunting): picked mon.noname-c con 0x7fbcd80994b0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:25.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcd809c1f0 0x7fbcd8072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:25.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient(hunting): picked mon.noname-a con 0x7fbcd809c1f0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:25.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbcd80a1ab0 0x7fbcd8073470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:25.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient(hunting): picked mon.noname-b con 0x7fbcd80a1ab0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:25.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:25.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd81455d0 con 0x7fbcd809c1f0 2024-09-16T05:16:25.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:25.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd8145450 con 0x7fbcd80a1ab0 2024-09-16T05:16:25.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:25.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd8145750 con 0x7fbcd80994b0 2024-09-16T05:16:25.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:25.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient(hunting): authenticate will time out at 1469.688868s 2024-09-16T05:16:25.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd659c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd80994b0 0x7fbcd81d3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd659c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd80994b0 0x7fbcd81d3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:46212/0 (socket says 172.21.15.114:46212) 2024-09-16T05:16:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcd809c1f0 0x7fbcd8072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd659c640 1 -- 172.21.15.114:0/336728008 learned_addr learned my addr 172.21.15.114:0/336728008 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd659c640 10 monclient(hunting): get_auth_request con 0x7fbcd80994b0 auth_method 0 2024-09-16T05:16:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 10 monclient(hunting): get_auth_request con 0x7fbcd809c1f0 auth_method 0 2024-09-16T05:16:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd659c640 1 -- 172.21.15.114:0/336728008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd80994b0 msgr2=0x7fbcd81d3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:16:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd659c640 1 -- 172.21.15.114:0/336728008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd80994b0 msgr2=0x7fbcd81d3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:16:25.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:25.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:25.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd659c640 1 --2- 172.21.15.114:0/336728008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd80994b0 0x7fbcd81d3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:16:25.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:25.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd659c640 1 --2- 172.21.15.114:0/336728008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd80994b0 0x7fbcd81d3a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:25.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 10 monclient(hunting): handle_auth_done global_id 74256 payload 824 2024-09-16T05:16:25.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 1 -- 172.21.15.114:0/336728008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd80994b0 msgr2=0x7fbcd81d3a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:16:25.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 1 --2- 172.21.15.114:0/336728008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd80994b0 0x7fbcd81d3a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:25.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd6d9d640 1 --2- 172.21.15.114:0/336728008 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbcd80a1ab0 0x7fbcd8073470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:25.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 1 -- 172.21.15.114:0/336728008 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbcd80a1ab0 msgr2=0x7fbcd8073470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:25.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 1 --2- 172.21.15.114:0/336728008 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbcd80a1ab0 0x7fbcd8073470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:25.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 10 monclient: _finish_hunting 0 2024-09-16T05:16:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 1 monclient: found mon.noname-a 2024-09-16T05:16:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 1 -- 172.21.15.114:0/336728008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd81d3f50 con 0x7fbcd809c1f0 2024-09-16T05:16:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccdd9b640 1 --2- 172.21.15.114:0/336728008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcd809c1f0 0x7fbcd8072f30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000d5a0 tx=0x7fbcc000da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 10 client.?.objecter ms_handle_connect 0x7fbcd809c1f0 2024-09-16T05:16:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 1 -- 172.21.15.114:0/336728008 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0005bb0 con 0x7fbcd809c1f0 2024-09-16T05:16:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 10 monclient: dump: 2024-09-16T05:16:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:25.074 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:25.074 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:25.074 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:25.074 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:25.074 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:25.074 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:25.074 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:25.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 10 monclient: _finish_auth 0 2024-09-16T05:16:25.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:16:25.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:25.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 5 monclient: authenticate success, global_id 74256 2024-09-16T05:16:25.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 1 -- 172.21.15.114:0/336728008 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc0018070 con 0x7fbcd809c1f0 2024-09-16T05:16:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient: _renew_subs 2024-09-16T05:16:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 -- 172.21.15.114:0/336728008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd80a4c70 con 0x7fbcd809c1f0 2024-09-16T05:16:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 1 -- 172.21.15.114:0/336728008 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0006050 con 0x7fbcd809c1f0 2024-09-16T05:16:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 client.74256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 10 monclient: dump: 2024-09-16T05:16:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:25.076 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd559a640 4 set_mon_vals no callback set 2024-09-16T05:16:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient: _renew_subs 2024-09-16T05:16:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 -- 172.21.15.114:0/336728008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcd806f420 con 0x7fbcd809c1f0 2024-09-16T05:16:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 librados: init done 2024-09-16T05:16:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 1 -- 172.21.15.114:0/336728008 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbcc00061f0 con 0x7fbcd809c1f0 2024-09-16T05:16:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcdcb5a640 1 -- 172.21.15.114:0/336728008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcd813b760 con 0x7fbcd809c1f0 2024-09-16T05:16:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 1 --2- 172.21.15.114:0/336728008 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbcb8037810 0x7fbcb8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 1 -- 172.21.15.114:0/336728008 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(355..355 src has 1..355) ==== 5406+0+0 (secure 0 0 0) 0x7fbcc0048ee0 con 0x7fbcd809c1f0 2024-09-16T05:16:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 10 client.74256.objecter ms_dispatch 0x7fbcd8000ba0 osd_map(355..355 src has 1..355) 2024-09-16T05:16:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 3 client.74256.objecter handle_osd_map got epochs [355,355] > 0 2024-09-16T05:16:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 3 client.74256.objecter handle_osd_map decoding full epoch 355 2024-09-16T05:16:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 20 client.74256.objecter dump_active .. 0 homeless 2024-09-16T05:16:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd659c640 1 --2- 172.21.15.114:0/336728008 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbcb8037810 0x7fbcb8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd659c640 10 monclient: get_auth_request con 0x7fbcb8037810 auth_method 0 2024-09-16T05:16:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbccf7fe640 10 client.74256.objecter ms_handle_connect 0x7fbcb8037810 2024-09-16T05:16:25.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.059+0000 7fbcd659c640 1 --2- 172.21.15.114:0/336728008 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbcb8037810 0x7fbcb8039cd0 secure :-1 s=READY pgs=3517 cs=0 l=1 rev1=1 crypto rx=0x7fbcc80095e0 tx=0x7fbcc8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:25.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.063+0000 7fbccf7fe640 1 -- 172.21.15.114:0/336728008 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fbcc001c3c0 con 0x7fbcd809c1f0 2024-09-16T05:16:25.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.063+0000 7fbccf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:25.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.063+0000 7fbccf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:25.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.299+0000 7fbcdcb5a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:16:25.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.299+0000 7fbcdcb5a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:25.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.299+0000 7fbcdcb5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:25.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.299+0000 7fbcdcb5a640 1 -- 172.21.15.114:0/336728008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbcd806dfd0 con 0x7fbcd809c1f0 2024-09-16T05:16:25.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.299+0000 7fbccf7fe640 1 -- 172.21.15.114:0/336728008 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fbcc001be40 con 0x7fbcd809c1f0 2024-09-16T05:16:25.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.299+0000 7fbccf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:25.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.299+0000 7fbccf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:25.300 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":38,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":99,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:16:25.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 10 librados: watch_flush enter 2024-09-16T05:16:25.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 10 librados: watch_flush exit 2024-09-16T05:16:25.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 20 client.74256.objecter shutdown clearing up homeless session... 2024-09-16T05:16:25.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 10 client.74256.objecter successfully canceled tick 2024-09-16T05:16:25.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 1 -- 172.21.15.114:0/336728008 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbcb8037810 msgr2=0x7fbcb8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:25.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 1 --2- 172.21.15.114:0/336728008 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbcb8037810 0x7fbcb8039cd0 secure :-1 s=READY pgs=3517 cs=0 l=1 rev1=1 crypto rx=0x7fbcc80095e0 tx=0x7fbcc8008040 comp rx=0 tx=0).stop 2024-09-16T05:16:25.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 10 monclient: shutdown 2024-09-16T05:16:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 1 -- 172.21.15.114:0/336728008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcd809c1f0 msgr2=0x7fbcd8072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 1 --2- 172.21.15.114:0/336728008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcd809c1f0 0x7fbcd8072f30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000d5a0 tx=0x7fbcc000da60 comp rx=0 tx=0).stop 2024-09-16T05:16:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 1 -- 172.21.15.114:0/336728008 shutdown_connections 2024-09-16T05:16:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 1 --2- 172.21.15.114:0/336728008 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbcb8037810 0x7fbcb8039cd0 unknown :-1 s=CLOSED pgs=3517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 1 --2- 172.21.15.114:0/336728008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcd809c1f0 0x7fbcd8072f30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 1 --2- 172.21.15.114:0/336728008 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbcd80a1ab0 0x7fbcd8073470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 1 --2- 172.21.15.114:0/336728008 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbcd80994b0 0x7fbcd81d3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 1 -- 172.21.15.114:0/336728008 >> 172.21.15.114:0/336728008 conn(0x7fbcd80a1fb0 msgr2=0x7fbcd8097840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 1 -- 172.21.15.114:0/336728008 shutdown_connections 2024-09-16T05:16:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 1 -- 172.21.15.114:0/336728008 wait complete. 2024-09-16T05:16:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:25.303+0000 7fbcdcb5a640 1 librados: shutdown 2024-09-16T05:16:28.318 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:16:28.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.447+0000 7f9312ee9640 10 monclient: get_monmap_and_config 2024-09-16T05:16:28.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.447+0000 7f9312ee9640 10 monclient: build_initial_monmap 2024-09-16T05:16:28.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.447+0000 7f9312ee9640 10 monclient: monmap: 2024-09-16T05:16:28.448 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:28.448 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:28.449 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:28.451794+0000 2024-09-16T05:16:28.449 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:28.451794+0000 2024-09-16T05:16:28.449 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:28.449 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:28.449 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:28.449 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:28.449 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:28.449 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:28.449 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:28.449 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:28.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9312ee9640 1 Processor -- start 2024-09-16T05:16:28.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9312ee9640 1 -- start start 2024-09-16T05:16:28.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9312ee9640 10 monclient: init 2024-09-16T05:16:28.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9312ee9640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:28.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9312ee9640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T05:16:28.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9312ee9640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f930c09f560 0x7f930c134dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9312ee9640 10 monclient(hunting): picked mon.noname-a con 0x7f930c09f560 addr v2:172.21.15.114:3300/0 2024-09-16T05:16:28.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9312ee9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f930c135300 addr v1:172.21.15.149:6789/0 2024-09-16T05:16:28.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9312ee9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f930c135770 addr v1:172.21.15.114:6789/0 2024-09-16T05:16:28.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9312ee9640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f930c09ee70 con 0x7f930c135770 2024-09-16T05:16:28.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9312ee9640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f930c135bc0 con 0x7f930c135300 2024-09-16T05:16:28.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9312ee9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:28.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9312ee9640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f930c09f940 con 0x7f930c09f560 2024-09-16T05:16:28.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9312ee9640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:28.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9312ee9640 10 monclient(hunting): authenticate will time out at 1473.081406s 2024-09-16T05:16:28.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f930c09f560 0x7f930c134dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f930c09f560 0x7f930c134dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32774/0 (socket says 172.21.15.114:32774) 2024-09-16T05:16:28.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 1 -- 172.21.15.114:0/237007505 learned_addr learned my addr 172.21.15.114:0/237007505 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:28.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 10 monclient(hunting): get_auth_request con 0x7f930c09f560 auth_method 0 2024-09-16T05:16:28.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:28.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:28.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:28.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:28.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:28.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:28.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 10 monclient(hunting): handle_auth_done global_id 74259 payload 274 2024-09-16T05:16:28.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 1 -- 172.21.15.114:0/237007505 >> v1:172.21.15.149:6789/0 conn(0x7f930c135300 legacy=0x7f930c1318c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:28.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 1 -- 172.21.15.114:0/237007505 >> v1:172.21.15.114:6789/0 conn(0x7f930c135770 legacy=0x7f930c131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:28.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 10 monclient: _finish_hunting 0 2024-09-16T05:16:28.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 1 monclient: found mon.noname-a 2024-09-16T05:16:28.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:28.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:28.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.451+0000 7f9310c85640 1 -- 172.21.15.114:0/237007505 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f930c0a0590 con 0x7f930c09f560 2024-09-16T05:16:28.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 1 --2- 172.21.15.114:0/237007505 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f930c09f560 0x7f930c134dc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009870 tx=0x7f92fc01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61562e4b3ccdf39 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:28.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f930b7fe640 1 -- 172.21.15.114:0/237007505 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc0b0070 con 0x7f930c09f560 2024-09-16T05:16:28.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f930b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:28.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f930b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:28.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f930b7fe640 10 monclient: dump: 2024-09-16T05:16:28.459 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:28.459 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:28.459 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.459 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.460 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:28.460 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:28.460 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:28.460 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:28.460 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:28.460 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:28.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f930b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:16:28.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f930b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:16:28.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f930b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:28.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f930b7fe640 1 -- 172.21.15.114:0/237007505 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92fc0ab600 con 0x7f930c09f560 2024-09-16T05:16:28.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f930b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:28.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f930b7fe640 1 -- 172.21.15.114:0/237007505 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc0ab9d0 con 0x7f930c09f560 2024-09-16T05:16:28.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 5 monclient: authenticate success, global_id 74259 2024-09-16T05:16:28.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 4 set_mon_vals no callback set 2024-09-16T05:16:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f930b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f930b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f930b7fe640 10 monclient: dump: 2024-09-16T05:16:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:28.461 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient: shutdown 2024-09-16T05:16:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 -- 172.21.15.114:0/237007505 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f930c09f560 msgr2=0x7f930c134dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 --2- 172.21.15.114:0/237007505 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f930c09f560 0x7f930c134dc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009870 tx=0x7f92fc01c470 comp rx=0 tx=0).stop 2024-09-16T05:16:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 -- 172.21.15.114:0/237007505 shutdown_connections 2024-09-16T05:16:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 --2- 172.21.15.114:0/237007505 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f930c09f560 0x7f930c134dc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 -- 172.21.15.114:0/237007505 >> 172.21.15.114:0/237007505 conn(0x7f930c0a7a70 msgr2=0x7f930c0a7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 -- 172.21.15.114:0/237007505 shutdown_connections 2024-09-16T05:16:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 -- 172.21.15.114:0/237007505 wait complete. 2024-09-16T05:16:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient: build_initial_monmap 2024-09-16T05:16:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient: monmap: 2024-09-16T05:16:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:28.463 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 librados: starting msgr at 2024-09-16T05:16:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 librados: starting objecter 2024-09-16T05:16:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 Processor -- start 2024-09-16T05:16:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 -- start start 2024-09-16T05:16:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 librados: setting wanted keys 2024-09-16T05:16:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 librados: calling monclient init 2024-09-16T05:16:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient: init 2024-09-16T05:16:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:16:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f930c09f560 0x7f930c1ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient(hunting): picked mon.noname-a con 0x7f930c09f560 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f930c135300 0x7f930c1cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient(hunting): picked mon.noname-b con 0x7f930c135300 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f930c135770 0x7f930c1cda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient(hunting): picked mon.noname-c con 0x7f930c135770 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f930c140b40 con 0x7f930c09f560 2024-09-16T05:16:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f930c1327d0 con 0x7f930c135300 2024-09-16T05:16:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f930c140cc0 con 0x7f930c135770 2024-09-16T05:16:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient(hunting): authenticate will time out at 1473.086826s 2024-09-16T05:16:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f930c09f560 0x7f930c1ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f930c09f560 0x7f930c1ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32780/0 (socket says 172.21.15.114:32780) 2024-09-16T05:16:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 1 -- 172.21.15.114:0/2473973430 learned_addr learned my addr 172.21.15.114:0/2473973430 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 10 monclient(hunting): get_auth_request con 0x7f930c09f560 auth_method 0 2024-09-16T05:16:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9311486640 1 --2- 172.21.15.114:0/2473973430 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f930c135770 0x7f930c1cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9311486640 10 monclient(hunting): get_auth_request con 0x7f930c135770 auth_method 0 2024-09-16T05:16:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9311486640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9311486640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9311486640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9311486640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9311486640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9311486640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 10 monclient(hunting): handle_auth_done global_id 74262 payload 824 2024-09-16T05:16:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f930bfff640 1 --2- 172.21.15.114:0/2473973430 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f930c135300 0x7f930c1cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 1 -- 172.21.15.114:0/2473973430 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f930c135770 msgr2=0x7f930c1cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 1 --2- 172.21.15.114:0/2473973430 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f930c135770 0x7f930c1cda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 1 -- 172.21.15.114:0/2473973430 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f930c135300 msgr2=0x7f930c1cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 1 --2- 172.21.15.114:0/2473973430 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f930c135300 0x7f930c1cd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 10 monclient: _finish_hunting 0 2024-09-16T05:16:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 1 monclient: found mon.noname-a 2024-09-16T05:16:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 1 -- 172.21.15.114:0/2473973430 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f930c073140 con 0x7f930c09f560 2024-09-16T05:16:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9310c85640 1 --2- 172.21.15.114:0/2473973430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f930c09f560 0x7f930c1ccfb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f92fc0ae040 tx=0x7f92fc019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f93097fa640 10 client.?.objecter ms_handle_connect 0x7f930c09f560 2024-09-16T05:16:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f93097fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f93097fa640 1 -- 172.21.15.114:0/2473973430 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc0b0040 con 0x7f930c09f560 2024-09-16T05:16:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f93097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f93097fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f93097fa640 10 monclient: dump: 2024-09-16T05:16:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:28.468 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f93097fa640 10 monclient: _finish_auth 0 2024-09-16T05:16:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f93097fa640 10 monclient: _check_auth_tickets 2024-09-16T05:16:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f93097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f93097fa640 1 -- 172.21.15.114:0/2473973430 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92fc0b9070 con 0x7f930c09f560 2024-09-16T05:16:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 5 monclient: authenticate success, global_id 74262 2024-09-16T05:16:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient: _renew_subs 2024-09-16T05:16:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 -- 172.21.15.114:0/2473973430 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f930c073420 con 0x7f930c09f560 2024-09-16T05:16:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f93097fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f93097fa640 1 -- 172.21.15.114:0/2473973430 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc0b4880 con 0x7f930c09f560 2024-09-16T05:16:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f93097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f93097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f93097fa640 10 monclient: dump: 2024-09-16T05:16:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:28.470 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 client.74262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient: _renew_subs 2024-09-16T05:16:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 -- 172.21.15.114:0/2473973430 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f930c072b80 con 0x7f930c09f560 2024-09-16T05:16:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f9312ee9640 1 librados: init done 2024-09-16T05:16:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.455+0000 7f930affd640 4 set_mon_vals no callback set 2024-09-16T05:16:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.459+0000 7f9312ee9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.459+0000 7f9312ee9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.459+0000 7f9312ee9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.459+0000 7f9312ee9640 1 -- 172.21.15.114:0/2473973430 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f930c134dc0 con 0x7f930c09f560 2024-09-16T05:16:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.459+0000 7f93097fa640 1 -- 172.21.15.114:0/2473973430 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f92fc0ab4c0 con 0x7f930c09f560 2024-09-16T05:16:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.459+0000 7f93097fa640 1 --2- 172.21.15.114:0/2473973430 >> v2:172.21.15.149:6812/1185531135 conn(0x7f92f8037810 0x7f92f8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.459+0000 7f93097fa640 1 -- 172.21.15.114:0/2473973430 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(357..357 src has 1..357) ==== 5406+0+0 (secure 0 0 0) 0x7f92fc0e2ba0 con 0x7f930c09f560 2024-09-16T05:16:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.459+0000 7f93097fa640 10 client.74262.objecter ms_dispatch 0x7f930c000ba0 osd_map(357..357 src has 1..357) 2024-09-16T05:16:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.459+0000 7f93097fa640 3 client.74262.objecter handle_osd_map got epochs [357,357] > 0 2024-09-16T05:16:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.459+0000 7f93097fa640 3 client.74262.objecter handle_osd_map decoding full epoch 357 2024-09-16T05:16:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.459+0000 7f93097fa640 20 client.74262.objecter dump_active .. 0 homeless 2024-09-16T05:16:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.459+0000 7f930bfff640 1 --2- 172.21.15.114:0/2473973430 >> v2:172.21.15.149:6812/1185531135 conn(0x7f92f8037810 0x7f92f8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.459+0000 7f930bfff640 10 monclient: get_auth_request con 0x7f92f8037810 auth_method 0 2024-09-16T05:16:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.459+0000 7f93097fa640 10 client.74262.objecter ms_handle_connect 0x7f92f8037810 2024-09-16T05:16:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.459+0000 7f930bfff640 1 --2- 172.21.15.114:0/2473973430 >> v2:172.21.15.149:6812/1185531135 conn(0x7f92f8037810 0x7f92f8039cd0 secure :-1 s=READY pgs=3518 cs=0 l=1 rev1=1 crypto rx=0x7f9300007fd0 tx=0x7f9300009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.463+0000 7f93097fa640 1 -- 172.21.15.114:0/2473973430 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f92fc0160f0 con 0x7f930c09f560 2024-09-16T05:16:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.463+0000 7f93097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.463+0000 7f93097fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:16:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 1 -- 172.21.15.114:0/2473973430 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f930c072dc0 con 0x7f930c09f560 2024-09-16T05:16:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f93097fa640 1 -- 172.21.15.114:0/2473973430 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f92fc0b4df0 con 0x7f930c09f560 2024-09-16T05:16:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f93097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f93097fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:28.690 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:16:28.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 10 librados: watch_flush enter 2024-09-16T05:16:28.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 10 librados: watch_flush exit 2024-09-16T05:16:28.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 20 client.74262.objecter shutdown clearing up homeless session... 2024-09-16T05:16:28.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 10 client.74262.objecter successfully canceled tick 2024-09-16T05:16:28.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 1 -- 172.21.15.114:0/2473973430 >> v2:172.21.15.149:6812/1185531135 conn(0x7f92f8037810 msgr2=0x7f92f8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:28.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 1 --2- 172.21.15.114:0/2473973430 >> v2:172.21.15.149:6812/1185531135 conn(0x7f92f8037810 0x7f92f8039cd0 secure :-1 s=READY pgs=3518 cs=0 l=1 rev1=1 crypto rx=0x7f9300007fd0 tx=0x7f9300009040 comp rx=0 tx=0).stop 2024-09-16T05:16:28.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 10 monclient: shutdown 2024-09-16T05:16:28.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:28.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 1 -- 172.21.15.114:0/2473973430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f930c09f560 msgr2=0x7f930c1ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:28.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 1 --2- 172.21.15.114:0/2473973430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f930c09f560 0x7f930c1ccfb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f92fc0ae040 tx=0x7f92fc019210 comp rx=0 tx=0).stop 2024-09-16T05:16:28.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 1 -- 172.21.15.114:0/2473973430 shutdown_connections 2024-09-16T05:16:28.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 1 --2- 172.21.15.114:0/2473973430 >> v2:172.21.15.149:6812/1185531135 conn(0x7f92f8037810 0x7f92f8039cd0 secure :-1 s=CLOSED pgs=3518 cs=0 l=1 rev1=1 crypto rx=0x7f9300007fd0 tx=0x7f9300009040 comp rx=0 tx=0).stop 2024-09-16T05:16:28.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 1 --2- 172.21.15.114:0/2473973430 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f930c135770 0x7f930c1cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 1 --2- 172.21.15.114:0/2473973430 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f930c135300 0x7f930c1cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 1 --2- 172.21.15.114:0/2473973430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f930c09f560 0x7f930c1ccfb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 1 -- 172.21.15.114:0/2473973430 >> 172.21.15.114:0/2473973430 conn(0x7f930c0a7a70 msgr2=0x7f930c0a5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:28.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 1 -- 172.21.15.114:0/2473973430 shutdown_connections 2024-09-16T05:16:28.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 1 -- 172.21.15.114:0/2473973430 wait complete. 2024-09-16T05:16:28.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.691+0000 7f9312ee9640 1 librados: shutdown 2024-09-16T05:16:28.706 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:16:28.706 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:16:28.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:16:28.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.827+0000 7f077917c640 10 monclient: get_monmap_and_config 2024-09-16T05:16:28.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.827+0000 7f077917c640 10 monclient: build_initial_monmap 2024-09-16T05:16:28.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.827+0000 7f077917c640 10 monclient: monmap: 2024-09-16T05:16:28.829 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:28.829 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:28.830 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:28.832141+0000 2024-09-16T05:16:28.830 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:28.832141+0000 2024-09-16T05:16:28.830 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:28.830 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:28.830 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:28.830 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:28.830 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:28.830 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:28.830 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:28.831 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:28.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 1 Processor -- start 2024-09-16T05:16:28.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 1 -- start start 2024-09-16T05:16:28.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 10 monclient: init 2024-09-16T05:16:28.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:28.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-16T05:16:28.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f077409b6c0 0x7f07740a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 10 monclient(hunting): picked mon.noname-a con 0x7f077409b6c0 addr v2:172.21.15.114:3300/0 2024-09-16T05:16:28.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07740a1400 0x7f07740a1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 10 monclient(hunting): picked mon.noname-b con 0x7f07740a1400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:28.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0774139640 addr v1:172.21.15.149:6789/0 2024-09-16T05:16:28.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f077409d4e0 con 0x7f0774139640 2024-09-16T05:16:28.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:28.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f077409e390 con 0x7f077409b6c0 2024-09-16T05:16:28.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:28.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f07740a1d50 con 0x7f07740a1400 2024-09-16T05:16:28.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:28.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077917c640 10 monclient(hunting): authenticate will time out at 1473.461300s 2024-09-16T05:16:28.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f0772d9d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f077409b6c0 0x7f07740a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07740a1400 0x7f07740a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f0772d9d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f077409b6c0 0x7f07740a0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32784/0 (socket says 172.21.15.114:32784) 2024-09-16T05:16:28.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f0772d9d640 1 -- 172.21.15.114:0/1665390270 learned_addr learned my addr 172.21.15.114:0/1665390270 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:28.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 10 monclient(hunting): get_auth_request con 0x7f07740a1400 auth_method 0 2024-09-16T05:16:28.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:28.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:28.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:28.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f0772d9d640 10 monclient(hunting): get_auth_request con 0x7f077409b6c0 auth_method 0 2024-09-16T05:16:28.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f0772d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:28.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f0772d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:28.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f0772d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:28.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:28.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:28.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:28.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f0772d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:28.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f0772d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:28.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f0772d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:28.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 10 monclient(hunting): handle_auth_done global_id 84104 payload 274 2024-09-16T05:16:28.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 1 -- 172.21.15.114:0/1665390270 >> v2:172.21.15.114:3300/0 conn(0x7f077409b6c0 msgr2=0x7f07740a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:28.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 1 --2- 172.21.15.114:0/1665390270 >> v2:172.21.15.114:3300/0 conn(0x7f077409b6c0 0x7f07740a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 1 -- 172.21.15.114:0/1665390270 >> v1:172.21.15.149:6789/0 conn(0x7f0774139640 legacy=0x7f077413ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:28.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 10 monclient: _finish_hunting 0 2024-09-16T05:16:28.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 1 monclient: found mon.noname-b 2024-09-16T05:16:28.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:28.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:16:28.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 1 -- 172.21.15.114:0/1665390270 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f077409f430 con 0x7f07740a1400 2024-09-16T05:16:28.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.831+0000 7f077259c640 1 --2- 172.21.15.114:0/1665390270 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07740a1400 0x7f07740a1810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f077409f600 tx=0x7f076801c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbc4d477ffa1a07b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:28.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0771d9b640 1 -- 172.21.15.114:0/1665390270 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07680b0070 con 0x7f07740a1400 2024-09-16T05:16:28.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0771d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:28.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0771d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:16:28.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0771d9b640 10 monclient: dump: 2024-09-16T05:16:28.848 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:28.848 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:28.848 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.848 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.848 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:28.848 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:28.849 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0771d9b640 10 monclient: _finish_auth 0 2024-09-16T05:16:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0771d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:16:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0771d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0771d9b640 1 -- 172.21.15.114:0/1665390270 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07680ab600 con 0x7f07740a1400 2024-09-16T05:16:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 5 monclient: authenticate success, global_id 84104 2024-09-16T05:16:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:16:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0771d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 4 set_mon_vals no callback set 2024-09-16T05:16:28.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0771d9b640 1 -- 172.21.15.114:0/1665390270 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07680ab9d0 con 0x7f07740a1400 2024-09-16T05:16:28.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0771d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:28.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0771d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:16:28.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0771d9b640 10 monclient: dump: 2024-09-16T05:16:28.850 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:28.850 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:28.850 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.850 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.850 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:28.850 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:28.850 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:28.850 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:28.850 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:28.850 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:28.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 10 monclient: shutdown 2024-09-16T05:16:28.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:28.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 -- 172.21.15.114:0/1665390270 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07740a1400 msgr2=0x7f07740a1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:28.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 --2- 172.21.15.114:0/1665390270 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07740a1400 0x7f07740a1810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f077409f600 tx=0x7f076801c300 comp rx=0 tx=0).stop 2024-09-16T05:16:28.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 -- 172.21.15.114:0/1665390270 shutdown_connections 2024-09-16T05:16:28.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 --2- 172.21.15.114:0/1665390270 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07740a1400 0x7f07740a1810 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 --2- 172.21.15.114:0/1665390270 >> v2:172.21.15.114:3300/0 conn(0x7f077409b6c0 0x7f07740a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 -- 172.21.15.114:0/1665390270 >> 172.21.15.114:0/1665390270 conn(0x7f07740a1fb0 msgr2=0x7f07740a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:28.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 -- 172.21.15.114:0/1665390270 shutdown_connections 2024-09-16T05:16:28.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 -- 172.21.15.114:0/1665390270 wait complete. 2024-09-16T05:16:28.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 10 monclient: build_initial_monmap 2024-09-16T05:16:28.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 10 monclient: monmap: 2024-09-16T05:16:28.851 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:28.851 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:28.852 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 librados: starting msgr at 2024-09-16T05:16:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 librados: starting objecter 2024-09-16T05:16:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 Processor -- start 2024-09-16T05:16:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 -- start start 2024-09-16T05:16:28.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 librados: setting wanted keys 2024-09-16T05:16:28.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 librados: calling monclient init 2024-09-16T05:16:28.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 10 monclient: init 2024-09-16T05:16:28.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:28.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:16:28.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f077409b6c0 0x7f07741ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 10 monclient(hunting): picked mon.noname-b con 0x7f077409b6c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:28.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07740a1400 0x7f07740985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 10 monclient(hunting): picked mon.noname-c con 0x7f07740a1400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:28.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774139640 0x7f0774098b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 10 monclient(hunting): picked mon.noname-a con 0x7f0774139640 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:28.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:28.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f077413f970 con 0x7f0774139640 2024-09-16T05:16:28.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:28.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f077409cb10 con 0x7f077409b6c0 2024-09-16T05:16:28.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:28.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f077413faf0 con 0x7f07740a1400 2024-09-16T05:16:28.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:28.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077917c640 10 monclient(hunting): authenticate will time out at 1473.466676s 2024-09-16T05:16:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774139640 0x7f0774098b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774139640 0x7f0774098b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32798/0 (socket says 172.21.15.114:32798) 2024-09-16T05:16:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 1 -- 172.21.15.114:0/2217518976 learned_addr learned my addr 172.21.15.114:0/2217518976 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 10 monclient(hunting): get_auth_request con 0x7f0774139640 auth_method 0 2024-09-16T05:16:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07740a1400 0x7f07740985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient(hunting): get_auth_request con 0x7f07740a1400 auth_method 0 2024-09-16T05:16:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0772d9d640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f077409b6c0 0x7f07741ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0772d9d640 10 monclient(hunting): get_auth_request con 0x7f077409b6c0 auth_method 0 2024-09-16T05:16:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0772d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0772d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0772d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient(hunting): handle_auth_done global_id 84107 payload 824 2024-09-16T05:16:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 1 -- 172.21.15.114:0/2217518976 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f077409b6c0 msgr2=0x7f07741ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f077409b6c0 0x7f07741ccf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 1 -- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774139640 msgr2=0x7f0774098b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774139640 0x7f0774098b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient: _finish_hunting 0 2024-09-16T05:16:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 1 monclient: found mon.noname-c 2024-09-16T05:16:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:16:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 1 -- 172.21.15.114:0/2217518976 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07740739e0 con 0x7f07740a1400 2024-09-16T05:16:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774139640 0x7f0774098b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:16:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07740a1400 0x7f07740985d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f07680ab000 tx=0x7f07680190a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 10 client.?.objecter ms_handle_connect 0x7f07740a1400 2024-09-16T05:16:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 1 -- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07740a1400 msgr2=0x7f07740985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:16:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 1 -- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07740a1400 msgr2=0x7f07740985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:16:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07740a1400 0x7f07740985d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f07680ab000 tx=0x7f07680190a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:16:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07740a1400 0x7f07740985d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f07680ab000 tx=0x7f07680190a0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07740a1400 msgr2=0x7f07740985d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07740a1400 0x7f07740985d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:16:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0760004e70 0x7f07600072c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f0760004e70 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774139640 0x7f0760009df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f0774139640 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f077409b6c0 0x7f076000c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f077409b6c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f076000d2d0 con 0x7f0774139640 2024-09-16T05:16:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f076000d450 con 0x7f077409b6c0 2024-09-16T05:16:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f076000d650 con 0x7f0760004e70 2024-09-16T05:16:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f075b7fe640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774139640 0x7f0760009df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient(hunting): get_auth_request con 0x7f0774139640 auth_method 0 2024-09-16T05:16:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0772d9d640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0760004e70 0x7f07600072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:16:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0772d9d640 10 monclient(hunting): get_auth_request con 0x7f0760004e70 auth_method 0 2024-09-16T05:16:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0772d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0772d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0772d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:16:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:16:28.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0772d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:28.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0772d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:28.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0772d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:16:28.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f077409b6c0 0x7f076000c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 10 monclient(hunting): get_auth_request con 0x7f077409b6c0 auth_method 0 2024-09-16T05:16:28.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:28.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:28.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f077359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:16:28.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.835+0000 7f0772d9d640 10 monclient(hunting): handle_auth_done global_id 84107 payload 293 2024-09-16T05:16:28.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f0772d9d640 1 -- 172.21.15.114:0/2217518976 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f077409b6c0 msgr2=0x7f076000c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:28.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f0772d9d640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f077409b6c0 0x7f076000c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f0772d9d640 1 -- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774139640 msgr2=0x7f0760009df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:28.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f0772d9d640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774139640 0x7f0760009df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f0772d9d640 10 monclient: _finish_hunting 0 2024-09-16T05:16:28.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f0772d9d640 1 monclient: found mon.noname-c 2024-09-16T05:16:28.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f0772d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:28.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f0772d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:16:28.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f0772d9d640 1 -- 172.21.15.114:0/2217518976 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076000d910 con 0x7f0760004e70 2024-09-16T05:16:28.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077259c640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774139640 0x7f0760009df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:16:28.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f0772d9d640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0760004e70 0x7f07600072c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f07640090e0 tx=0x7f07680a6000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:28.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 client.?.objecter ms_handle_connect 0x7f0760004e70 2024-09-16T05:16:28.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:28.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0764019070 con 0x7f0760004e70 2024-09-16T05:16:28.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:28.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:16:28.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 monclient: dump: 2024-09-16T05:16:28.862 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:28.863 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:16:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:16:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0764011b40 con 0x7f0760004e70 2024-09-16T05:16:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077917c640 5 monclient: authenticate success, global_id 84107 2024-09-16T05:16:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077917c640 10 monclient: _renew_subs 2024-09-16T05:16:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077917c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:16:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077917c640 1 -- 172.21.15.114:0/2217518976 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f077408e7a0 con 0x7f0760004e70 2024-09-16T05:16:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07640096a0 con 0x7f0760004e70 2024-09-16T05:16:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:16:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 monclient: dump: 2024-09-16T05:16:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.865 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.865 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:28.865 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:28.865 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:28.866 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:28.866 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:28.866 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:28.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f0770d99640 4 set_mon_vals no callback set 2024-09-16T05:16:28.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077917c640 10 client.84107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:28.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077917c640 10 monclient: _renew_subs 2024-09-16T05:16:28.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077917c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:16:28.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077917c640 1 -- 172.21.15.114:0/2217518976 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f077408d190 con 0x7f0760004e70 2024-09-16T05:16:28.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077917c640 1 librados: init done 2024-09-16T05:16:28.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f076401c070 con 0x7f0760004e70 2024-09-16T05:16:28.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077917c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:28.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077917c640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f077408ec50 0x7f07741ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077917c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:28.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077917c640 1 -- 172.21.15.114:0/2217518976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f077409c6a0 con 0x7f077408ec50 2024-09-16T05:16:28.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077259c640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f077408ec50 0x7f07741ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 1 --2- 172.21.15.114:0/2217518976 >> v2:172.21.15.149:6812/1185531135 conn(0x7f07740a1400 0x7f0760042050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077259c640 10 monclient: get_auth_request con 0x7f077408ec50 auth_method 0 2024-09-16T05:16:28.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:28.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:28.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:28.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(357..357 src has 1..357) ==== 5406+0+0 (secure 0 0 0) 0x7f0764047f30 con 0x7f0760004e70 2024-09-16T05:16:28.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 client.84107.objecter ms_dispatch 0x7f0774000ba0 osd_map(357..357 src has 1..357) 2024-09-16T05:16:28.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 3 client.84107.objecter handle_osd_map got epochs [357,357] > 0 2024-09-16T05:16:28.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 3 client.84107.objecter handle_osd_map decoding full epoch 357 2024-09-16T05:16:28.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:28.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:28.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:28.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 20 client.84107.objecter dump_active .. 0 homeless 2024-09-16T05:16:28.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077259c640 10 monclient(hunting): handle_auth_done global_id 74271 payload 824 2024-09-16T05:16:28.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077259c640 1 -- 172.21.15.114:0/2217518976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f077408d420 con 0x7f077408ec50 2024-09-16T05:16:28.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077359e640 1 --2- 172.21.15.114:0/2217518976 >> v2:172.21.15.149:6812/1185531135 conn(0x7f07740a1400 0x7f0760042050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077359e640 10 monclient: get_auth_request con 0x7f07740a1400 auth_method 0 2024-09-16T05:16:28.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 client.84107.objecter ms_handle_connect 0x7f077408ec50 2024-09-16T05:16:28.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 client.84107.objecter resend_mon_ops 2024-09-16T05:16:28.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077259c640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f077408ec50 0x7f07741ccf40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f077413cd10 tx=0x7f07680a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:28.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768006770 con 0x7f077408ec50 2024-09-16T05:16:28.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:28.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:28.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 monclient: dump: 2024-09-16T05:16:28.868 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:28.869 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:28.869 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.869 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.869 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:28.869 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:28.869 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:28.869 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:28.869 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:28.869 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:28.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f07680b9070 con 0x7f077408ec50 2024-09-16T05:16:28.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:28.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:28.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f077408ec50 msgr2=0x7f07741ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:28.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f077408ec50 0x7f07741ccf40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f077413cd10 tx=0x7f07680a8000 comp rx=0 tx=0).stop 2024-09-16T05:16:28.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f077408ec50 msgr2=0x7f07741ccf40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:28.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f077408ec50 0x7f07741ccf40 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f077413cd10 tx=0x7f07680a8000 comp rx=0 tx=0).stop 2024-09-16T05:16:28.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f075b7fe640 10 client.84107.objecter ms_handle_connect 0x7f07740a1400 2024-09-16T05:16:28.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.839+0000 7f077359e640 1 --2- 172.21.15.114:0/2217518976 >> v2:172.21.15.149:6812/1185531135 conn(0x7f07740a1400 0x7f0760042050 secure :-1 s=READY pgs=3519 cs=0 l=1 rev1=1 crypto rx=0x7f07741ce000 tx=0x7f076c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:28.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:16:28.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774072020 0x7f0774072430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:28.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:28.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 1 -- 172.21.15.114:0/2217518976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0774069390 con 0x7f0774072020 2024-09-16T05:16:28.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077259c640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774072020 0x7f0774072430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:28.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077259c640 10 monclient: get_auth_request con 0x7f0774072020 auth_method 0 2024-09-16T05:16:28.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:28.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:28.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:28.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:28.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:28.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:28.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077259c640 10 monclient(hunting): handle_auth_done global_id 74274 payload 824 2024-09-16T05:16:28.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077259c640 1 -- 172.21.15.114:0/2217518976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f07740a0ec0 con 0x7f0774072020 2024-09-16T05:16:28.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077259c640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774072020 0x7f0774072430 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f076801cba0 tx=0x7f0768006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:28.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f075b7fe640 10 client.84107.objecter ms_handle_connect 0x7f0774072020 2024-09-16T05:16:28.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f075b7fe640 10 client.84107.objecter resend_mon_ops 2024-09-16T05:16:28.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07680d0670 con 0x7f0774072020 2024-09-16T05:16:28.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f075b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:28.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f075b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:28.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f075b7fe640 10 monclient: dump: 2024-09-16T05:16:28.874 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:28.874 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:28.874 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.874 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:28.874 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:28.874 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:28.875 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:28.875 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:28.875 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:28.875 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:28.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f07680d0670 con 0x7f0774072020 2024-09-16T05:16:28.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f075b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:16:28.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f075b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:28.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774072020 msgr2=0x7f0774072430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:28.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f075b7fe640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774072020 0x7f0774072430 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f076801cba0 tx=0x7f0768006f90 comp rx=0 tx=0).stop 2024-09-16T05:16:28.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f075b7fe640 1 -- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774072020 msgr2=0x7f0774072430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:28.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f075b7fe640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0774072020 0x7f0774072430 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:28.876 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:16:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:16:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:16:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:16:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 40, 2024-09-16T05:16:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 147902, 2024-09-16T05:16:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:16:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:16:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:16:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:16:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 1, 2024-09-16T05:16:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:16:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:16:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:16:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:16:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:16:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:16:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:16:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:16:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:16:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:16:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:16:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:16:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:16:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:16:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:16:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:16:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:16:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:16:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:16:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:16:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:16:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:16:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:16:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:16:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:16:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:16:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:16:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:16:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:16:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:16:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:16:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:16:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:16:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:16:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:16:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:16:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:16:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:16:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:16:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:16:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:16:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:16:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:16:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:16:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:16:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:16:28.889 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:16:28.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 10 librados: watch_flush enter 2024-09-16T05:16:28.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 10 librados: watch_flush exit 2024-09-16T05:16:28.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 20 client.84107.objecter shutdown clearing up homeless session... 2024-09-16T05:16:28.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 10 client.84107.objecter successfully canceled tick 2024-09-16T05:16:28.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 1 -- 172.21.15.114:0/2217518976 >> v2:172.21.15.149:6812/1185531135 conn(0x7f07740a1400 msgr2=0x7f0760042050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:28.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 1 --2- 172.21.15.114:0/2217518976 >> v2:172.21.15.149:6812/1185531135 conn(0x7f07740a1400 0x7f0760042050 secure :-1 s=READY pgs=3519 cs=0 l=1 rev1=1 crypto rx=0x7f07741ce000 tx=0x7f076c009040 comp rx=0 tx=0).stop 2024-09-16T05:16:28.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 10 monclient: shutdown 2024-09-16T05:16:28.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:28.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 1 -- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0760004e70 msgr2=0x7f07600072c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:28.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 1 --2- 172.21.15.114:0/2217518976 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0760004e70 0x7f07600072c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f07640090e0 tx=0x7f07680a6000 comp rx=0 tx=0).stop 2024-09-16T05:16:28.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077359e640 1 -- 172.21.15.114:0/2217518976 reap_dead start 2024-09-16T05:16:28.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077359e640 1 -- 172.21.15.114:0/2217518976 reap_dead start 2024-09-16T05:16:28.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 1 -- 172.21.15.114:0/2217518976 shutdown_connections 2024-09-16T05:16:28.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 1 -- 172.21.15.114:0/2217518976 >> 172.21.15.114:0/2217518976 conn(0x7f07740a1fb0 msgr2=0x7f0774138970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:28.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 1 -- 172.21.15.114:0/2217518976 shutdown_connections 2024-09-16T05:16:28.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 1 -- 172.21.15.114:0/2217518976 wait complete. 2024-09-16T05:16:28.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:28.847+0000 7f077917c640 1 librados: shutdown 2024-09-16T05:16:28.893 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:16:28.893 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:16:28.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:16:29.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.007+0000 7f1e969ae640 10 monclient: get_monmap_and_config 2024-09-16T05:16:29.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.007+0000 7f1e969ae640 10 monclient: build_initial_monmap 2024-09-16T05:16:29.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.007+0000 7f1e969ae640 10 monclient: monmap: 2024-09-16T05:16:29.009 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:29.009 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.009 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:29.012090+0000 2024-09-16T05:16:29.009 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:29.012090+0000 2024-09-16T05:16:29.009 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:29.009 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.009 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:29.009 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:29.009 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:29.009 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:29.009 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:29.010 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 1 Processor -- start 2024-09-16T05:16:29.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 1 -- start start 2024-09-16T05:16:29.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 10 monclient: init 2024-09-16T05:16:29.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:29.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T05:16:29.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e9009b6c0 0x7f1e900a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 10 monclient(hunting): picked mon.noname-b con 0x7f1e9009b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:29.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f1e900a1400 0x7f1e900a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 10 monclient(hunting): picked mon.noname-c con 0x7f1e900a1400 addr v2:172.21.15.149:3300/0 2024-09-16T05:16:29.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e90139640 addr v1:172.21.15.114:6789/0 2024-09-16T05:16:29.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e9009d4e0 con 0x7f1e90139640 2024-09-16T05:16:29.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f1e9009e390 con 0x7f1e900a1400 2024-09-16T05:16:29.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1e9009ece0 con 0x7f1e9009b6c0 2024-09-16T05:16:29.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:29.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e969ae640 10 monclient(hunting): authenticate will time out at 1473.641507s 2024-09-16T05:16:29.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e9009b6c0 0x7f1e900a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e94f4b640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f1e90139640 0x7f1e9013ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33160/0 (socket says 172.21.15.114:33160) 2024-09-16T05:16:29.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e94f4b640 1 -- 172.21.15.114:0/4205418035 learned_addr learned my addr 172.21.15.114:0/4205418035 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:16:29.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 10 monclient(hunting): get_auth_request con 0x7f1e9009b6c0 auth_method 0 2024-09-16T05:16:29.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:29.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:29.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:29.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8f7fe640 1 --2- 172.21.15.114:0/4205418035 >> v2:172.21.15.149:3300/0 conn(0x7f1e900a1400 0x7f1e900a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8f7fe640 10 monclient(hunting): get_auth_request con 0x7f1e900a1400 auth_method 0 2024-09-16T05:16:29.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_done global_id 84110 payload 274 2024-09-16T05:16:29.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8effd640 1 -- 172.21.15.114:0/4205418035 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1468234001 0 0) 0x7f1e9009d4e0 con 0x7f1e90139640 2024-09-16T05:16:29.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 1 -- 172.21.15.114:0/4205418035 >> v2:172.21.15.149:3300/0 conn(0x7f1e900a1400 msgr2=0x7f1e900a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:29.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 1 --2- 172.21.15.114:0/4205418035 >> v2:172.21.15.149:3300/0 conn(0x7f1e900a1400 0x7f1e900a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 1 -- 172.21.15.114:0/4205418035 >> v1:172.21.15.114:6789/0 conn(0x7f1e90139640 legacy=0x7f1e9013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:16:29.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 1 monclient: found mon.noname-b 2024-09-16T05:16:29.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:29.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:16:29.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8ffff640 1 -- 172.21.15.114:0/4205418035 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9009f430 con 0x7f1e9009b6c0 2024-09-16T05:16:29.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.011+0000 7f1e8effd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:16:29.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8ffff640 1 --2- 172.21.15.114:0/4205418035 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e9009b6c0 0x7f1e900a0ec0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1e800097c0 tx=0x7f1e8001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bc78da6c52cdc6d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:29.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8effd640 1 -- 172.21.15.114:0/4205418035 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e800b0070 con 0x7f1e9009b6c0 2024-09-16T05:16:29.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:16:29.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8effd640 10 monclient: dump: 2024-09-16T05:16:29.022 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.022 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.022 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.023 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.023 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.023 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.023 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.023 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.023 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.023 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8effd640 10 monclient: _finish_auth 0 2024-09-16T05:16:29.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8effd640 10 monclient: _check_auth_tickets 2024-09-16T05:16:29.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:29.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 5 monclient: authenticate success, global_id 84110 2024-09-16T05:16:29.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:16:29.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8effd640 1 -- 172.21.15.114:0/4205418035 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e800ab600 con 0x7f1e9009b6c0 2024-09-16T05:16:29.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:29.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:29.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 4 set_mon_vals no callback set 2024-09-16T05:16:29.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8effd640 1 -- 172.21.15.114:0/4205418035 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e800ab9d0 con 0x7f1e9009b6c0 2024-09-16T05:16:29.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:16:29.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8effd640 10 monclient: dump: 2024-09-16T05:16:29.024 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.024 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.024 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.024 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.024 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.024 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.024 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.025 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.025 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.025 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient: shutdown 2024-09-16T05:16:29.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:29.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 -- 172.21.15.114:0/4205418035 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e9009b6c0 msgr2=0x7f1e900a0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 --2- 172.21.15.114:0/4205418035 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e9009b6c0 0x7f1e900a0ec0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1e800097c0 tx=0x7f1e8001c360 comp rx=0 tx=0).stop 2024-09-16T05:16:29.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 -- 172.21.15.114:0/4205418035 shutdown_connections 2024-09-16T05:16:29.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 --2- 172.21.15.114:0/4205418035 >> v2:172.21.15.149:3300/0 conn(0x7f1e900a1400 0x7f1e900a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 --2- 172.21.15.114:0/4205418035 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e9009b6c0 0x7f1e900a0ec0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 -- 172.21.15.114:0/4205418035 >> 172.21.15.114:0/4205418035 conn(0x7f1e900a1fb0 msgr2=0x7f1e900a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:29.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 -- 172.21.15.114:0/4205418035 shutdown_connections 2024-09-16T05:16:29.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 -- 172.21.15.114:0/4205418035 wait complete. 2024-09-16T05:16:29.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient: build_initial_monmap 2024-09-16T05:16:29.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient: monmap: 2024-09-16T05:16:29.026 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:29.026 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:29.026 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:29.026 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:29.026 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:29.026 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.026 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:29.026 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:29.026 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:29.026 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 librados: starting msgr at 2024-09-16T05:16:29.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 librados: starting objecter 2024-09-16T05:16:29.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 Processor -- start 2024-09-16T05:16:29.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 -- start start 2024-09-16T05:16:29.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 librados: setting wanted keys 2024-09-16T05:16:29.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 librados: calling monclient init 2024-09-16T05:16:29.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient: init 2024-09-16T05:16:29.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:29.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:16:29.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e9009b6c0 0x7f1e90139060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient(hunting): picked mon.noname-b con 0x7f1e9009b6c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:29.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e900a1400 0x7f1e900994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient(hunting): picked mon.noname-c con 0x7f1e900a1400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:29.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e90139640 0x7f1e900999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient(hunting): picked mon.noname-a con 0x7f1e90139640 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:29.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9013f970 con 0x7f1e90139640 2024-09-16T05:16:29.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9009cad0 con 0x7f1e9009b6c0 2024-09-16T05:16:29.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1e9013faf0 con 0x7f1e900a1400 2024-09-16T05:16:29.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:29.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient(hunting): authenticate will time out at 1473.646518s 2024-09-16T05:16:29.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e900a1400 0x7f1e900994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e94f4b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e90139640 0x7f1e900999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e900a1400 0x7f1e900994b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:50684/0 (socket says 172.21.15.114:50684) 2024-09-16T05:16:29.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 1 -- 172.21.15.114:0/2076399135 learned_addr learned my addr 172.21.15.114:0/2076399135 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:29.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 10 monclient(hunting): get_auth_request con 0x7f1e900a1400 auth_method 0 2024-09-16T05:16:29.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e94f4b640 10 monclient(hunting): get_auth_request con 0x7f1e90139640 auth_method 0 2024-09-16T05:16:29.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e94f4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e94f4b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e94f4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:29.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:29.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:29.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e94f4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:29.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e94f4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:29.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e94f4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:29.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8ffff640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e9009b6c0 0x7f1e90139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8ffff640 10 monclient(hunting): get_auth_request con 0x7f1e9009b6c0 auth_method 0 2024-09-16T05:16:29.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 10 monclient(hunting): handle_auth_done global_id 84113 payload 824 2024-09-16T05:16:29.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 1 -- 172.21.15.114:0/2076399135 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e9009b6c0 msgr2=0x7f1e90139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e9009b6c0 0x7f1e90139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 1 -- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e90139640 msgr2=0x7f1e900999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e90139640 0x7f1e900999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:16:29.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 1 monclient: found mon.noname-c 2024-09-16T05:16:29.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:29.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:16:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 1 -- 172.21.15.114:0/2076399135 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e90094320 con 0x7f1e900a1400 2024-09-16T05:16:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e94f4b640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e90139640 0x7f1e900999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:16:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8f7fe640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e900a1400 0x7f1e900994b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400d830 tx=0x7f1e8400dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8cff9640 10 client.?.objecter ms_handle_connect 0x7f1e900a1400 2024-09-16T05:16:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e840059a0 con 0x7f1e900a1400 2024-09-16T05:16:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:16:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8cff9640 10 monclient: dump: 2024-09-16T05:16:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.032 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.032 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.032 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.032 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8cff9640 10 monclient: _finish_auth 0 2024-09-16T05:16:29.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8cff9640 10 monclient: _check_auth_tickets 2024-09-16T05:16:29.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:29.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e84005b40 con 0x7f1e900a1400 2024-09-16T05:16:29.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 5 monclient: authenticate success, global_id 84113 2024-09-16T05:16:29.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient: _renew_subs 2024-09-16T05:16:29.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:16:29.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 -- 172.21.15.114:0/2076399135 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e90139bb0 con 0x7f1e900a1400 2024-09-16T05:16:29.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:29.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e84005d30 con 0x7f1e900a1400 2024-09-16T05:16:29.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:16:29.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8cff9640 10 monclient: dump: 2024-09-16T05:16:29.033 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.033 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.033 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.033 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.033 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.033 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.033 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.033 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.033 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.034 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e8e7fc640 4 set_mon_vals no callback set 2024-09-16T05:16:29.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 client.84113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:29.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient: _renew_subs 2024-09-16T05:16:29.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:16:29.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 -- 172.21.15.114:0/2076399135 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e90093d60 con 0x7f1e900a1400 2024-09-16T05:16:29.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.015+0000 7f1e969ae640 1 librados: init done 2024-09-16T05:16:29.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e969ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:29.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e969ae640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90099f30 0x7f1e900709b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e969ae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e969ae640 1 -- 172.21.15.114:0/2076399135 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1e90069390 con 0x7f1e90099f30 2024-09-16T05:16:29.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90099f30 0x7f1e900709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient: get_auth_request con 0x7f1e90099f30 auth_method 0 2024-09-16T05:16:29.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:29.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:29.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1e8401b020 con 0x7f1e900a1400 2024-09-16T05:16:29.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:29.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_done global_id 84116 payload 824 2024-09-16T05:16:29.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 1 -- 172.21.15.114:0/2076399135 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e9009a310 con 0x7f1e90099f30 2024-09-16T05:16:29.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 1 --2- 172.21.15.114:0/2076399135 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1e7c037810 0x7f1e7c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(357..357 src has 1..357) ==== 5406+0+0 (secure 0 0 0) 0x7f1e84048d90 con 0x7f1e900a1400 2024-09-16T05:16:29.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 10 client.84113.objecter ms_dispatch 0x7f1e90000ba0 osd_map(357..357 src has 1..357) 2024-09-16T05:16:29.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 3 client.84113.objecter handle_osd_map got epochs [357,357] > 0 2024-09-16T05:16:29.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 3 client.84113.objecter handle_osd_map decoding full epoch 357 2024-09-16T05:16:29.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 20 client.84113.objecter dump_active .. 0 homeless 2024-09-16T05:16:29.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90099f30 0x7f1e900709b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1e9009c630 tx=0x7f1e8001aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:29.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 10 client.84113.objecter ms_handle_connect 0x7f1e90099f30 2024-09-16T05:16:29.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 10 client.84113.objecter resend_mon_ops 2024-09-16T05:16:29.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 1 -- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90099f30 msgr2=0x7f1e900709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:16:29.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 1 -- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90099f30 msgr2=0x7f1e900709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:16:29.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90099f30 0x7f1e900709b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1e9009c630 tx=0x7f1e8001aba0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:16:29.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90099f30 0x7f1e900709b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1e9009c630 tx=0x7f1e8001aba0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90099f30 msgr2=0x7f1e900709b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:29.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90099f30 0x7f1e900709b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e7c03fee0 0x7f1e7c042320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90099f30 msgr2=0x7f1e900709b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:29.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90099f30 0x7f1e900709b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1e7c042980 con 0x7f1e7c03fee0 2024-09-16T05:16:29.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e7c03fee0 0x7f1e7c042320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient: get_auth_request con 0x7f1e7c03fee0 auth_method 0 2024-09-16T05:16:29.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:29.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:29.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e94f4b640 1 --2- 172.21.15.114:0/2076399135 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1e7c037810 0x7f1e7c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:29.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e94f4b640 10 monclient: get_auth_request con 0x7f1e7c037810 auth_method 0 2024-09-16T05:16:29.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_done global_id 84119 payload 824 2024-09-16T05:16:29.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 1 -- 172.21.15.114:0/2076399135 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e84001d40 con 0x7f1e7c03fee0 2024-09-16T05:16:29.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8ffff640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e7c03fee0 0x7f1e7c042320 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1e800ae040 tx=0x7f1e800a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:29.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 10 client.84113.objecter ms_handle_connect 0x7f1e7c03fee0 2024-09-16T05:16:29.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 10 client.84113.objecter resend_mon_ops 2024-09-16T05:16:29.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e800b0050 con 0x7f1e7c03fee0 2024-09-16T05:16:29.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:16:29.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 10 monclient: dump: 2024-09-16T05:16:29.038 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.038 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.038 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.038 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.038 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.038 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.038 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.039 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.039 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.039 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e94f4b640 1 --2- 172.21.15.114:0/2076399135 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1e7c037810 0x7f1e7c039cd0 secure :-1 s=READY pgs=3520 cs=0 l=1 rev1=1 crypto rx=0x7f1e9013c0b0 tx=0x7f1e88009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:29.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 10 client.84113.objecter ms_handle_connect 0x7f1e7c037810 2024-09-16T05:16:29.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1e800b1040 con 0x7f1e7c03fee0 2024-09-16T05:16:29.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:29.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:29.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e7c03fee0 msgr2=0x7f1e7c042320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e7c03fee0 0x7f1e7c042320 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1e800ae040 tx=0x7f1e800a8000 comp rx=0 tx=0).stop 2024-09-16T05:16:29.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e7c03fee0 msgr2=0x7f1e7c042320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:29.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.019+0000 7f1e8cff9640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e7c03fee0 0x7f1e7c042320 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1e800ae040 tx=0x7f1e800a8000 comp rx=0 tx=0).stop 2024-09-16T05:16:29.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.023+0000 7f1e969ae640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:16:29.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.023+0000 7f1e969ae640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90093f30 0x7f1e90070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.023+0000 7f1e969ae640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.023+0000 7f1e969ae640 1 -- 172.21.15.114:0/2076399135 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1e9013c750 con 0x7f1e90093f30 2024-09-16T05:16:29.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8ffff640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90093f30 0x7f1e90070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8ffff640 10 monclient: get_auth_request con 0x7f1e90093f30 auth_method 0 2024-09-16T05:16:29.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8ffff640 10 monclient(hunting): handle_auth_done global_id 84122 payload 824 2024-09-16T05:16:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8ffff640 1 -- 172.21.15.114:0/2076399135 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1e900a0ec0 con 0x7f1e90093f30 2024-09-16T05:16:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8ffff640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90093f30 0x7f1e90070ef0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1e80000c10 tx=0x7f1e80019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8cff9640 10 client.84113.objecter ms_handle_connect 0x7f1e90093f30 2024-09-16T05:16:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8cff9640 10 client.84113.objecter resend_mon_ops 2024-09-16T05:16:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e800b5b10 con 0x7f1e90093f30 2024-09-16T05:16:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:16:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8cff9640 10 monclient: dump: 2024-09-16T05:16:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.044 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.044 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.044 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.044 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.044 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.044 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.044 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.044 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.044 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f1e800b5b10 con 0x7f1e90093f30 2024-09-16T05:16:29.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:16:29.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:29.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90093f30 msgr2=0x7f1e90070ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8cff9640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90093f30 0x7f1e90070ef0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1e80000c10 tx=0x7f1e80019100 comp rx=0 tx=0).stop 2024-09-16T05:16:29.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8cff9640 1 -- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90093f30 msgr2=0x7f1e90070ef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:29.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e8cff9640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e90093f30 0x7f1e90070ef0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1e80000c10 tx=0x7f1e80019100 comp rx=0 tx=0).stop 2024-09-16T05:16:29.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e94f4b640 1 -- 172.21.15.114:0/2076399135 reap_dead start 2024-09-16T05:16:29.045 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:16:29.045 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:16:29.045 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:16:29.045 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:16:29.046 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 40, 2024-09-16T05:16:29.046 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 10795, 2024-09-16T05:16:29.046 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:16:29.046 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:16:29.046 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:16:29.046 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:16:29.046 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:29.046 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:16:29.046 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:29.046 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:16:29.046 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:16:29.046 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:29.046 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:29.046 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:29.047 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:29.047 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:29.047 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:29.047 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:29.047 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:29.047 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:29.047 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:29.047 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:29.047 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:29.047 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:16:29.047 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:16:29.047 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:29.047 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:29.048 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:29.048 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:29.048 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:29.048 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:29.048 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:29.048 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:29.048 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:29.048 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:29.048 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:29.048 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:29.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.048 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:16:29.048 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:16:29.048 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:16:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:16:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:16:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:16:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:16:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:16:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:16:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:16:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:16:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:16:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:16:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:16:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:16:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.051 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:16:29.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.051 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:16:29.051 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:16:29.051 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:29.051 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:29.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.051 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:29.051 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:16:29.051 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:29.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.051 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:29.052 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:16:29.052 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:29.052 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:29.052 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:29.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.052 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:16:29.052 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:16:29.052 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:29.052 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:29.052 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:29.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.052 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:16:29.052 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:16:29.053 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:29.053 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:29.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.053 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:29.053 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:16:29.053 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:29.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.053 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:29.053 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:16:29.053 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:29.053 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:29.053 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:29.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.054 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:16:29.054 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:16:29.054 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:29.054 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:29.054 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:29.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.054 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:16:29.054 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:16:29.054 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:29.054 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:29.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.054 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:29.054 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:16:29.055 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:29.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.055 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:29.055 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:16:29.055 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:29.055 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:29.055 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:29.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.055 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:16:29.055 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:16:29.055 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:29.055 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:29.055 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:29.056 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:29.056 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:29.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.056 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:16:29.056 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:16:29.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.056 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:29.056 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:29.056 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:16:29.056 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:29.056 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:29.056 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:16:29.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.057 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:29.057 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:29.057 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:16:29.057 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:29.057 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:29.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.057 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:16:29.057 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:16:29.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 10 librados: watch_flush enter 2024-09-16T05:16:29.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 10 librados: watch_flush exit 2024-09-16T05:16:29.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 20 client.84113.objecter shutdown clearing up homeless session... 2024-09-16T05:16:29.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 10 client.84113.objecter successfully canceled tick 2024-09-16T05:16:29.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 1 -- 172.21.15.114:0/2076399135 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1e7c037810 msgr2=0x7f1e7c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 1 --2- 172.21.15.114:0/2076399135 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1e7c037810 0x7f1e7c039cd0 secure :-1 s=READY pgs=3520 cs=0 l=1 rev1=1 crypto rx=0x7f1e9013c0b0 tx=0x7f1e88009040 comp rx=0 tx=0).stop 2024-09-16T05:16:29.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 10 monclient: shutdown 2024-09-16T05:16:29.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:29.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 1 -- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e900a1400 msgr2=0x7f1e900994b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e900a1400 0x7f1e900994b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400d830 tx=0x7f1e8400dcf0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 1 -- 172.21.15.114:0/2076399135 shutdown_connections 2024-09-16T05:16:29.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 1 --2- 172.21.15.114:0/2076399135 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1e7c037810 0x7f1e7c039cd0 unknown :-1 s=CLOSED pgs=3520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 1 --2- 172.21.15.114:0/2076399135 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e900a1400 0x7f1e900994b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 1 -- 172.21.15.114:0/2076399135 >> 172.21.15.114:0/2076399135 conn(0x7f1e900a1fb0 msgr2=0x7f1e900a4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:29.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 1 -- 172.21.15.114:0/2076399135 shutdown_connections 2024-09-16T05:16:29.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 1 -- 172.21.15.114:0/2076399135 wait complete. 2024-09-16T05:16:29.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.027+0000 7f1e969ae640 1 librados: shutdown 2024-09-16T05:16:29.060 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:16:29.060 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:16:29.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:16:29.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 10 monclient: get_monmap_and_config 2024-09-16T05:16:29.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 10 monclient: build_initial_monmap 2024-09-16T05:16:29.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 10 monclient: monmap: 2024-09-16T05:16:29.173 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:29.173 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.174 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:29.176957+0000 2024-09-16T05:16:29.174 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:29.176957+0000 2024-09-16T05:16:29.174 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:29.174 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.174 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:29.174 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:29.174 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:29.174 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:29.174 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:29.174 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 1 Processor -- start 2024-09-16T05:16:29.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 1 -- start start 2024-09-16T05:16:29.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 10 monclient: init 2024-09-16T05:16:29.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:29.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-16T05:16:29.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd71009b6c0 0x7fd7100a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 10 monclient(hunting): picked mon.noname-a con 0x7fd71009b6c0 addr v2:172.21.15.114:3300/0 2024-09-16T05:16:29.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7100a1400 addr v1:172.21.15.114:6789/0 2024-09-16T05:16:29.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd7100a1810 0x7fd71013cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 10 monclient(hunting): picked mon.noname-c con 0x7fd7100a1810 addr v2:172.21.15.149:3300/0 2024-09-16T05:16:29.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71009d4e0 con 0x7fd7100a1400 2024-09-16T05:16:29.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd71009e390 con 0x7fd71009b6c0 2024-09-16T05:16:29.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fd71009ece0 con 0x7fd7100a1810 2024-09-16T05:16:29.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:29.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd716057640 10 monclient(hunting): authenticate will time out at 1473.806132s 2024-09-16T05:16:29.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70effd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd7100a1400 0x7fd710139640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33164/0 (socket says 172.21.15.114:33164) 2024-09-16T05:16:29.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70effd640 1 -- 172.21.15.114:0/2092644070 learned_addr learned my addr 172.21.15.114:0/2092644070 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:16:29.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 1 --2- 172.21.15.114:0/2092644070 >> v2:172.21.15.114:3300/0 conn(0x7fd71009b6c0 0x7fd7100a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 10 monclient(hunting): get_auth_request con 0x7fd71009b6c0 auth_method 0 2024-09-16T05:16:29.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:29.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:29.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:29.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70ffff640 1 --2- 172.21.15.114:0/2092644070 >> v2:172.21.15.149:3300/0 conn(0x7fd7100a1810 0x7fd71013cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70ffff640 10 monclient(hunting): get_auth_request con 0x7fd7100a1810 auth_method 0 2024-09-16T05:16:29.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 10 monclient(hunting): handle_auth_done global_id 74283 payload 274 2024-09-16T05:16:29.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 1 -- 172.21.15.114:0/2092644070 >> v2:172.21.15.149:3300/0 conn(0x7fd7100a1810 msgr2=0x7fd71013cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:29.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 1 --2- 172.21.15.114:0/2092644070 >> v2:172.21.15.149:3300/0 conn(0x7fd7100a1810 0x7fd71013cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 1 -- 172.21.15.114:0/2092644070 >> v1:172.21.15.114:6789/0 conn(0x7fd7100a1400 legacy=0x7fd710139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:16:29.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 1 monclient: found mon.noname-a 2024-09-16T05:16:29.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:29.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:29.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.175+0000 7fd70f7fe640 1 -- 172.21.15.114:0/2092644070 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71009f430 con 0x7fd71009b6c0 2024-09-16T05:16:29.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70f7fe640 1 --2- 172.21.15.114:0/2092644070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd71009b6c0 0x7fd7100a0ec0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd71009d220 tx=0x7fd70801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23bf5bd5c14f742e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:29.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70e7fc640 1 -- 172.21.15.114:0/2092644070 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7080b2070 con 0x7fd71009b6c0 2024-09-16T05:16:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70e7fc640 10 monclient: dump: 2024-09-16T05:16:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.182 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:16:29.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:16:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70e7fc640 1 -- 172.21.15.114:0/2092644070 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7080a57a0 con 0x7fd71009b6c0 2024-09-16T05:16:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 5 monclient: authenticate success, global_id 74283 2024-09-16T05:16:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:16:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 4 set_mon_vals no callback set 2024-09-16T05:16:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70e7fc640 1 -- 172.21.15.114:0/2092644070 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7080a5b70 con 0x7fd71009b6c0 2024-09-16T05:16:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70e7fc640 10 monclient: dump: 2024-09-16T05:16:29.183 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.184 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.184 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.184 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.184 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.184 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.184 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.184 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.184 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.185 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 10 monclient: shutdown 2024-09-16T05:16:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 -- 172.21.15.114:0/2092644070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd71009b6c0 msgr2=0x7fd7100a0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 --2- 172.21.15.114:0/2092644070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd71009b6c0 0x7fd7100a0ec0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd71009d220 tx=0x7fd70801c7f0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 -- 172.21.15.114:0/2092644070 shutdown_connections 2024-09-16T05:16:29.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 --2- 172.21.15.114:0/2092644070 >> v2:172.21.15.149:3300/0 conn(0x7fd7100a1810 0x7fd71013cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 --2- 172.21.15.114:0/2092644070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd71009b6c0 0x7fd7100a0ec0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 -- 172.21.15.114:0/2092644070 >> 172.21.15.114:0/2092644070 conn(0x7fd7100a1fb0 msgr2=0x7fd7100a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 -- 172.21.15.114:0/2092644070 shutdown_connections 2024-09-16T05:16:29.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 -- 172.21.15.114:0/2092644070 wait complete. 2024-09-16T05:16:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 10 monclient: build_initial_monmap 2024-09-16T05:16:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 10 monclient: monmap: 2024-09-16T05:16:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:29.196 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 librados: starting msgr at 2024-09-16T05:16:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 librados: starting objecter 2024-09-16T05:16:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 Processor -- start 2024-09-16T05:16:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 -- start start 2024-09-16T05:16:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 librados: setting wanted keys 2024-09-16T05:16:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 librados: calling monclient init 2024-09-16T05:16:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 10 monclient: init 2024-09-16T05:16:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:16:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd71009b6c0 0x7fd7100986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 10 monclient(hunting): picked mon.noname-c con 0x7fd71009b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7100a1400 0x7fd710091d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 10 monclient(hunting): picked mon.noname-a con 0x7fd7100a1400 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7100a1810 0x7fd710092240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 10 monclient(hunting): picked mon.noname-b con 0x7fd7100a1810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd71009cad0 con 0x7fd7100a1400 2024-09-16T05:16:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fd71009dd40 con 0x7fd7100a1810 2024-09-16T05:16:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd71013fad0 con 0x7fd71009b6c0 2024-09-16T05:16:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd716057640 10 monclient(hunting): authenticate will time out at 1473.811519s 2024-09-16T05:16:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7100a1400 0x7fd710091d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70f7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd71009b6c0 0x7fd7100986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7100a1400 0x7fd710091d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32864/0 (socket says 172.21.15.114:32864) 2024-09-16T05:16:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 1 -- 172.21.15.114:0/468674862 learned_addr learned my addr 172.21.15.114:0/468674862 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 10 monclient(hunting): get_auth_request con 0x7fd7100a1400 auth_method 0 2024-09-16T05:16:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70f7fe640 10 monclient(hunting): get_auth_request con 0x7fd71009b6c0 auth_method 0 2024-09-16T05:16:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70ffff640 1 --2- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7100a1810 0x7fd710092240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70ffff640 10 monclient(hunting): get_auth_request con 0x7fd7100a1810 auth_method 0 2024-09-16T05:16:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 10 monclient(hunting): handle_auth_done global_id 74289 payload 824 2024-09-16T05:16:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 1 -- 172.21.15.114:0/468674862 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd71009b6c0 msgr2=0x7fd7100986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 1 --2- 172.21.15.114:0/468674862 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd71009b6c0 0x7fd7100986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 1 -- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7100a1810 msgr2=0x7fd710092240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 1 --2- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7100a1810 0x7fd710092240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 10 monclient: _finish_hunting 0 2024-09-16T05:16:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 1 monclient: found mon.noname-a 2024-09-16T05:16:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.179+0000 7fd70effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70effd640 1 -- 172.21.15.114:0/468674862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd710095960 con 0x7fd7100a1400 2024-09-16T05:16:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70effd640 1 --2- 172.21.15.114:0/468674862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7100a1400 0x7fd710091d00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd6f800c400 tx=0x7fd6f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 client.?.objecter ms_handle_connect 0x7fd7100a1400 2024-09-16T05:16:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 1 -- 172.21.15.114:0/468674862 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f8017070 con 0x7fd7100a1400 2024-09-16T05:16:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 monclient: dump: 2024-09-16T05:16:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.202 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 monclient: _finish_auth 0 2024-09-16T05:16:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 monclient: _check_auth_tickets 2024-09-16T05:16:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 1 -- 172.21.15.114:0/468674862 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6f8007500 con 0x7fd7100a1400 2024-09-16T05:16:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd716057640 5 monclient: authenticate success, global_id 74289 2024-09-16T05:16:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd716057640 10 monclient: _renew_subs 2024-09-16T05:16:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd716057640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd716057640 1 -- 172.21.15.114:0/468674862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71006e2a0 con 0x7fd7100a1400 2024-09-16T05:16:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 1 -- 172.21.15.114:0/468674862 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f80059f0 con 0x7fd7100a1400 2024-09-16T05:16:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 monclient: dump: 2024-09-16T05:16:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.204 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70dffb640 4 set_mon_vals no callback set 2024-09-16T05:16:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd716057640 10 client.74289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd716057640 10 monclient: _renew_subs 2024-09-16T05:16:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd716057640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd716057640 1 -- 172.21.15.114:0/468674862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd710072e20 con 0x7fd7100a1400 2024-09-16T05:16:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd716057640 1 librados: init done 2024-09-16T05:16:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd716057640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd716057640 1 --2- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd71013cc20 0x7fd71002d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd716057640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd716057640 1 -- 172.21.15.114:0/468674862 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fd710069390 con 0x7fd71013cc20 2024-09-16T05:16:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 1 -- 172.21.15.114:0/468674862 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd6f801a070 con 0x7fd7100a1400 2024-09-16T05:16:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70f7fe640 1 --2- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd71013cc20 0x7fd71002d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70f7fe640 10 monclient: get_auth_request con 0x7fd71013cc20 auth_method 0 2024-09-16T05:16:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 1 --2- 172.21.15.114:0/468674862 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd6f0037810 0x7fd6f0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 1 -- 172.21.15.114:0/468674862 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(357..357 src has 1..357) ==== 5406+0+0 (secure 0 0 0) 0x7fd6f8048170 con 0x7fd7100a1400 2024-09-16T05:16:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 client.74289.objecter ms_dispatch 0x7fd710000ba0 osd_map(357..357 src has 1..357) 2024-09-16T05:16:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 3 client.74289.objecter handle_osd_map got epochs [357,357] > 0 2024-09-16T05:16:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 3 client.74289.objecter handle_osd_map decoding full epoch 357 2024-09-16T05:16:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:29.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 20 client.74289.objecter dump_active .. 0 homeless 2024-09-16T05:16:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70ffff640 1 --2- 172.21.15.114:0/468674862 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd6f0037810 0x7fd6f0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70ffff640 10 monclient: get_auth_request con 0x7fd6f0037810 auth_method 0 2024-09-16T05:16:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70f7fe640 10 monclient(hunting): handle_auth_done global_id 64450 payload 824 2024-09-16T05:16:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70f7fe640 1 -- 172.21.15.114:0/468674862 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7100a0ec0 con 0x7fd71013cc20 2024-09-16T05:16:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 client.74289.objecter ms_handle_connect 0x7fd71013cc20 2024-09-16T05:16:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 client.74289.objecter resend_mon_ops 2024-09-16T05:16:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70f7fe640 1 --2- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd71013cc20 0x7fd71002d7d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd71009c630 tx=0x7fd708019590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 client.74289.objecter ms_handle_connect 0x7fd6f0037810 2024-09-16T05:16:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd70ffff640 1 --2- 172.21.15.114:0/468674862 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd6f0037810 0x7fd6f0039cd0 secure :-1 s=READY pgs=3521 cs=0 l=1 rev1=1 crypto rx=0x7fd70000b950 tx=0x7fd700008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 1 -- 172.21.15.114:0/468674862 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7080b2040 con 0x7fd71013cc20 2024-09-16T05:16:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:16:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 monclient: dump: 2024-09-16T05:16:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.207 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 1 -- 172.21.15.114:0/468674862 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd7080b3040 con 0x7fd71013cc20 2024-09-16T05:16:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 1 -- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd71013cc20 msgr2=0x7fd71002d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 1 --2- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd71013cc20 0x7fd71002d7d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd71009c630 tx=0x7fd708019590 comp rx=0 tx=0).stop 2024-09-16T05:16:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 1 -- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd71013cc20 msgr2=0x7fd71002d7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.183+0000 7fd707fff640 1 --2- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd71013cc20 0x7fd71002d7d0 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd71009c630 tx=0x7fd708019590 comp rx=0 tx=0).stop 2024-09-16T05:16:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd716057640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:16:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd716057640 1 --2- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd710092780 0x7fd710062ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd716057640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd716057640 1 -- 172.21.15.114:0/468674862 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fd710092b90 con 0x7fd710092780 2024-09-16T05:16:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd70f7fe640 1 --2- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd710092780 0x7fd710062ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd70f7fe640 10 monclient: get_auth_request con 0x7fd710092780 auth_method 0 2024-09-16T05:16:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd70f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd70f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd70f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd70f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd70f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd70f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd70f7fe640 10 monclient(hunting): handle_auth_done global_id 64453 payload 824 2024-09-16T05:16:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd70f7fe640 1 -- 172.21.15.114:0/468674862 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd71008e710 con 0x7fd710092780 2024-09-16T05:16:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd70f7fe640 1 --2- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd710092780 0x7fd710062ea0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd70801a420 tx=0x7fd7080a7000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd707fff640 10 client.74289.objecter ms_handle_connect 0x7fd710092780 2024-09-16T05:16:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd707fff640 10 client.74289.objecter resend_mon_ops 2024-09-16T05:16:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd707fff640 1 -- 172.21.15.114:0/468674862 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7080c39f0 con 0x7fd710092780 2024-09-16T05:16:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd707fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd707fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:16:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd707fff640 10 monclient: dump: 2024-09-16T05:16:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.210 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd707fff640 1 -- 172.21.15.114:0/468674862 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fd7080c7070 con 0x7fd710092780 2024-09-16T05:16:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd707fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:16:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd707fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd707fff640 1 -- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd710092780 msgr2=0x7fd710062ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd707fff640 1 --2- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd710092780 0x7fd710062ea0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd70801a420 tx=0x7fd7080a7000 comp rx=0 tx=0).stop 2024-09-16T05:16:29.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd707fff640 1 -- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd710092780 msgr2=0x7fd710062ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:29.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd707fff640 1 --2- 172.21.15.114:0/468674862 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd710092780 0x7fd710062ea0 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd70801a420 tx=0x7fd7080a7000 comp rx=0 tx=0).stop 2024-09-16T05:16:29.211 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:16:29.211 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:16:29.211 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:16:29.211 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:16:29.211 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 40, 2024-09-16T05:16:29.211 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 273324, 2024-09-16T05:16:29.212 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:16:29.212 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:16:29.212 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:16:29.212 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:16:29.212 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:29.212 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:16:29.212 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:29.212 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:16:29.212 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:16:29.212 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:29.212 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:29.212 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:29.212 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:29.213 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:29.213 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:29.213 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:29.213 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:29.213 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:29.213 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:29.213 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:29.213 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:29.213 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:16:29.213 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:16:29.213 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:29.213 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:29.213 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:29.214 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:29.214 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:29.214 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:29.214 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:29.214 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:29.214 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:29.214 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:29.214 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:29.214 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:29.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.214 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:16:29.214 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:16:29.214 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:16:29.214 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:16:29.215 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:16:29.215 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:16:29.215 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:29.215 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:29.215 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:16:29.215 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:16:29.215 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:16:29.215 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:16:29.215 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:16:29.215 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:16:29.215 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:16:29.215 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:29.215 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:16:29.216 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:29.216 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:29.216 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:29.216 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:29.216 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:29.216 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:29.216 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:29.216 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:29.216 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:29.216 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:29.216 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:29.216 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:29.216 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:16:29.217 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.217 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:16:29.217 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.217 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:16:29.217 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:16:29.217 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:29.217 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:29.217 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.217 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:29.217 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:16:29.217 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:29.217 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.217 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.218 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:29.218 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:16:29.218 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:29.218 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:29.218 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:29.218 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.218 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:16:29.218 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:16:29.218 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:29.218 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:29.218 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:29.218 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.218 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.219 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:16:29.219 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:16:29.219 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:29.219 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:29.219 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.219 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:29.219 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:16:29.219 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:29.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.219 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.219 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:29.219 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:16:29.219 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:29.219 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:29.220 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:29.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.220 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:16:29.220 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:16:29.220 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:29.220 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:29.220 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:29.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.220 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.220 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:16:29.220 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:16:29.220 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:29.220 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:29.220 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.221 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:29.221 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:16:29.221 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:29.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.221 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:29.221 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:16:29.221 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:29.221 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:29.221 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:29.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.221 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:16:29.221 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:16:29.221 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:29.222 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:29.222 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:29.222 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:29.222 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:29.222 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.222 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:16:29.222 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:16:29.222 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.222 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:29.222 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:29.222 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:16:29.222 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:29.222 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:29.223 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:16:29.223 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.223 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:29.223 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:29.223 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:16:29.223 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:29.223 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:29.223 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:16:29.223 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.223 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:29.223 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:29.223 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:16:29.223 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:29.223 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:29.224 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:16:29.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:29.224 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:29.224 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:29.224 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:16:29.224 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:29.224 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:29.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:29.224 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:16:29.224 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:16:29.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd716057640 10 librados: watch_flush enter 2024-09-16T05:16:29.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.191+0000 7fd716057640 10 librados: watch_flush exit 2024-09-16T05:16:29.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.195+0000 7fd716057640 20 client.74289.objecter shutdown clearing up homeless session... 2024-09-16T05:16:29.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.195+0000 7fd716057640 10 client.74289.objecter successfully canceled tick 2024-09-16T05:16:29.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.195+0000 7fd716057640 1 -- 172.21.15.114:0/468674862 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd6f0037810 msgr2=0x7fd6f0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.195+0000 7fd716057640 1 --2- 172.21.15.114:0/468674862 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd6f0037810 0x7fd6f0039cd0 secure :-1 s=READY pgs=3521 cs=0 l=1 rev1=1 crypto rx=0x7fd70000b950 tx=0x7fd700008040 comp rx=0 tx=0).stop 2024-09-16T05:16:29.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.195+0000 7fd716057640 10 monclient: shutdown 2024-09-16T05:16:29.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.195+0000 7fd716057640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:29.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.195+0000 7fd716057640 1 -- 172.21.15.114:0/468674862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7100a1400 msgr2=0x7fd710091d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.195+0000 7fd716057640 1 --2- 172.21.15.114:0/468674862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7100a1400 0x7fd710091d00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd6f800c400 tx=0x7fd6f800c8c0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.195+0000 7fd70ffff640 1 -- 172.21.15.114:0/468674862 reap_dead start 2024-09-16T05:16:29.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.195+0000 7fd70ffff640 1 -- 172.21.15.114:0/468674862 reap_dead start 2024-09-16T05:16:29.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.195+0000 7fd716057640 1 -- 172.21.15.114:0/468674862 shutdown_connections 2024-09-16T05:16:29.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.195+0000 7fd716057640 1 -- 172.21.15.114:0/468674862 >> 172.21.15.114:0/468674862 conn(0x7fd7100a1fb0 msgr2=0x7fd710071b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:29.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.195+0000 7fd716057640 1 -- 172.21.15.114:0/468674862 shutdown_connections 2024-09-16T05:16:29.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.195+0000 7fd716057640 1 -- 172.21.15.114:0/468674862 wait complete. 2024-09-16T05:16:29.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.195+0000 7fd716057640 1 librados: shutdown 2024-09-16T05:16:29.227 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:16:29.227 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:16:29.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:16:29.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.339+0000 7f89de8ca640 10 monclient: get_monmap_and_config 2024-09-16T05:16:29.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.339+0000 7f89de8ca640 10 monclient: build_initial_monmap 2024-09-16T05:16:29.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.339+0000 7f89de8ca640 10 monclient: monmap: 2024-09-16T05:16:29.340 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:29.340 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.340 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:29.343442+0000 2024-09-16T05:16:29.340 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:29.343442+0000 2024-09-16T05:16:29.340 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:29.340 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.340 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:29.340 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:29.340 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:29.341 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:29.341 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:29.341 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.339+0000 7f89de8ca640 1 Processor -- start 2024-09-16T05:16:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.339+0000 7f89de8ca640 1 -- start start 2024-09-16T05:16:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.339+0000 7f89de8ca640 10 monclient: init 2024-09-16T05:16:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-16T05:16:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89d80a0ec0 addr v1:172.21.15.114:6789/0 2024-09-16T05:16:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f89d809e1d0 addr v1:172.21.15.149:6789/0 2024-09-16T05:16:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89d809e5e0 0x7f89d813cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 10 monclient(hunting): picked mon.noname-b con 0x7f89d809e5e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89d809a760 con 0x7f89d80a0ec0 2024-09-16T05:16:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89d80a1870 con 0x7f89d809e1d0 2024-09-16T05:16:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f89d80a12a0 con 0x7f89d809e5e0 2024-09-16T05:16:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:29.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 10 monclient(hunting): authenticate will time out at 1473.972550s 2024-09-16T05:16:29.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d7fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f89d80a0ec0 0x7f89d809dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33172/0 (socket says 172.21.15.114:33172) 2024-09-16T05:16:29.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d7fff640 1 -- 172.21.15.114:0/214732447 learned_addr learned my addr 172.21.15.114:0/214732447 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:16:29.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 1 --2- 172.21.15.114:0/214732447 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89d809e5e0 0x7f89d813cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 10 monclient(hunting): get_auth_request con 0x7f89d809e5e0 auth_method 0 2024-09-16T05:16:29.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:29.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:29.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:29.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 10 monclient(hunting): handle_auth_done global_id 84128 payload 274 2024-09-16T05:16:29.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d77fe640 1 -- 172.21.15.114:0/214732447 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 349104733 0 0) 0x7f89d809a760 con 0x7f89d80a0ec0 2024-09-16T05:16:29.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 1 -- 172.21.15.114:0/214732447 >> v1:172.21.15.149:6789/0 conn(0x7f89d809e1d0 legacy=0x7f89d8139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:29.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 1 -- 172.21.15.114:0/214732447 >> v1:172.21.15.114:6789/0 conn(0x7f89d80a0ec0 legacy=0x7f89d809dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 10 monclient: _finish_hunting 0 2024-09-16T05:16:29.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 1 monclient: found mon.noname-b 2024-09-16T05:16:29.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:29.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:16:29.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 1 -- 172.21.15.114:0/214732447 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d8099790 con 0x7f89d809e5e0 2024-09-16T05:16:29.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d77fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:16:29.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89dce67640 1 --2- 172.21.15.114:0/214732447 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89d809e5e0 0x7f89d813cd70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f89d0009d90 tx=0x7f89d001c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3c7cf1036bb1877 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:29.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d77fe640 1 -- 172.21.15.114:0/214732447 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d00b2070 con 0x7f89d809e5e0 2024-09-16T05:16:29.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:16:29.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d77fe640 10 monclient: dump: 2024-09-16T05:16:29.351 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.351 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.351 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.351 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.351 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.352 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d77fe640 10 monclient: _finish_auth 0 2024-09-16T05:16:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d77fe640 10 monclient: _check_auth_tickets 2024-09-16T05:16:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d77fe640 1 -- 172.21.15.114:0/214732447 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d00a5850 con 0x7f89d809e5e0 2024-09-16T05:16:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 5 monclient: authenticate success, global_id 84128 2024-09-16T05:16:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:16:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d77fe640 1 -- 172.21.15.114:0/214732447 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d00a5c20 con 0x7f89d809e5e0 2024-09-16T05:16:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:16:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89d77fe640 10 monclient: dump: 2024-09-16T05:16:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.353 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 4 set_mon_vals no callback set 2024-09-16T05:16:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 10 monclient: shutdown 2024-09-16T05:16:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 1 -- 172.21.15.114:0/214732447 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89d809e5e0 msgr2=0x7f89d813cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 1 --2- 172.21.15.114:0/214732447 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89d809e5e0 0x7f89d813cd70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f89d0009d90 tx=0x7f89d001c880 comp rx=0 tx=0).stop 2024-09-16T05:16:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 1 -- 172.21.15.114:0/214732447 shutdown_connections 2024-09-16T05:16:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 1 --2- 172.21.15.114:0/214732447 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89d809e5e0 0x7f89d813cd70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 1 -- 172.21.15.114:0/214732447 >> 172.21.15.114:0/214732447 conn(0x7f89d80a1fb0 msgr2=0x7f89d80a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 1 -- 172.21.15.114:0/214732447 shutdown_connections 2024-09-16T05:16:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.343+0000 7f89de8ca640 1 -- 172.21.15.114:0/214732447 wait complete. 2024-09-16T05:16:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient: build_initial_monmap 2024-09-16T05:16:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient: monmap: 2024-09-16T05:16:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:29.355 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 librados: starting msgr at 2024-09-16T05:16:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 librados: starting objecter 2024-09-16T05:16:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 Processor -- start 2024-09-16T05:16:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 -- start start 2024-09-16T05:16:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 librados: setting wanted keys 2024-09-16T05:16:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 librados: calling monclient init 2024-09-16T05:16:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient: init 2024-09-16T05:16:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:16:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f89d809e1d0 0x7f89d8137b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient(hunting): picked mon.noname-b con 0x7f89d809e1d0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89d809e5e0 0x7f89d81380b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient(hunting): picked mon.noname-a con 0x7f89d809e5e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89d80a0ec0 0x7f89d81385f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient(hunting): picked mon.noname-c con 0x7f89d80a0ec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f89d813fc50 con 0x7f89d809e5e0 2024-09-16T05:16:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f89d813fad0 con 0x7f89d809e1d0 2024-09-16T05:16:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f89d813fdd0 con 0x7f89d80a0ec0 2024-09-16T05:16:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient(hunting): authenticate will time out at 1473.977324s 2024-09-16T05:16:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89d80a0ec0 0x7f89d81385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89cffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89d809e5e0 0x7f89d81380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89d80a0ec0 0x7f89d81385f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:50748/0 (socket says 172.21.15.114:50748) 2024-09-16T05:16:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 1 -- 172.21.15.114:0/49684796 learned_addr learned my addr 172.21.15.114:0/49684796 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89cffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89d809e5e0 0x7f89d81380b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32876/0 (socket says 172.21.15.114:32876) 2024-09-16T05:16:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 10 monclient(hunting): get_auth_request con 0x7f89d80a0ec0 auth_method 0 2024-09-16T05:16:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89cffff640 10 monclient(hunting): get_auth_request con 0x7f89d809e5e0 auth_method 0 2024-09-16T05:16:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d7fff640 1 --2- 172.21.15.114:0/49684796 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f89d809e1d0 0x7f89d8137b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 10 monclient(hunting): handle_auth_done global_id 84131 payload 824 2024-09-16T05:16:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 1 -- 172.21.15.114:0/49684796 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f89d809e1d0 msgr2=0x7f89d8137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 1 --2- 172.21.15.114:0/49684796 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f89d809e1d0 0x7f89d8137b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 1 -- 172.21.15.114:0/49684796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89d809e5e0 msgr2=0x7f89d81380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 1 --2- 172.21.15.114:0/49684796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89d809e5e0 0x7f89d81380b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 10 monclient: _finish_hunting 0 2024-09-16T05:16:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 1 monclient: found mon.noname-c 2024-09-16T05:16:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:16:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 1 -- 172.21.15.114:0/49684796 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d813b5a0 con 0x7f89d80a0ec0 2024-09-16T05:16:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d7fff640 10 monclient: get_auth_request con 0x7f89d809e1d0 auth_method 0 2024-09-16T05:16:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89cffff640 1 --2- 172.21.15.114:0/49684796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89d809e5e0 0x7f89d81380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:16:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d7fff640 1 --2- 172.21.15.114:0/49684796 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f89d809e1d0 0x7f89d8137b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:16:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89dce67640 1 --2- 172.21.15.114:0/49684796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89d80a0ec0 0x7f89d81385f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f89d0009d90 tx=0x7f89d0019620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 10 client.?.objecter ms_handle_connect 0x7f89d80a0ec0 2024-09-16T05:16:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 1 -- 172.21.15.114:0/49684796 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d00bd020 con 0x7f89d80a0ec0 2024-09-16T05:16:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:16:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 10 monclient: dump: 2024-09-16T05:16:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.360 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 10 monclient: _finish_auth 0 2024-09-16T05:16:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 10 monclient: _check_auth_tickets 2024-09-16T05:16:29.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 5 monclient: authenticate success, global_id 84131 2024-09-16T05:16:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 1 -- 172.21.15.114:0/49684796 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d00ad6c0 con 0x7f89d80a0ec0 2024-09-16T05:16:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient: _renew_subs 2024-09-16T05:16:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 1 -- 172.21.15.114:0/49684796 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d00b2040 con 0x7f89d80a0ec0 2024-09-16T05:16:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:16:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 -- 172.21.15.114:0/49684796 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d8139540 con 0x7f89d80a0ec0 2024-09-16T05:16:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:29.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d6ffd640 4 set_mon_vals no callback set 2024-09-16T05:16:29.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:16:29.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 10 monclient: dump: 2024-09-16T05:16:29.365 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:29.365 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:29.365 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.365 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:29.365 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:29.365 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:29.366 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:29.366 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:29.366 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:29.366 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:29.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 client.84131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:29.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient: _renew_subs 2024-09-16T05:16:29.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:16:29.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 -- 172.21.15.114:0/49684796 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d80a49e0 con 0x7f89d80a0ec0 2024-09-16T05:16:29.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 librados: init done 2024-09-16T05:16:29.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:29.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:29.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:16:29.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89de8ca640 1 -- 172.21.15.114:0/49684796 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89d8139c00 con 0x7f89d80a0ec0 2024-09-16T05:16:29.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 1 -- 172.21.15.114:0/49684796 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f89d00c0020 con 0x7f89d80a0ec0 2024-09-16T05:16:29.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 1 --2- 172.21.15.114:0/49684796 >> v2:172.21.15.149:6812/1185531135 conn(0x7f89b8037810 0x7f89b8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:29.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 1 -- 172.21.15.114:0/49684796 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(357..357 src has 1..357) ==== 5406+0+0 (secure 0 0 0) 0x7f89d00e3fe0 con 0x7f89d80a0ec0 2024-09-16T05:16:29.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 10 client.84131.objecter ms_dispatch 0x7f89d8000ba0 osd_map(357..357 src has 1..357) 2024-09-16T05:16:29.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 3 client.84131.objecter handle_osd_map got epochs [357,357] > 0 2024-09-16T05:16:29.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 3 client.84131.objecter handle_osd_map decoding full epoch 357 2024-09-16T05:16:29.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d57fa640 20 client.84131.objecter dump_active .. 0 homeless 2024-09-16T05:16:29.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d7fff640 1 --2- 172.21.15.114:0/49684796 >> v2:172.21.15.149:6812/1185531135 conn(0x7f89b8037810 0x7f89b8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:29.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.347+0000 7f89d7fff640 10 monclient: get_auth_request con 0x7f89b8037810 auth_method 0 2024-09-16T05:16:29.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.351+0000 7f89d57fa640 10 client.84131.objecter ms_handle_connect 0x7f89b8037810 2024-09-16T05:16:29.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.351+0000 7f89d7fff640 1 --2- 172.21.15.114:0/49684796 >> v2:172.21.15.149:6812/1185531135 conn(0x7f89b8037810 0x7f89b8039cd0 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 crypto rx=0x7f89c8000f70 tx=0x7f89c8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:29.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.351+0000 7f89d57fa640 1 -- 172.21.15.114:0/49684796 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f89d00c7030 con 0x7f89d80a0ec0 2024-09-16T05:16:29.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.351+0000 7f89d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:29.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.351+0000 7f89d57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:29.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.559+0000 7f89de8ca640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:16:29.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.559+0000 7f89de8ca640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:16:29.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.559+0000 7f89de8ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:16:29.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.559+0000 7f89de8ca640 1 -- 172.21.15.114:0/49684796 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f89d80686f0 con 0x7f89d80a0ec0 2024-09-16T05:16:29.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.559+0000 7f89d57fa640 1 -- 172.21.15.114:0/49684796 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f89d0016500 con 0x7f89d80a0ec0 2024-09-16T05:16:29.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.559+0000 7f89d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:16:29.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.559+0000 7f89d57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:29.560 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.559+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.561 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.559+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=1,health=11,kv=12,logm=61} crc {auth=1103988062,config=1747058506,health=1019831743,kv=2757161437,logm=3213550310}) 2024-09-16T05:16:29.561 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.559+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=11,kv=12,logm=60} crc {auth=1103988062,config=2586702516,health=3885201563,kv=2757161437,logm=265409200}) 2024-09-16T05:16:29.561 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.559+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.561 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.559+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=1,health=11,kv=12,logm=61} crc {auth=1103988062,config=1747058506,health=1019831743,kv=2757161437,logm=3213550310}) 2024-09-16T05:16:29.561 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.559+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=2,health=12,kv=11,logm=59} crc {auth=2252617029,config=2586702516,health=3623406937,kv=2757161437,logm=133288893}) 2024-09-16T05:16:29.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 10 librados: watch_flush enter 2024-09-16T05:16:29.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 10 librados: watch_flush exit 2024-09-16T05:16:29.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 20 client.84131.objecter shutdown clearing up homeless session... 2024-09-16T05:16:29.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 10 client.84131.objecter successfully canceled tick 2024-09-16T05:16:29.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 1 -- 172.21.15.114:0/49684796 >> v2:172.21.15.149:6812/1185531135 conn(0x7f89b8037810 msgr2=0x7f89b8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 1 --2- 172.21.15.114:0/49684796 >> v2:172.21.15.149:6812/1185531135 conn(0x7f89b8037810 0x7f89b8039cd0 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 crypto rx=0x7f89c8000f70 tx=0x7f89c8009040 comp rx=0 tx=0).stop 2024-09-16T05:16:29.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 10 monclient: shutdown 2024-09-16T05:16:29.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:29.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 1 -- 172.21.15.114:0/49684796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89d80a0ec0 msgr2=0x7f89d81385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:29.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 1 --2- 172.21.15.114:0/49684796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89d80a0ec0 0x7f89d81385f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f89d0009d90 tx=0x7f89d0019620 comp rx=0 tx=0).stop 2024-09-16T05:16:29.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 1 -- 172.21.15.114:0/49684796 shutdown_connections 2024-09-16T05:16:29.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 1 --2- 172.21.15.114:0/49684796 >> v2:172.21.15.149:6812/1185531135 conn(0x7f89b8037810 0x7f89b8039cd0 unknown :-1 s=CLOSED pgs=3522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 1 --2- 172.21.15.114:0/49684796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89d809e5e0 0x7f89d81380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 1 --2- 172.21.15.114:0/49684796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f89d80a0ec0 0x7f89d81385f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 1 --2- 172.21.15.114:0/49684796 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f89d809e1d0 0x7f89d8137b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:29.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 1 -- 172.21.15.114:0/49684796 >> 172.21.15.114:0/49684796 conn(0x7f89d80a1fb0 msgr2=0x7f89d80a33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:29.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 1 -- 172.21.15.114:0/49684796 shutdown_connections 2024-09-16T05:16:29.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 1 -- 172.21.15.114:0/49684796 wait complete. 2024-09-16T05:16:29.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f89de8ca640 1 librados: shutdown 2024-09-16T05:16:29.565 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.565 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665895182}) 2024-09-16T05:16:29.568 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=968435565}) 2024-09-16T05:16:29.568 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.569 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665895182}) 2024-09-16T05:16:29.569 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.563+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=213820733}) 2024-09-16T05:16:29.571 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.567+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.571 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.567+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3535521654}) 2024-09-16T05:16:29.571 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.567+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=576650765}) 2024-09-16T05:16:29.571 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.567+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.571 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.567+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3535521654}) 2024-09-16T05:16:29.571 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.567+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1560762218}) 2024-09-16T05:16:29.571 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.567+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.571 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.567+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272387389}) 2024-09-16T05:16:29.571 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.567+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3160311602}) 2024-09-16T05:16:29.571 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.571+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.571 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.571+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272387389}) 2024-09-16T05:16:29.571 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.571+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2911651529}) 2024-09-16T05:16:29.572 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.571+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.572 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.571+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771534695}) 2024-09-16T05:16:29.572 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.571+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3750425946}) 2024-09-16T05:16:29.572 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.571+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.572 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.571+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771534695}) 2024-09-16T05:16:29.572 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.571+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2126608638}) 2024-09-16T05:16:29.574 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.575+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.574 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.575+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1985416233}) 2024-09-16T05:16:29.574 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.575+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1419494601}) 2024-09-16T05:16:29.574 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.575+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.574 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.575+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1985416233}) 2024-09-16T05:16:29.574 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.575+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3032039746}) 2024-09-16T05:16:29.576 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.575+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.576 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.575+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3951210582}) 2024-09-16T05:16:29.576 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.575+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2906612247}) 2024-09-16T05:16:29.576 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.575+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.576 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.575+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3951210582}) 2024-09-16T05:16:29.576 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.575+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2793091504}) 2024-09-16T05:16:29.577 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:16:29.578 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.579+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.578 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.579+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260046161}) 2024-09-16T05:16:29.578 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.579+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3708290505}) 2024-09-16T05:16:29.579 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.579+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.579 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.579+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260046161}) 2024-09-16T05:16:29.579 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.579+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2639362887}) 2024-09-16T05:16:29.581 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.583+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.581 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.583+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884337592}) 2024-09-16T05:16:29.581 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.583+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1887053808}) 2024-09-16T05:16:29.581 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.583+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.581 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.583+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884337592}) 2024-09-16T05:16:29.581 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.583+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4214357346}) 2024-09-16T05:16:29.583 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.583+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.583 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.583+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3035796716}) 2024-09-16T05:16:29.583 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.583+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2504957305}) 2024-09-16T05:16:29.583 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.583+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.583 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.583+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3035796716}) 2024-09-16T05:16:29.583 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.583+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2822408643}) 2024-09-16T05:16:29.585 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.587+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.585 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.587+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701639665}) 2024-09-16T05:16:29.586 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.587+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1945752944}) 2024-09-16T05:16:29.586 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.587+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.586 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.587+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701639665}) 2024-09-16T05:16:29.586 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.587+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2189162485}) 2024-09-16T05:16:29.588 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.587+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.588 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.587+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613617610}) 2024-09-16T05:16:29.588 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.587+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3346933151}) 2024-09-16T05:16:29.588 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.587+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.588 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.587+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613617610}) 2024-09-16T05:16:29.588 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.587+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3696178878}) 2024-09-16T05:16:29.590 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.591+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.590 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.591+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1210601171}) 2024-09-16T05:16:29.590 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.591+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=466775877}) 2024-09-16T05:16:29.590 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.591+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.591 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.591+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1210601171}) 2024-09-16T05:16:29.591 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.591+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681944140}) 2024-09-16T05:16:29.593 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.595+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.593 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.595+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647496668}) 2024-09-16T05:16:29.593 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.595+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3125316681}) 2024-09-16T05:16:29.593 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.595+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.593 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.595+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647496668}) 2024-09-16T05:16:29.593 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.595+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4112706852}) 2024-09-16T05:16:29.595 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.595+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.595 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.595+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=7079412}) 2024-09-16T05:16:29.595 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.595+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2942888853}) 2024-09-16T05:16:29.595 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.595+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.595 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.595+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=7079412}) 2024-09-16T05:16:29.596 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.595+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2216543532}) 2024-09-16T05:16:29.597 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.599+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.598 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.599+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=787211525}) 2024-09-16T05:16:29.598 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.599+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2883442800}) 2024-09-16T05:16:29.598 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.599+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.598 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.599+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=787211525}) 2024-09-16T05:16:29.598 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.599+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3406094474}) 2024-09-16T05:16:29.600 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.599+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.600 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.599+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477858350}) 2024-09-16T05:16:29.600 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.599+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1525841793}) 2024-09-16T05:16:29.600 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.599+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.600 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.599+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477858350}) 2024-09-16T05:16:29.600 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.599+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=635166048}) 2024-09-16T05:16:29.602 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.603+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.602 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.603+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3632779288}) 2024-09-16T05:16:29.602 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.603+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=58633880}) 2024-09-16T05:16:29.602 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.603+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.602 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.603+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3632779288}) 2024-09-16T05:16:29.602 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.603+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2130624092}) 2024-09-16T05:16:29.604 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.603+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.604 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.603+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2794273752}) 2024-09-16T05:16:29.604 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.603+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984959148}) 2024-09-16T05:16:29.604 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.603+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.605 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.603+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2794273752}) 2024-09-16T05:16:29.605 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.603+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1738403849}) 2024-09-16T05:16:29.606 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.607+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.606 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.607+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2463472815}) 2024-09-16T05:16:29.606 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.607+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2603049520}) 2024-09-16T05:16:29.607 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.607+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.607 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.607+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2463472815}) 2024-09-16T05:16:29.607 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.607+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=94879708}) 2024-09-16T05:16:29.609 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.611+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.609 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.611+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767804295}) 2024-09-16T05:16:29.609 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.611+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4208580529}) 2024-09-16T05:16:29.609 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.611+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.609 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.611+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767804295}) 2024-09-16T05:16:29.609 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.611+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3698566729}) 2024-09-16T05:16:29.611 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.611+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.611 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.611+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3769523113}) 2024-09-16T05:16:29.612 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.611+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1847502150}) 2024-09-16T05:16:29.612 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.611+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.612 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.611+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3769523113}) 2024-09-16T05:16:29.613 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.611+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2882830186}) 2024-09-16T05:16:29.614 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.615+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.614 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.615+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3466446667}) 2024-09-16T05:16:29.614 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.615+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3402158608}) 2024-09-16T05:16:29.614 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.615+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.614 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.615+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3466446667}) 2024-09-16T05:16:29.614 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.615+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4082894200}) 2024-09-16T05:16:29.615 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.615+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.615 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.615+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867788744}) 2024-09-16T05:16:29.615 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.615+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4249467675}) 2024-09-16T05:16:29.615 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.615+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.615 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.615+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867788744}) 2024-09-16T05:16:29.615 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.615+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1378023797}) 2024-09-16T05:16:29.617 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.619+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.617 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.619+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2089375580}) 2024-09-16T05:16:29.617 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.619+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3096915294}) 2024-09-16T05:16:29.617 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.619+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.617 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.619+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2089375580}) 2024-09-16T05:16:29.617 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.619+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2634705442}) 2024-09-16T05:16:29.619 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.619+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.619 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.619+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3685127091}) 2024-09-16T05:16:29.619 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.619+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=68749559}) 2024-09-16T05:16:29.619 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.619+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.619 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.619+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3685127091}) 2024-09-16T05:16:29.619 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.619+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4055662345}) 2024-09-16T05:16:29.621 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.623+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.621 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.623+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4183024856}) 2024-09-16T05:16:29.621 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.623+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1368607465}) 2024-09-16T05:16:29.621 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.623+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.621 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.623+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4183024856}) 2024-09-16T05:16:29.621 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.623+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2373947041}) 2024-09-16T05:16:29.623 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.623+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.623 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.623+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=418784384}) 2024-09-16T05:16:29.623 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.623+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1215029691}) 2024-09-16T05:16:29.623 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.623+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.623 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.623+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=418784384}) 2024-09-16T05:16:29.623 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.623+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2660748844}) 2024-09-16T05:16:29.625 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.627+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.625 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.627+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3378937189}) 2024-09-16T05:16:29.625 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.627+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2217062272}) 2024-09-16T05:16:29.625 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.627+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.625 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.627+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3378937189}) 2024-09-16T05:16:29.625 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.627+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516081182}) 2024-09-16T05:16:29.627 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.627+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.627 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.627+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301487572}) 2024-09-16T05:16:29.627 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.627+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1989368852}) 2024-09-16T05:16:29.627 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.627+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.627 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.627+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301487572}) 2024-09-16T05:16:29.627 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.627+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2495508052}) 2024-09-16T05:16:29.629 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.631+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.629 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.631+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546003223}) 2024-09-16T05:16:29.629 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.631+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3844758478}) 2024-09-16T05:16:29.629 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.631+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.629 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.631+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546003223}) 2024-09-16T05:16:29.629 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.631+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2892694097}) 2024-09-16T05:16:29.631 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.631+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.631 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.631+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4051400597}) 2024-09-16T05:16:29.631 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.631+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=9057194}) 2024-09-16T05:16:29.631 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.631+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.631 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.631+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4051400597}) 2024-09-16T05:16:29.631 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.631+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2757889564}) 2024-09-16T05:16:29.633 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.635+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.633 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.635+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=392124377}) 2024-09-16T05:16:29.633 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.635+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=721899702}) 2024-09-16T05:16:29.633 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.635+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.633 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.635+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=392124377}) 2024-09-16T05:16:29.633 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.635+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3431737918}) 2024-09-16T05:16:29.635 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.635+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.635 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.635+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2775041247}) 2024-09-16T05:16:29.635 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.635+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2865183555}) 2024-09-16T05:16:29.635 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.635+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.635 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.635+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2775041247}) 2024-09-16T05:16:29.635 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.635+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1013073874}) 2024-09-16T05:16:29.637 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.639+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.637 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.639+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3381931773}) 2024-09-16T05:16:29.637 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.639+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1573218394}) 2024-09-16T05:16:29.637 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.639+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.637 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.639+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3381931773}) 2024-09-16T05:16:29.637 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.639+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3258156597}) 2024-09-16T05:16:29.639 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.639+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.639 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.639+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1810363551}) 2024-09-16T05:16:29.639 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.639+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3163328392}) 2024-09-16T05:16:29.639 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.639+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.639 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.639+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1810363551}) 2024-09-16T05:16:29.639 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.639+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3240081734}) 2024-09-16T05:16:29.641 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.643+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.641 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.643+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883994080}) 2024-09-16T05:16:29.641 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.643+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=29663965}) 2024-09-16T05:16:29.641 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.643+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.641 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.643+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883994080}) 2024-09-16T05:16:29.641 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.643+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=118946466}) 2024-09-16T05:16:29.642 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.643+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.642 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.643+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3994530385}) 2024-09-16T05:16:29.642 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.643+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3853140869}) 2024-09-16T05:16:29.643 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.643+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.643 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.643+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3994530385}) 2024-09-16T05:16:29.643 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.643+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2710520661}) 2024-09-16T05:16:29.645 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.643+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.645 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.647+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299159958}) 2024-09-16T05:16:29.645 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.647+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=931010606}) 2024-09-16T05:16:29.645 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.647+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.645 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.647+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299159958}) 2024-09-16T05:16:29.645 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.647+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1548525693}) 2024-09-16T05:16:29.647 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.647+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.647 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.647+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=66,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=13} crc {logm=2519156887,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=682779482,mon_config_key=19200820}) 2024-09-16T05:16:29.648 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.647+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=65,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14} crc {logm=3934474781,mdsmap=2485929294,mgr=633530160,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=1979953317,mon_config_key=330048401}) 2024-09-16T05:16:29.648 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.647+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.648 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.647+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=66,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=13} crc {logm=2519156887,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=682779482,mon_config_key=19200820}) 2024-09-16T05:16:29.648 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.647+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14} crc {logm=1533986896,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=1979953317,mon_config_key=2039119123}) 2024-09-16T05:16:29.649 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.651+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.649 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.651+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=56} crc {mon_config_key=1833543056,monmap=3045472550,osd_metadata=487805116,osd_pg_creating=1852403922,osd_snap=1206188278,osdmap=23629683}) 2024-09-16T05:16:29.649 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.651+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=30,osdmap=57} crc {mon_config_key=1833543056,monmap=3045472550,osd_metadata=1484216675,osd_pg_creating=1852403922,osd_snap=3425096391,osdmap=330670683}) 2024-09-16T05:16:29.649 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.651+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.649 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.651+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=56} crc {mon_config_key=1833543056,monmap=3045472550,osd_metadata=487805116,osd_pg_creating=1852403922,osd_snap=1206188278,osdmap=23629683}) 2024-09-16T05:16:29.649 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.651+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=30,osdmap=58} crc {mon_config_key=1833543056,monmap=3045472550,osd_metadata=3740175435,osd_pg_creating=1852403922,osd_snap=1202343519,osdmap=2202373757}) 2024-09-16T05:16:29.651 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.651+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.651 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.651+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1855671755}) 2024-09-16T05:16:29.651 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.651+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1366009013}) 2024-09-16T05:16:29.652 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.651+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.652 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.651+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1855671755}) 2024-09-16T05:16:29.652 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.651+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=243808739}) 2024-09-16T05:16:29.653 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.655+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.653 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.655+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4293509287}) 2024-09-16T05:16:29.654 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.655+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1847891853}) 2024-09-16T05:16:29.654 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.655+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.654 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.655+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4293509287}) 2024-09-16T05:16:29.654 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.655+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=727129453}) 2024-09-16T05:16:29.656 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.655+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.656 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.655+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1188813517}) 2024-09-16T05:16:29.656 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.655+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3765021817}) 2024-09-16T05:16:29.656 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.655+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.656 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.655+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1188813517}) 2024-09-16T05:16:29.657 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.659+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2522707916}) 2024-09-16T05:16:29.660 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.663+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.661 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.663+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1154645046}) 2024-09-16T05:16:29.661 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.663+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1627516627}) 2024-09-16T05:16:29.661 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.663+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.661 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.663+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1154645046}) 2024-09-16T05:16:29.661 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.663+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3992645315}) 2024-09-16T05:16:29.663 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.663+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.663 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.663+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=62} crc {osdmap=3885521989}) 2024-09-16T05:16:29.664 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.663+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=67} crc {osdmap=2696595218}) 2024-09-16T05:16:29.664 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.663+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:16:29.664 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.663+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=62} crc {osdmap=3885521989}) 2024-09-16T05:16:29.664 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:16:29.663+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=65} crc {osdmap=3055713824}) 2024-09-16T05:16:30.578 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:16:30.578 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:16:30.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 10 monclient: get_monmap_and_config 2024-09-16T05:16:30.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 10 monclient: build_initial_monmap 2024-09-16T05:16:30.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 10 monclient: monmap: 2024-09-16T05:16:30.707 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:30.707 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:30.707 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:30.710180+0000 2024-09-16T05:16:30.707 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:30.710180+0000 2024-09-16T05:16:30.707 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:30.707 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:30.707 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:30.707 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:30.707 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:30.707 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:30.707 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:30.707 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:30.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 1 Processor -- start 2024-09-16T05:16:30.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 1 -- start start 2024-09-16T05:16:30.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 10 monclient: init 2024-09-16T05:16:30.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:30.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-16T05:16:30.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc38068340 addr v1:172.21.15.114:6789/0 2024-09-16T05:16:30.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbc380a1d20 0x7fbc3813a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:30.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 10 monclient(hunting): picked mon.noname-a con 0x7fbc380a1d20 addr v2:172.21.15.114:3300/0 2024-09-16T05:16:30.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc380a2170 0x7fbc3813c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:30.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 10 monclient(hunting): picked mon.noname-b con 0x7fbc380a2170 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:30.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc3809ed90 con 0x7fbc38068340 2024-09-16T05:16:30.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:30.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fbc38068720 con 0x7fbc380a1d20 2024-09-16T05:16:30.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:30.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbc3809bd90 con 0x7fbc380a2170 2024-09-16T05:16:30.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:30.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3f048640 10 monclient(hunting): authenticate will time out at 1475.339489s 2024-09-16T05:16:30.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3cde4640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fbc38068340 0x7fbc380a1610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33180/0 (socket says 172.21.15.114:33180) 2024-09-16T05:16:30.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3cde4640 1 -- 172.21.15.114:0/597129205 learned_addr learned my addr 172.21.15.114:0/597129205 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:16:30.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3d5e5640 1 --2- 172.21.15.114:0/597129205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc380a2170 0x7fbc3813c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:30.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc37fff640 1 --2- 172.21.15.114:0/597129205 >> v2:172.21.15.114:3300/0 conn(0x7fbc380a1d20 0x7fbc3813a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:30.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3d5e5640 10 monclient(hunting): get_auth_request con 0x7fbc380a2170 auth_method 0 2024-09-16T05:16:30.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3d5e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:30.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3d5e5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:30.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3d5e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:30.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc37fff640 10 monclient(hunting): get_auth_request con 0x7fbc380a1d20 auth_method 0 2024-09-16T05:16:30.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:30.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:30.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:30.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:30.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:30.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:30.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3d5e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:30.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.707+0000 7fbc3d5e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:30.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3d5e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:30.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc37fff640 10 monclient(hunting): handle_auth_done global_id 74298 payload 274 2024-09-16T05:16:30.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc37fff640 1 -- 172.21.15.114:0/597129205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc380a2170 msgr2=0x7fbc3813c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:30.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc37fff640 1 --2- 172.21.15.114:0/597129205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc380a2170 0x7fbc3813c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:30.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc37fff640 1 -- 172.21.15.114:0/597129205 >> v1:172.21.15.114:6789/0 conn(0x7fbc38068340 legacy=0x7fbc380a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:30.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc37fff640 10 monclient: _finish_hunting 0 2024-09-16T05:16:30.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc37fff640 1 monclient: found mon.noname-a 2024-09-16T05:16:30.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:30.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc37fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:30.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc37fff640 1 -- 172.21.15.114:0/597129205 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc3809f2b0 con 0x7fbc380a1d20 2024-09-16T05:16:30.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc37fff640 1 --2- 172.21.15.114:0/597129205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc380a1d20 0x7fbc3813a110 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0097c0 tx=0x7fbc2c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd303032d8a0bcf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:30.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc377fe640 1 -- 172.21.15.114:0/597129205 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c0b0070 con 0x7fbc380a1d20 2024-09-16T05:16:30.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:30.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:30.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc377fe640 10 monclient: dump: 2024-09-16T05:16:30.718 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:30.718 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:30.718 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:30.718 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:30.718 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:30.719 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:30.719 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:30.719 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:30.719 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:30.719 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:30.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc377fe640 10 monclient: _finish_auth 0 2024-09-16T05:16:30.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc377fe640 10 monclient: _check_auth_tickets 2024-09-16T05:16:30.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:30.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc377fe640 1 -- 172.21.15.114:0/597129205 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc2c0ab600 con 0x7fbc380a1d20 2024-09-16T05:16:30.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 5 monclient: authenticate success, global_id 74298 2024-09-16T05:16:30.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:16:30.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc377fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:30.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:30.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc377fe640 1 -- 172.21.15.114:0/597129205 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c0ab9d0 con 0x7fbc380a1d20 2024-09-16T05:16:30.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 4 set_mon_vals no callback set 2024-09-16T05:16:30.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:30.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:30.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc377fe640 10 monclient: dump: 2024-09-16T05:16:30.720 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:30.720 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:30.720 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:30.720 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:30.720 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:30.720 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:30.720 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:30.720 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:30.720 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:30.721 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:30.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 10 monclient: shutdown 2024-09-16T05:16:30.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:30.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 1 -- 172.21.15.114:0/597129205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc380a1d20 msgr2=0x7fbc3813a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:30.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 1 --2- 172.21.15.114:0/597129205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc380a1d20 0x7fbc3813a110 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0097c0 tx=0x7fbc2c01c3f0 comp rx=0 tx=0).stop 2024-09-16T05:16:30.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 1 -- 172.21.15.114:0/597129205 shutdown_connections 2024-09-16T05:16:30.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 1 --2- 172.21.15.114:0/597129205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc380a2170 0x7fbc3813c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:30.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 1 --2- 172.21.15.114:0/597129205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc380a1d20 0x7fbc3813a110 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:30.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 1 -- 172.21.15.114:0/597129205 >> 172.21.15.114:0/597129205 conn(0x7fbc380a6390 msgr2=0x7fbc380a2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:30.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 1 -- 172.21.15.114:0/597129205 shutdown_connections 2024-09-16T05:16:30.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 1 -- 172.21.15.114:0/597129205 wait complete. 2024-09-16T05:16:30.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 10 monclient: build_initial_monmap 2024-09-16T05:16:30.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.711+0000 7fbc3f048640 10 monclient: monmap: 2024-09-16T05:16:30.722 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:30.722 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:30.722 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:30.722 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:30.722 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:30.722 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:30.722 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:30.722 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:30.722 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:30.722 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:30.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 librados: starting msgr at 2024-09-16T05:16:30.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 librados: starting objecter 2024-09-16T05:16:30.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 Processor -- start 2024-09-16T05:16:30.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 -- start start 2024-09-16T05:16:30.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 librados: setting wanted keys 2024-09-16T05:16:30.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 librados: calling monclient init 2024-09-16T05:16:30.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient: init 2024-09-16T05:16:30.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:30.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:16:30.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbc38068340 0x7fbc3813c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:30.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient(hunting): picked mon.noname-b con 0x7fbc38068340 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:30.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc380a1d20 0x7fbc3809cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:30.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient(hunting): picked mon.noname-a con 0x7fbc380a1d20 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:30.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc380a2170 0x7fbc3809d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:30.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient(hunting): picked mon.noname-c con 0x7fbc380a2170 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:30.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:30.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3813f750 con 0x7fbc380a1d20 2024-09-16T05:16:30.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:30.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3813f5d0 con 0x7fbc38068340 2024-09-16T05:16:30.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:30.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbc3813f8d0 con 0x7fbc380a2170 2024-09-16T05:16:30.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:30.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient(hunting): authenticate will time out at 1475.344775s 2024-09-16T05:16:30.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc37fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc380a1d20 0x7fbc3809cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:30.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc380a2170 0x7fbc3809d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:30.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc380a2170 0x7fbc3809d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:50774/0 (socket says 172.21.15.114:50774) 2024-09-16T05:16:30.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 1 -- 172.21.15.114:0/1102131258 learned_addr learned my addr 172.21.15.114:0/1102131258 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:30.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 10 monclient(hunting): get_auth_request con 0x7fbc380a2170 auth_method 0 2024-09-16T05:16:30.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:30.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:30.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:30.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc37fff640 10 monclient(hunting): get_auth_request con 0x7fbc380a1d20 auth_method 0 2024-09-16T05:16:30.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:30.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:30.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:30.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:30.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:30.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:30.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:30.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:30.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:30.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 10 monclient(hunting): handle_auth_done global_id 84137 payload 824 2024-09-16T05:16:30.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3cde4640 1 --2- 172.21.15.114:0/1102131258 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbc38068340 0x7fbc3813c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:30.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 1 -- 172.21.15.114:0/1102131258 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbc38068340 msgr2=0x7fbc3813c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:30.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 1 --2- 172.21.15.114:0/1102131258 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbc38068340 0x7fbc3813c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:30.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 1 -- 172.21.15.114:0/1102131258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc380a1d20 msgr2=0x7fbc3809cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:30.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 1 --2- 172.21.15.114:0/1102131258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc380a1d20 0x7fbc3809cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:30.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 10 monclient: _finish_hunting 0 2024-09-16T05:16:30.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 1 monclient: found mon.noname-c 2024-09-16T05:16:30.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:30.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:16:30.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 1 -- 172.21.15.114:0/1102131258 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc3809e690 con 0x7fbc380a2170 2024-09-16T05:16:30.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc37fff640 1 --2- 172.21.15.114:0/1102131258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc380a1d20 0x7fbc3809cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:16:30.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3d5e5640 1 --2- 172.21.15.114:0/1102131258 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc380a2170 0x7fbc3809d0c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009dd0 tx=0x7fbc3000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:30.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 10 client.?.objecter ms_handle_connect 0x7fbc380a2170 2024-09-16T05:16:30.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:30.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 1 -- 172.21.15.114:0/1102131258 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc30018020 con 0x7fbc380a2170 2024-09-16T05:16:30.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:30.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:16:30.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 10 monclient: dump: 2024-09-16T05:16:30.727 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:30.727 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:30.727 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:30.727 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:30.727 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:30.727 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:30.727 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:30.728 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:30.728 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:30.728 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:30.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 10 monclient: _finish_auth 0 2024-09-16T05:16:30.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 10 monclient: _check_auth_tickets 2024-09-16T05:16:30.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:30.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 5 monclient: authenticate success, global_id 84137 2024-09-16T05:16:30.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 1 -- 172.21.15.114:0/1102131258 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc30014480 con 0x7fbc380a2170 2024-09-16T05:16:30.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:30.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 1 -- 172.21.15.114:0/1102131258 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc30014870 con 0x7fbc380a2170 2024-09-16T05:16:30.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:30.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:16:30.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 10 monclient: dump: 2024-09-16T05:16:30.729 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:30.729 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:30.729 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:30.729 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:30.729 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:30.729 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:30.729 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:30.729 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:30.729 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:30.729 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:30.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc36ffd640 4 set_mon_vals no callback set 2024-09-16T05:16:30.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient: _renew_subs 2024-09-16T05:16:30.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:16:30.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 -- 172.21.15.114:0/1102131258 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc38138a30 con 0x7fbc380a2170 2024-09-16T05:16:30.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 client.84137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:30.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient: _renew_subs 2024-09-16T05:16:30.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:16:30.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 -- 172.21.15.114:0/1102131258 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc38138750 con 0x7fbc380a2170 2024-09-16T05:16:30.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 librados: init done 2024-09-16T05:16:30.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:30.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:30.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:16:30.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 1 -- 172.21.15.114:0/1102131258 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbc3001b020 con 0x7fbc380a2170 2024-09-16T05:16:30.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3f048640 1 -- 172.21.15.114:0/1102131258 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc380a1610 con 0x7fbc380a2170 2024-09-16T05:16:30.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 1 --2- 172.21.15.114:0/1102131258 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbc24037810 0x7fbc24039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:30.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 1 -- 172.21.15.114:0/1102131258 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(357..357 src has 1..357) ==== 5406+0+0 (secure 0 0 0) 0x7fbc300497d0 con 0x7fbc380a2170 2024-09-16T05:16:30.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 10 client.84137.objecter ms_dispatch 0x7fbc38000ba0 osd_map(357..357 src has 1..357) 2024-09-16T05:16:30.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 3 client.84137.objecter handle_osd_map got epochs [357,357] > 0 2024-09-16T05:16:30.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 3 client.84137.objecter handle_osd_map decoding full epoch 357 2024-09-16T05:16:30.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc357fa640 20 client.84137.objecter dump_active .. 0 homeless 2024-09-16T05:16:30.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3cde4640 1 --2- 172.21.15.114:0/1102131258 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbc24037810 0x7fbc24039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:30.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.715+0000 7fbc3cde4640 10 monclient: get_auth_request con 0x7fbc24037810 auth_method 0 2024-09-16T05:16:30.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.719+0000 7fbc3cde4640 1 --2- 172.21.15.114:0/1102131258 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbc24037810 0x7fbc24039cd0 secure :-1 s=READY pgs=3523 cs=0 l=1 rev1=1 crypto rx=0x7fbc3806d290 tx=0x7fbc28009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:30.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.719+0000 7fbc357fa640 10 client.84137.objecter ms_handle_connect 0x7fbc24037810 2024-09-16T05:16:30.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.719+0000 7fbc357fa640 1 -- 172.21.15.114:0/1102131258 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fbc30022070 con 0x7fbc380a2170 2024-09-16T05:16:30.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.719+0000 7fbc357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:30.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.719+0000 7fbc357fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:30.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:16:30.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:30.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:16:30.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 1 -- 172.21.15.114:0/1102131258 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbc3806bf50 con 0x7fbc380a2170 2024-09-16T05:16:30.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc357fa640 1 -- 172.21.15.114:0/1102131258 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fbc30010040 con 0x7fbc380a2170 2024-09-16T05:16:30.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:30.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc357fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:30.942 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:16:30.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 10 librados: watch_flush enter 2024-09-16T05:16:30.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 10 librados: watch_flush exit 2024-09-16T05:16:30.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 20 client.84137.objecter shutdown clearing up homeless session... 2024-09-16T05:16:30.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 10 client.84137.objecter successfully canceled tick 2024-09-16T05:16:30.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 1 -- 172.21.15.114:0/1102131258 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbc24037810 msgr2=0x7fbc24039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:30.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 1 --2- 172.21.15.114:0/1102131258 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbc24037810 0x7fbc24039cd0 secure :-1 s=READY pgs=3523 cs=0 l=1 rev1=1 crypto rx=0x7fbc3806d290 tx=0x7fbc28009040 comp rx=0 tx=0).stop 2024-09-16T05:16:30.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 10 monclient: shutdown 2024-09-16T05:16:30.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:30.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 1 -- 172.21.15.114:0/1102131258 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc380a2170 msgr2=0x7fbc3809d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:30.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 1 --2- 172.21.15.114:0/1102131258 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc380a2170 0x7fbc3809d0c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009dd0 tx=0x7fbc3000c490 comp rx=0 tx=0).stop 2024-09-16T05:16:30.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 1 -- 172.21.15.114:0/1102131258 shutdown_connections 2024-09-16T05:16:30.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 1 --2- 172.21.15.114:0/1102131258 >> v2:172.21.15.149:6812/1185531135 conn(0x7fbc24037810 0x7fbc24039cd0 unknown :-1 s=CLOSED pgs=3523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:30.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 1 --2- 172.21.15.114:0/1102131258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc380a1d20 0x7fbc3809cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:30.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 1 --2- 172.21.15.114:0/1102131258 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc380a2170 0x7fbc3809d0c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:30.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 1 --2- 172.21.15.114:0/1102131258 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbc38068340 0x7fbc3813c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:30.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.943+0000 7fbc3f048640 1 -- 172.21.15.114:0/1102131258 >> 172.21.15.114:0/1102131258 conn(0x7fbc380a6390 msgr2=0x7fbc38131610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:30.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.947+0000 7fbc3f048640 1 -- 172.21.15.114:0/1102131258 shutdown_connections 2024-09-16T05:16:30.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.947+0000 7fbc3f048640 1 -- 172.21.15.114:0/1102131258 wait complete. 2024-09-16T05:16:30.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:30.947+0000 7fbc3f048640 1 librados: shutdown 2024-09-16T05:16:30.959 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:16:30.959 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:16:30.959 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:16:30.959 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:16:31.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 10 monclient: get_monmap_and_config 2024-09-16T05:16:31.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 10 monclient: build_initial_monmap 2024-09-16T05:16:31.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 10 monclient: monmap: 2024-09-16T05:16:31.070 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:31.070 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.070 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:31.073845+0000 2024-09-16T05:16:31.070 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:31.073845+0000 2024-09-16T05:16:31.070 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:31.071 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.071 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:31.071 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:31.071 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:31.071 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:31.071 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:31.071 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 1 Processor -- start 2024-09-16T05:16:31.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 1 -- start start 2024-09-16T05:16:31.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 10 monclient: init 2024-09-16T05:16:31.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:31.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-16T05:16:31.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff64c068250 0x7ff64c0a0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 10 monclient(hunting): picked mon.noname-a con 0x7ff64c068250 addr v2:172.21.15.114:3300/0 2024-09-16T05:16:31.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff64c0a0c40 0x7ff64c0a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 10 monclient(hunting): picked mon.noname-b con 0x7ff64c0a0c40 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:31.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff64c134d90 addr v1:172.21.15.149:6789/0 2024-09-16T05:16:31.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff64c0a34c0 con 0x7ff64c134d90 2024-09-16T05:16:31.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff64c0a2270 con 0x7ff64c068250 2024-09-16T05:16:31.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff64c068630 con 0x7ff64c0a0c40 2024-09-16T05:16:31.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:31.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff653c94640 10 monclient(hunting): authenticate will time out at 1475.703632s 2024-09-16T05:16:31.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff651a30640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff64c068250 0x7ff64c0a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff65122f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff64c0a0c40 0x7ff64c0a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff65122f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff64c0a0c40 0x7ff64c0a10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:50780/0 (socket says 172.21.15.114:50780) 2024-09-16T05:16:31.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff65122f640 1 -- 172.21.15.114:0/3648925204 learned_addr learned my addr 172.21.15.114:0/3648925204 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:31.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff65122f640 10 monclient(hunting): get_auth_request con 0x7ff64c0a0c40 auth_method 0 2024-09-16T05:16:31.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff65122f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff65122f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff65122f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff651a30640 10 monclient(hunting): get_auth_request con 0x7ff64c068250 auth_method 0 2024-09-16T05:16:31.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff651a30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff651a30640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff651a30640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff65122f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff65122f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff65122f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff651a30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff651a30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.071+0000 7ff651a30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff651a30640 10 monclient(hunting): handle_auth_done global_id 74307 payload 274 2024-09-16T05:16:31.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff651a30640 1 -- 172.21.15.114:0/3648925204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff64c0a0c40 msgr2=0x7ff64c0a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:31.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff651a30640 1 --2- 172.21.15.114:0/3648925204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff64c0a0c40 0x7ff64c0a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff651a30640 1 -- 172.21.15.114:0/3648925204 >> v1:172.21.15.149:6789/0 conn(0x7ff64c134d90 legacy=0x7ff64c1351e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:31.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff651a30640 10 monclient: _finish_hunting 0 2024-09-16T05:16:31.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff651a30640 1 monclient: found mon.noname-a 2024-09-16T05:16:31.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff651a30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:31.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff651a30640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:31.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff651a30640 1 -- 172.21.15.114:0/3648925204 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64c0a1890 con 0x7ff64c068250 2024-09-16T05:16:31.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff65122f640 1 --2- 172.21.15.114:0/3648925204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff64c0a0c40 0x7ff64c0a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:16:31.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff651a30640 1 --2- 172.21.15.114:0/3648925204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c068250 0x7ff64c0a0700 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff6480097f0 tx=0x7ff64801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=228a0e1a2120083e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff650a2e640 1 -- 172.21.15.114:0/3648925204 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6480b0070 con 0x7ff64c068250 2024-09-16T05:16:31.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff650a2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff650a2e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:31.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff650a2e640 10 monclient: dump: 2024-09-16T05:16:31.084 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.084 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.084 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.084 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.084 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.084 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.084 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.084 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.085 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.085 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff650a2e640 10 monclient: _finish_auth 0 2024-09-16T05:16:31.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff650a2e640 10 monclient: _check_auth_tickets 2024-09-16T05:16:31.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff650a2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:31.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 5 monclient: authenticate success, global_id 74307 2024-09-16T05:16:31.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff650a2e640 1 -- 172.21.15.114:0/3648925204 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6480ab600 con 0x7ff64c068250 2024-09-16T05:16:31.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:16:31.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff650a2e640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:31.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:31.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 4 set_mon_vals no callback set 2024-09-16T05:16:31.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff650a2e640 1 -- 172.21.15.114:0/3648925204 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6480ab9d0 con 0x7ff64c068250 2024-09-16T05:16:31.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff650a2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff650a2e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:31.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff650a2e640 10 monclient: dump: 2024-09-16T05:16:31.086 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.086 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.086 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.086 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.086 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.086 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.086 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.086 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.086 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.086 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 10 monclient: shutdown 2024-09-16T05:16:31.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:31.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 1 -- 172.21.15.114:0/3648925204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c068250 msgr2=0x7ff64c0a0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 1 --2- 172.21.15.114:0/3648925204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c068250 0x7ff64c0a0700 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff6480097f0 tx=0x7ff64801c3b0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 1 -- 172.21.15.114:0/3648925204 shutdown_connections 2024-09-16T05:16:31.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 1 --2- 172.21.15.114:0/3648925204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff64c0a0c40 0x7ff64c0a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 1 --2- 172.21.15.114:0/3648925204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c068250 0x7ff64c0a0700 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 1 -- 172.21.15.114:0/3648925204 >> 172.21.15.114:0/3648925204 conn(0x7ff64c0a6380 msgr2=0x7ff64c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:31.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 1 -- 172.21.15.114:0/3648925204 shutdown_connections 2024-09-16T05:16:31.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 1 -- 172.21.15.114:0/3648925204 wait complete. 2024-09-16T05:16:31.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 10 monclient: build_initial_monmap 2024-09-16T05:16:31.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 10 monclient: monmap: 2024-09-16T05:16:31.087 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:31.088 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:31.088 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:31.088 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:31.088 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:31.088 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.088 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:31.088 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:31.088 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:31.088 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 1 librados: starting msgr at 2024-09-16T05:16:31.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 1 librados: starting objecter 2024-09-16T05:16:31.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 1 Processor -- start 2024-09-16T05:16:31.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 1 -- start start 2024-09-16T05:16:31.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 1 librados: setting wanted keys 2024-09-16T05:16:31.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 1 librados: calling monclient init 2024-09-16T05:16:31.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.075+0000 7ff653c94640 10 monclient: init 2024-09-16T05:16:31.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:31.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:16:31.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff64c068250 0x7ff64c13c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 monclient(hunting): picked mon.noname-b con 0x7ff64c068250 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:31.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff64c0a0c40 0x7ff64c09d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 monclient(hunting): picked mon.noname-c con 0x7ff64c0a0c40 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:31.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c134d90 0x7ff64c09d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 monclient(hunting): picked mon.noname-a con 0x7ff64c134d90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:31.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c13fb20 con 0x7ff64c134d90 2024-09-16T05:16:31.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c0a2640 con 0x7ff64c068250 2024-09-16T05:16:31.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff64c13fca0 con 0x7ff64c0a0c40 2024-09-16T05:16:31.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:31.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 monclient(hunting): authenticate will time out at 1475.708495s 2024-09-16T05:16:31.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff64c0a0c40 0x7ff64c09d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff652231640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c134d90 0x7ff64c09d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff64c0a0c40 0x7ff64c09d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:50782/0 (socket says 172.21.15.114:50782) 2024-09-16T05:16:31.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 1 -- 172.21.15.114:0/2410204239 learned_addr learned my addr 172.21.15.114:0/2410204239 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:31.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 10 monclient(hunting): get_auth_request con 0x7ff64c0a0c40 auth_method 0 2024-09-16T05:16:31.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff652231640 10 monclient(hunting): get_auth_request con 0x7ff64c134d90 auth_method 0 2024-09-16T05:16:31.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff652231640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff652231640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff652231640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff652231640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff652231640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff652231640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 10 monclient(hunting): handle_auth_done global_id 84143 payload 824 2024-09-16T05:16:31.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 1 -- 172.21.15.114:0/2410204239 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff64c068250 msgr2=0x7ff64c13c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff651a30640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff64c068250 0x7ff64c13c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff64c068250 0x7ff64c13c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 1 -- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c134d90 msgr2=0x7ff64c09d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c134d90 0x7ff64c09d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 10 monclient: _finish_hunting 0 2024-09-16T05:16:31.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 1 monclient: found mon.noname-c 2024-09-16T05:16:31.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:31.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:16:31.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 1 -- 172.21.15.114:0/2410204239 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64c13bce0 con 0x7ff64c0a0c40 2024-09-16T05:16:31.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff652231640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c134d90 0x7ff64c09d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:16:31.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff65122f640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff64c0a0c40 0x7ff64c09d230 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00ea00 tx=0x7ff63c00eec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 client.?.objecter ms_handle_connect 0x7ff64c0a0c40 2024-09-16T05:16:31.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:31.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 1 -- 172.21.15.114:0/2410204239 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c017070 con 0x7ff64c0a0c40 2024-09-16T05:16:31.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:16:31.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 monclient: dump: 2024-09-16T05:16:31.093 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.093 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.093 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.093 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.093 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.093 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.093 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.093 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.093 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.093 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 monclient: _finish_auth 0 2024-09-16T05:16:31.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 monclient: _check_auth_tickets 2024-09-16T05:16:31.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:31.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 5 monclient: authenticate success, global_id 84143 2024-09-16T05:16:31.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 1 -- 172.21.15.114:0/2410204239 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff63c00ca00 con 0x7ff64c0a0c40 2024-09-16T05:16:31.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:31.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 1 -- 172.21.15.114:0/2410204239 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c00cdb0 con 0x7ff64c0a0c40 2024-09-16T05:16:31.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 monclient: _renew_subs 2024-09-16T05:16:31.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:16:31.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 1 -- 172.21.15.114:0/2410204239 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff64c13a830 con 0x7ff64c0a0c40 2024-09-16T05:16:31.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff643fff640 4 set_mon_vals no callback set 2024-09-16T05:16:31.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:16:31.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 monclient: dump: 2024-09-16T05:16:31.095 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.095 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.095 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.095 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.095 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.095 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.095 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.095 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.095 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.095 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 client.84143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:31.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 monclient: _renew_subs 2024-09-16T05:16:31.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:16:31.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 1 -- 172.21.15.114:0/2410204239 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff64c13ba00 con 0x7ff64c0a0c40 2024-09-16T05:16:31.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 1 librados: init done 2024-09-16T05:16:31.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:31.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c09dcb0 0x7ff64c12c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff653c94640 1 -- 172.21.15.114:0/2410204239 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c09edf0 con 0x7ff64c09dcb0 2024-09-16T05:16:31.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 1 -- 172.21.15.114:0/2410204239 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff63c01a070 con 0x7ff64c0a0c40 2024-09-16T05:16:31.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff651a30640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c09dcb0 0x7ff64c12c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff651a30640 10 monclient: get_auth_request con 0x7ff64c09dcb0 auth_method 0 2024-09-16T05:16:31.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff651a30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff651a30640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff651a30640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff651a30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff651a30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff651a30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 1 --2- 172.21.15.114:0/2410204239 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff630037810 0x7ff630039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff651a30640 10 monclient(hunting): handle_auth_done global_id 74313 payload 824 2024-09-16T05:16:31.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff651a30640 1 -- 172.21.15.114:0/2410204239 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff64c09e090 con 0x7ff64c09dcb0 2024-09-16T05:16:31.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 1 -- 172.21.15.114:0/2410204239 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(357..357 src has 1..357) ==== 5406+0+0 (secure 0 0 0) 0x7ff63c048d40 con 0x7ff64c0a0c40 2024-09-16T05:16:31.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 client.84143.objecter ms_dispatch 0x7ff64c000ba0 osd_map(357..357 src has 1..357) 2024-09-16T05:16:31.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 3 client.84143.objecter handle_osd_map got epochs [357,357] > 0 2024-09-16T05:16:31.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 3 client.84143.objecter handle_osd_map decoding full epoch 357 2024-09-16T05:16:31.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 20 client.84143.objecter dump_active .. 0 homeless 2024-09-16T05:16:31.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff651a30640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c09dcb0 0x7ff64c12c660 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff64c09ff60 tx=0x7ff64801abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 client.84143.objecter ms_handle_connect 0x7ff64c09dcb0 2024-09-16T05:16:31.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 client.84143.objecter resend_mon_ops 2024-09-16T05:16:31.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 1 -- 172.21.15.114:0/2410204239 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6480b0050 con 0x7ff64c09dcb0 2024-09-16T05:16:31.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:31.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 monclient: dump: 2024-09-16T05:16:31.098 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.098 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.098 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.098 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.098 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.098 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.098 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.098 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.098 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.098 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff652231640 1 --2- 172.21.15.114:0/2410204239 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff630037810 0x7ff630039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff652231640 10 monclient: get_auth_request con 0x7ff630037810 auth_method 0 2024-09-16T05:16:31.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 1 -- 172.21.15.114:0/2410204239 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff6480b1040 con 0x7ff64c09dcb0 2024-09-16T05:16:31.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:31.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:31.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 1 -- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c09dcb0 msgr2=0x7ff64c12c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c09dcb0 0x7ff64c12c660 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff64c09ff60 tx=0x7ff64801abf0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 1 -- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c09dcb0 msgr2=0x7ff64c12c660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:31.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c09dcb0 0x7ff64c12c660 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff6427fc640 10 client.84143.objecter ms_handle_connect 0x7ff630037810 2024-09-16T05:16:31.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.079+0000 7ff652231640 1 --2- 172.21.15.114:0/2410204239 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff630037810 0x7ff630039cd0 secure :-1 s=READY pgs=3524 cs=0 l=1 rev1=1 crypto rx=0x7ff64400f220 tx=0x7ff644003d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:16:31.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c12cba0 0x7ff64c12cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 1 -- 172.21.15.114:0/2410204239 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c13c100 con 0x7ff64c12cba0 2024-09-16T05:16:31.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff651a30640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c12cba0 0x7ff64c12cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff651a30640 10 monclient: get_auth_request con 0x7ff64c12cba0 auth_method 0 2024-09-16T05:16:31.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff651a30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff651a30640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff651a30640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff651a30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff651a30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff651a30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff651a30640 10 monclient(hunting): handle_auth_done global_id 74316 payload 824 2024-09-16T05:16:31.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff651a30640 1 -- 172.21.15.114:0/2410204239 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff64c0a0700 con 0x7ff64c12cba0 2024-09-16T05:16:31.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff6427fc640 10 client.84143.objecter ms_handle_connect 0x7ff64c12cba0 2024-09-16T05:16:31.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff6427fc640 10 client.84143.objecter resend_mon_ops 2024-09-16T05:16:31.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff651a30640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c12cba0 0x7ff64c12cf80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff648009740 tx=0x7ff648019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff6427fc640 1 -- 172.21.15.114:0/2410204239 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6480b7b90 con 0x7ff64c12cba0 2024-09-16T05:16:31.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff6427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff6427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:31.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff6427fc640 10 monclient: dump: 2024-09-16T05:16:31.103 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.103 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.103 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.103 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.103 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.103 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.103 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.103 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.103 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.103 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff6427fc640 1 -- 172.21.15.114:0/2410204239 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7ff6480c1070 con 0x7ff64c12cba0 2024-09-16T05:16:31.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff6427fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:16:31.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff6427fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:31.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff6427fc640 1 -- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c12cba0 msgr2=0x7ff64c12cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff6427fc640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c12cba0 0x7ff64c12cf80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff648009740 tx=0x7ff648019150 comp rx=0 tx=0).stop 2024-09-16T05:16:31.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff6427fc640 1 -- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c12cba0 msgr2=0x7ff64c12cf80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:31.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff6427fc640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff64c12cba0 0x7ff64c12cf80 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff648009740 tx=0x7ff648019150 comp rx=0 tx=0).stop 2024-09-16T05:16:31.104 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:16:31.104 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:16:31.104 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:16:31.104 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:16:31.104 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 40, 2024-09-16T05:16:31.105 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 150142, 2024-09-16T05:16:31.105 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:16:31.105 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:16:31.105 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:16:31.105 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:16:31.105 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:31.105 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 4, 2024-09-16T05:16:31.105 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:31.105 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:16:31.105 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:16:31.105 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:31.105 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:31.105 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:31.106 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:31.106 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:31.106 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:31.106 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:31.106 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:31.106 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:31.106 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:31.106 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:31.106 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:31.106 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:16:31.106 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:16:31.106 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:31.106 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:31.106 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:31.107 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:31.107 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:31.107 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:31.107 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:31.107 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:31.107 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:31.107 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:31.107 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:31.107 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.107 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:16:31.107 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:16:31.107 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:16:31.107 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:16:31.108 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:16:31.108 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:16:31.108 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:31.108 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:31.108 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:16:31.108 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:16:31.108 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:16:31.108 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:16:31.108 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:16:31.108 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:16:31.108 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:16:31.108 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:31.109 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:16:31.109 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:31.109 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:31.109 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:31.109 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:31.109 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:31.109 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:31.109 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:31.109 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:31.109 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:31.109 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:31.109 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:31.109 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:31.109 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:16:31.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.110 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:16:31.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.110 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:16:31.110 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:16:31.110 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:31.110 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:31.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.110 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:31.110 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:16:31.110 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.110 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:31.111 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:16:31.111 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.111 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.111 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.111 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:16:31.111 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:16:31.111 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:31.111 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:31.111 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:31.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.111 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.111 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:16:31.112 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:16:31.112 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:31.112 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:31.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.112 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:31.112 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:16:31.112 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.112 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:31.112 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:16:31.112 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.112 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.112 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.113 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:16:31.113 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:16:31.113 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:31.113 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:31.113 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:31.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.113 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:16:31.113 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:16:31.113 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:31.113 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:31.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.114 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:31.114 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:16:31.114 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.114 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:31.114 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:16:31.114 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.114 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.114 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.114 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:16:31.114 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:16:31.115 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:31.115 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:31.115 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:31.115 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.115 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.115 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:16:31.115 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:16:31.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.115 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:31.115 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:31.115 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:16:31.115 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.115 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:31.116 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:16:31.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.116 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:31.116 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:31.116 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:16:31.116 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.116 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:31.116 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:16:31.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.116 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:31.116 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:31.116 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:16:31.116 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.117 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.117 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:16:31.117 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:16:31.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 10 librados: watch_flush enter 2024-09-16T05:16:31.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 10 librados: watch_flush exit 2024-09-16T05:16:31.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 20 client.84143.objecter shutdown clearing up homeless session... 2024-09-16T05:16:31.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 10 client.84143.objecter successfully canceled tick 2024-09-16T05:16:31.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 1 -- 172.21.15.114:0/2410204239 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff630037810 msgr2=0x7ff630039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 1 --2- 172.21.15.114:0/2410204239 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff630037810 0x7ff630039cd0 secure :-1 s=READY pgs=3524 cs=0 l=1 rev1=1 crypto rx=0x7ff64400f220 tx=0x7ff644003d50 comp rx=0 tx=0).stop 2024-09-16T05:16:31.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 10 monclient: shutdown 2024-09-16T05:16:31.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:31.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 1 -- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff64c0a0c40 msgr2=0x7ff64c09d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff652231640 1 -- 172.21.15.114:0/2410204239 reap_dead start 2024-09-16T05:16:31.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff64c0a0c40 0x7ff64c09d230 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00ea00 tx=0x7ff63c00eec0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 1 -- 172.21.15.114:0/2410204239 shutdown_connections 2024-09-16T05:16:31.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 1 --2- 172.21.15.114:0/2410204239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff64c0a0c40 0x7ff64c09d230 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 1 -- 172.21.15.114:0/2410204239 >> 172.21.15.114:0/2410204239 conn(0x7ff64c0a6380 msgr2=0x7ff64c131b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 1 -- 172.21.15.114:0/2410204239 shutdown_connections 2024-09-16T05:16:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 1 -- 172.21.15.114:0/2410204239 wait complete. 2024-09-16T05:16:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.087+0000 7ff653c94640 1 librados: shutdown 2024-09-16T05:16:31.120 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:16:31.120 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:16:31.120 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:16:31.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.231+0000 7fd3c293a640 10 monclient: get_monmap_and_config 2024-09-16T05:16:31.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.231+0000 7fd3c293a640 10 monclient: build_initial_monmap 2024-09-16T05:16:31.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.231+0000 7fd3c293a640 10 monclient: monmap: 2024-09-16T05:16:31.233 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:31.233 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.233 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:31.235948+0000 2024-09-16T05:16:31.233 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:31.235948+0000 2024-09-16T05:16:31.233 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:31.233 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.233 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:31.233 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:31.233 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:31.233 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:31.234 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:31.234 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 1 Processor -- start 2024-09-16T05:16:31.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 1 -- start start 2024-09-16T05:16:31.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 10 monclient: init 2024-09-16T05:16:31.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:31.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T05:16:31.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3bc06bcc0 addr v1:172.21.15.114:6789/0 2024-09-16T05:16:31.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd3bc0a3580 0x7fd3bc09f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 10 monclient(hunting): picked mon.noname-a con 0x7fd3bc0a3580 addr v2:172.21.15.114:3300/0 2024-09-16T05:16:31.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd3bc06c7b0 0x7fd3bc13e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 10 monclient(hunting): picked mon.noname-c con 0x7fd3bc06c7b0 addr v2:172.21.15.149:3300/0 2024-09-16T05:16:31.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3bc06d120 con 0x7fd3bc06bcc0 2024-09-16T05:16:31.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd3bc09afc0 con 0x7fd3bc0a3580 2024-09-16T05:16:31.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fd3bc0a3220 con 0x7fd3bc06c7b0 2024-09-16T05:16:31.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:31.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 10 monclient(hunting): authenticate will time out at 1475.864997s 2024-09-16T05:16:31.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd3bc0a3580 0x7fd3bc09f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd3bc0a3580 0x7fd3bc09f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32918/0 (socket says 172.21.15.114:32918) 2024-09-16T05:16:31.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 1 -- 172.21.15.114:0/2832825107 learned_addr learned my addr 172.21.15.114:0/2832825107 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 10 monclient(hunting): get_auth_request con 0x7fd3bc0a3580 auth_method 0 2024-09-16T05:16:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c0ed7640 1 --2- 172.21.15.114:0/2832825107 >> v2:172.21.15.149:3300/0 conn(0x7fd3bc06c7b0 0x7fd3bc13e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 10 monclient(hunting): handle_auth_done global_id 74319 payload 274 2024-09-16T05:16:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 1 -- 172.21.15.114:0/2832825107 >> v2:172.21.15.149:3300/0 conn(0x7fd3bc06c7b0 msgr2=0x7fd3bc13e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 1 --2- 172.21.15.114:0/2832825107 >> v2:172.21.15.149:3300/0 conn(0x7fd3bc06c7b0 0x7fd3bc13e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 1 -- 172.21.15.114:0/2832825107 >> v1:172.21.15.114:6789/0 conn(0x7fd3bc06bcc0 legacy=0x7fd3bc06c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:16:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 1 monclient: found mon.noname-a 2024-09-16T05:16:31.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:31.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:31.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 1 -- 172.21.15.114:0/2832825107 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3bc13fec0 con 0x7fd3bc0a3580 2024-09-16T05:16:31.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3b37fe640 1 --2- 172.21.15.114:0/2832825107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3bc0a3580 0x7fd3bc09f070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc09e670 tx=0x7fd3a401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d75d067adb59110 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3bb7fe640 1 -- 172.21.15.114:0/2832825107 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a40b0070 con 0x7fd3bc0a3580 2024-09-16T05:16:31.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3bb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:31.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3bb7fe640 10 monclient: dump: 2024-09-16T05:16:31.239 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.239 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.239 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.239 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.239 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.239 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.239 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.240 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.240 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.240 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3bb7fe640 10 monclient: _finish_auth 0 2024-09-16T05:16:31.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3bb7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:16:31.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:31.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3bb7fe640 1 -- 172.21.15.114:0/2832825107 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a40ab600 con 0x7fd3bc0a3580 2024-09-16T05:16:31.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3bb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:31.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3bb7fe640 1 -- 172.21.15.114:0/2832825107 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a40ab9d0 con 0x7fd3bc0a3580 2024-09-16T05:16:31.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3bb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:31.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3bb7fe640 10 monclient: dump: 2024-09-16T05:16:31.240 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.240 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.241 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.241 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.241 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.241 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.241 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.241 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.241 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.241 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 5 monclient: authenticate success, global_id 74319 2024-09-16T05:16:31.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:31.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.235+0000 7fd3c293a640 4 set_mon_vals no callback set 2024-09-16T05:16:31.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient: shutdown 2024-09-16T05:16:31.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:31.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 -- 172.21.15.114:0/2832825107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3bc0a3580 msgr2=0x7fd3bc09f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 --2- 172.21.15.114:0/2832825107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3bc0a3580 0x7fd3bc09f070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc09e670 tx=0x7fd3a401c3e0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 -- 172.21.15.114:0/2832825107 shutdown_connections 2024-09-16T05:16:31.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 --2- 172.21.15.114:0/2832825107 >> v2:172.21.15.149:3300/0 conn(0x7fd3bc06c7b0 0x7fd3bc13e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 --2- 172.21.15.114:0/2832825107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3bc0a3580 0x7fd3bc09f070 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 -- 172.21.15.114:0/2832825107 >> 172.21.15.114:0/2832825107 conn(0x7fd3bc0a4280 msgr2=0x7fd3bc0a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:31.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 -- 172.21.15.114:0/2832825107 shutdown_connections 2024-09-16T05:16:31.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 -- 172.21.15.114:0/2832825107 wait complete. 2024-09-16T05:16:31.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient: build_initial_monmap 2024-09-16T05:16:31.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient: monmap: 2024-09-16T05:16:31.250 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:31.250 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:31.250 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:31.250 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:31.250 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:31.250 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.250 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:31.250 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:31.250 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:31.250 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 librados: starting msgr at 2024-09-16T05:16:31.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 librados: starting objecter 2024-09-16T05:16:31.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 Processor -- start 2024-09-16T05:16:31.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 -- start start 2024-09-16T05:16:31.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 librados: setting wanted keys 2024-09-16T05:16:31.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 librados: calling monclient init 2024-09-16T05:16:31.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient: init 2024-09-16T05:16:31.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:31.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:16:31.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd3bc06bcc0 0x7fd3bc139990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient(hunting): picked mon.noname-b con 0x7fd3bc06bcc0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:31.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc06c7b0 0x7fd3bc139ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient(hunting): picked mon.noname-c con 0x7fd3bc06c7b0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:31.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3bc0a3580 0x7fd3bc13a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient(hunting): picked mon.noname-a con 0x7fd3bc0a3580 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:31.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd3bc13fa50 con 0x7fd3bc0a3580 2024-09-16T05:16:31.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fd3bc13f8d0 con 0x7fd3bc06bcc0 2024-09-16T05:16:31.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd3bc13fbd0 con 0x7fd3bc06c7b0 2024-09-16T05:16:31.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:31.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient(hunting): authenticate will time out at 1475.869975s 2024-09-16T05:16:31.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b37fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc06c7b0 0x7fd3bc139ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3bc0a3580 0x7fd3bc13a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3bc0a3580 0x7fd3bc13a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32924/0 (socket says 172.21.15.114:32924) 2024-09-16T05:16:31.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b37fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc06c7b0 0x7fd3bc139ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:50794/0 (socket says 172.21.15.114:50794) 2024-09-16T05:16:31.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 1 -- 172.21.15.114:0/1304015044 learned_addr learned my addr 172.21.15.114:0/1304015044 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:31.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 10 monclient(hunting): get_auth_request con 0x7fd3bc0a3580 auth_method 0 2024-09-16T05:16:31.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b37fe640 10 monclient(hunting): get_auth_request con 0x7fd3bc06c7b0 auth_method 0 2024-09-16T05:16:31.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3bbfff640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd3bc06bcc0 0x7fd3bc139990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 10 monclient(hunting): handle_auth_done global_id 74322 payload 824 2024-09-16T05:16:31.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 1 -- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc06c7b0 msgr2=0x7fd3bc139ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc06c7b0 0x7fd3bc139ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 1 -- 172.21.15.114:0/1304015044 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd3bc06bcc0 msgr2=0x7fd3bc139990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd3bc06bcc0 0x7fd3bc139990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 10 monclient: _finish_hunting 0 2024-09-16T05:16:31.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 1 monclient: found mon.noname-a 2024-09-16T05:16:31.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:31.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:31.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 1 -- 172.21.15.114:0/1304015044 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3bc13cab0 con 0x7fd3bc0a3580 2024-09-16T05:16:31.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b37fe640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc06c7b0 0x7fd3bc139ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:16:31.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3bbfff640 10 monclient: get_auth_request con 0x7fd3bc06bcc0 auth_method 0 2024-09-16T05:16:31.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3bbfff640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd3bc06bcc0 0x7fd3bc139990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:16:31.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c0ed7640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3bc0a3580 0x7fd3bc13a410 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400cd70 tx=0x7fd3b400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 10 client.?.objecter ms_handle_connect 0x7fd3bc0a3580 2024-09-16T05:16:31.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:31.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 1 -- 172.21.15.114:0/1304015044 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4003ba0 con 0x7fd3bc0a3580 2024-09-16T05:16:31.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:31.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 10 monclient: dump: 2024-09-16T05:16:31.255 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.255 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.255 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.255 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.255 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.255 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.255 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.255 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.255 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.256 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 10 monclient: _finish_auth 0 2024-09-16T05:16:31.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 10 monclient: _check_auth_tickets 2024-09-16T05:16:31.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:31.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 5 monclient: authenticate success, global_id 74322 2024-09-16T05:16:31.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 1 -- 172.21.15.114:0/1304015044 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3b4003d40 con 0x7fd3bc0a3580 2024-09-16T05:16:31.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient: _renew_subs 2024-09-16T05:16:31.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:31.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 -- 172.21.15.114:0/1304015044 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3bc13c7a0 con 0x7fd3bc0a3580 2024-09-16T05:16:31.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:31.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 1 -- 172.21.15.114:0/1304015044 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4004240 con 0x7fd3bc0a3580 2024-09-16T05:16:31.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:31.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 10 monclient: dump: 2024-09-16T05:16:31.257 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.257 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.257 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.257 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.257 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.257 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.257 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.257 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.257 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.257 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 client.74322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:31.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient: _renew_subs 2024-09-16T05:16:31.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:31.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 -- 172.21.15.114:0/1304015044 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3bc13b030 con 0x7fd3bc0a3580 2024-09-16T05:16:31.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 librados: init done 2024-09-16T05:16:31.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3ba7fc640 4 set_mon_vals no callback set 2024-09-16T05:16:31.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:31.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc137720 0x7fd3bc137b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3c293a640 1 -- 172.21.15.114:0/1304015044 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd3bc0a7080 con 0x7fd3bc137720 2024-09-16T05:16:31.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 1 -- 172.21.15.114:0/1304015044 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd3b40043e0 con 0x7fd3bc0a3580 2024-09-16T05:16:31.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3bbfff640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc137720 0x7fd3bc137b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3bbfff640 10 monclient: get_auth_request con 0x7fd3bc137720 auth_method 0 2024-09-16T05:16:31.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 1 --2- 172.21.15.114:0/1304015044 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd39c037840 0x7fd39c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 1 -- 172.21.15.114:0/1304015044 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(357..357 src has 1..357) ==== 5406+0+0 (secure 0 0 0) 0x7fd3b4049820 con 0x7fd3bc0a3580 2024-09-16T05:16:31.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 10 client.74322.objecter ms_dispatch 0x7fd3bc000ba0 osd_map(357..357 src has 1..357) 2024-09-16T05:16:31.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 3 client.74322.objecter handle_osd_map got epochs [357,357] > 0 2024-09-16T05:16:31.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.239+0000 7fd3b97fa640 3 client.74322.objecter handle_osd_map decoding full epoch 357 2024-09-16T05:16:31.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3bbfff640 10 monclient(hunting): handle_auth_done global_id 84149 payload 824 2024-09-16T05:16:31.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3bbfff640 1 -- 172.21.15.114:0/1304015044 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd3bc13cdc0 con 0x7fd3bc137720 2024-09-16T05:16:31.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b97fa640 20 client.74322.objecter dump_active .. 0 homeless 2024-09-16T05:16:31.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b97fa640 10 client.74322.objecter ms_handle_connect 0x7fd3bc137720 2024-09-16T05:16:31.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3bbfff640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc137720 0x7fd3bc137b00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00d420 tx=0x7fd3ac00d8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b97fa640 10 client.74322.objecter resend_mon_ops 2024-09-16T05:16:31.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b37fe640 1 --2- 172.21.15.114:0/1304015044 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd39c037840 0x7fd39c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b37fe640 10 monclient: get_auth_request con 0x7fd39c037840 auth_method 0 2024-09-16T05:16:31.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b97fa640 1 -- 172.21.15.114:0/1304015044 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac005480 con 0x7fd3bc137720 2024-09-16T05:16:31.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:16:31.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b97fa640 10 monclient: dump: 2024-09-16T05:16:31.261 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.261 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.261 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.261 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.261 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.261 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.261 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.262 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.262 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.262 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b97fa640 1 -- 172.21.15.114:0/1304015044 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd3ac017070 con 0x7fd3bc137720 2024-09-16T05:16:31.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:31.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:31.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b97fa640 1 -- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc137720 msgr2=0x7fd3bc137b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b97fa640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc137720 0x7fd3bc137b00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00d420 tx=0x7fd3ac00d8e0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b97fa640 1 -- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc137720 msgr2=0x7fd3bc137b00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:31.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b97fa640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc137720 0x7fd3bc137b00 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00d420 tx=0x7fd3ac00d8e0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b97fa640 10 client.74322.objecter ms_handle_connect 0x7fd39c037840 2024-09-16T05:16:31.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.243+0000 7fd3b37fe640 1 --2- 172.21.15.114:0/1304015044 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd39c037840 0x7fd39c039d00 secure :-1 s=READY pgs=3525 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40ad000 tx=0x7fd3a40a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c293a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:16:31.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c293a640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc138040 0x7fd3bc0713b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c293a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c293a640 1 -- 172.21.15.114:0/1304015044 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd3bc09e7a0 con 0x7fd3bc138040 2024-09-16T05:16:31.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3bbfff640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc138040 0x7fd3bc0713b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3bbfff640 10 monclient: get_auth_request con 0x7fd3bc138040 auth_method 0 2024-09-16T05:16:31.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3bbfff640 10 monclient(hunting): handle_auth_done global_id 84152 payload 824 2024-09-16T05:16:31.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3bbfff640 1 -- 172.21.15.114:0/1304015044 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd3bc13d400 con 0x7fd3bc138040 2024-09-16T05:16:31.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3bbfff640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc138040 0x7fd3bc0713b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00d420 tx=0x7fd3ac000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3b97fa640 10 client.74322.objecter ms_handle_connect 0x7fd3bc138040 2024-09-16T05:16:31.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3b97fa640 10 client.74322.objecter resend_mon_ops 2024-09-16T05:16:31.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3b97fa640 1 -- 172.21.15.114:0/1304015044 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac003b30 con 0x7fd3bc138040 2024-09-16T05:16:31.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:16:31.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3b97fa640 10 monclient: dump: 2024-09-16T05:16:31.264 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.264 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.264 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.264 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.265 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.265 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.265 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.265 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.265 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.265 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3b97fa640 1 -- 172.21.15.114:0/1304015044 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7fd3ac012410 con 0x7fd3bc138040 2024-09-16T05:16:31.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3b97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:16:31.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3b97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:31.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3b97fa640 1 -- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc138040 msgr2=0x7fd3bc0713b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3b97fa640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc138040 0x7fd3bc0713b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00d420 tx=0x7fd3ac000ee0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3b97fa640 1 -- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc138040 msgr2=0x7fd3bc0713b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:31.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3b97fa640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3bc138040 0x7fd3bc0713b0 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00d420 tx=0x7fd3ac000ee0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.266 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:16:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:16:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:16:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:16:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 40, 2024-09-16T05:16:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 13015, 2024-09-16T05:16:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:16:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:16:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:16:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:16:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 4, 2024-09-16T05:16:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:16:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:16:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:16:31.269 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:16:31.269 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:31.269 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:31.269 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:31.269 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:31.269 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:31.269 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:31.269 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:31.269 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:31.269 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:31.269 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:31.269 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:31.269 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.270 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.270 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:16:31.270 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:16:31.270 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:16:31.270 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:16:31.270 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:16:31.270 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:16:31.270 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:31.270 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:31.270 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:16:31.270 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:16:31.270 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:16:31.271 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:16:31.271 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:16:31.271 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:16:31.271 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:16:31.271 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:31.271 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:16:31.271 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:31.271 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:31.271 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:31.271 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:31.271 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:31.271 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:31.271 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:31.272 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:31.272 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:31.272 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:31.272 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:31.272 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:31.272 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:16:31.272 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.272 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:16:31.272 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.272 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:16:31.272 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:16:31.272 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:31.272 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:31.272 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.273 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:31.273 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:16:31.273 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.273 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.273 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.273 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:31.273 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:16:31.273 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.273 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.273 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.273 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.273 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:16:31.273 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:16:31.274 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:31.274 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:31.274 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:31.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.274 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.274 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:16:31.274 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:16:31.274 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:31.274 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:31.274 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.274 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:31.274 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:16:31.274 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.275 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.275 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:31.275 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:16:31.275 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.275 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.275 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.275 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.275 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:16:31.275 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:16:31.275 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:31.275 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:31.275 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:31.275 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.275 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.275 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:16:31.276 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:16:31.276 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:31.276 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:31.276 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.276 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:31.276 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:16:31.276 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.276 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.276 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.276 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:31.276 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:16:31.276 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.276 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.276 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.277 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.277 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:16:31.277 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:16:31.277 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:31.277 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:31.277 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:31.277 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.277 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.277 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.277 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:16:31.277 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:16:31.277 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.277 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:31.278 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:31.278 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:16:31.278 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.278 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:31.278 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:16:31.278 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.278 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:31.278 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:31.278 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:16:31.278 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.278 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.278 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.278 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:16:31.278 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:16:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c293a640 10 librados: watch_flush enter 2024-09-16T05:16:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c293a640 10 librados: watch_flush exit 2024-09-16T05:16:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c293a640 20 client.74322.objecter shutdown clearing up homeless session... 2024-09-16T05:16:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c293a640 10 client.74322.objecter successfully canceled tick 2024-09-16T05:16:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c293a640 1 -- 172.21.15.114:0/1304015044 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd39c037840 msgr2=0x7fd39c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c293a640 1 --2- 172.21.15.114:0/1304015044 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd39c037840 0x7fd39c039d00 secure :-1 s=READY pgs=3525 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40ad000 tx=0x7fd3a40a6000 comp rx=0 tx=0).stop 2024-09-16T05:16:31.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c293a640 10 monclient: shutdown 2024-09-16T05:16:31.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c293a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:31.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c293a640 1 -- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3bc0a3580 msgr2=0x7fd3bc13a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c293a640 1 --2- 172.21.15.114:0/1304015044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3bc0a3580 0x7fd3bc13a410 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400cd70 tx=0x7fd3b400a660 comp rx=0 tx=0).stop 2024-09-16T05:16:31.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c0ed7640 1 -- 172.21.15.114:0/1304015044 reap_dead start 2024-09-16T05:16:31.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.247+0000 7fd3c0ed7640 1 -- 172.21.15.114:0/1304015044 reap_dead start 2024-09-16T05:16:31.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.251+0000 7fd3c293a640 1 -- 172.21.15.114:0/1304015044 shutdown_connections 2024-09-16T05:16:31.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.251+0000 7fd3c293a640 1 -- 172.21.15.114:0/1304015044 >> 172.21.15.114:0/1304015044 conn(0x7fd3bc0a4280 msgr2=0x7fd3bc0a04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:31.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.251+0000 7fd3c293a640 1 -- 172.21.15.114:0/1304015044 shutdown_connections 2024-09-16T05:16:31.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.251+0000 7fd3c293a640 1 -- 172.21.15.114:0/1304015044 wait complete. 2024-09-16T05:16:31.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.251+0000 7fd3c293a640 1 librados: shutdown 2024-09-16T05:16:31.282 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:16:31.282 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:16:31.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:16:31.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 10 monclient: get_monmap_and_config 2024-09-16T05:16:31.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 10 monclient: build_initial_monmap 2024-09-16T05:16:31.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 10 monclient: monmap: 2024-09-16T05:16:31.386 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:31.386 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.386 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:31.389355+0000 2024-09-16T05:16:31.386 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:31.389355+0000 2024-09-16T05:16:31.386 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:31.387 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.387 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:31.387 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:31.387 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:31.387 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:31.387 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:31.387 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 1 Processor -- start 2024-09-16T05:16:31.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 1 -- start start 2024-09-16T05:16:31.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 10 monclient: init 2024-09-16T05:16:31.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:31.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-16T05:16:31.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f8c09b6c0 0x7f1f8c0a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 10 monclient(hunting): picked mon.noname-b con 0x7f1f8c09b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:31.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f8c0a1400 addr v1:172.21.15.149:6789/0 2024-09-16T05:16:31.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1f8c0a17e0 0x7f1f8c13cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8c0a17e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:16:31.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8c09d4e0 con 0x7f1f8c0a1400 2024-09-16T05:16:31.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f1f8c09e390 con 0x7f1f8c0a17e0 2024-09-16T05:16:31.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1f8c09ece0 con 0x7f1f8c09b6c0 2024-09-16T05:16:31.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:31.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f941ea640 10 monclient(hunting): authenticate will time out at 1476.018579s 2024-09-16T05:16:31.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f91f86640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f8c09b6c0 0x7f1f8c0a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1f8c0a17e0 0x7f1f8c13cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1f8c0a17e0 0x7f1f8c13cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32926/0 (socket says 172.21.15.114:32926) 2024-09-16T05:16:31.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f91f86640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f8c09b6c0 0x7f1f8c0a0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:50832/0 (socket says 172.21.15.114:50832) 2024-09-16T05:16:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 1 -- 172.21.15.114:0/3466632328 learned_addr learned my addr 172.21.15.114:0/3466632328 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 10 monclient(hunting): get_auth_request con 0x7f1f8c0a17e0 auth_method 0 2024-09-16T05:16:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f91f86640 10 monclient(hunting): get_auth_request con 0x7f1f8c09b6c0 auth_method 0 2024-09-16T05:16:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f91f86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f91f86640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f91f86640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f91f86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f91f86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f91f86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 10 monclient(hunting): handle_auth_done global_id 74325 payload 274 2024-09-16T05:16:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 1 -- 172.21.15.114:0/3466632328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f8c09b6c0 msgr2=0x7f1f8c0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 1 --2- 172.21.15.114:0/3466632328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f8c09b6c0 0x7f1f8c0a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 1 -- 172.21.15.114:0/3466632328 >> v1:172.21.15.149:6789/0 conn(0x7f1f8c0a1400 legacy=0x7f1f8c139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 10 monclient: _finish_hunting 0 2024-09-16T05:16:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 1 monclient: found mon.noname-a 2024-09-16T05:16:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f92787640 1 -- 172.21.15.114:0/3466632328 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8c09f430 con 0x7f1f8c0a17e0 2024-09-16T05:16:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.387+0000 7f1f91f86640 1 --2- 172.21.15.114:0/3466632328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f8c09b6c0 0x7f1f8c0a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:16:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f92787640 1 --2- 172.21.15.114:0/3466632328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f8c0a17e0 0x7f1f8c13cd70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1f84009e50 tx=0x7f1f8401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf700644c99a8437 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f90f84640 1 -- 172.21.15.114:0/3466632328 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f840a5620 con 0x7f1f8c0a17e0 2024-09-16T05:16:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f90f84640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f90f84640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f90f84640 10 monclient: dump: 2024-09-16T05:16:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.402 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f90f84640 10 monclient: _finish_auth 0 2024-09-16T05:16:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f90f84640 10 monclient: _check_auth_tickets 2024-09-16T05:16:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f90f84640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f90f84640 1 -- 172.21.15.114:0/3466632328 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f840a57c0 con 0x7f1f8c0a17e0 2024-09-16T05:16:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f90f84640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f90f84640 1 -- 172.21.15.114:0/3466632328 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f840a5b90 con 0x7f1f8c0a17e0 2024-09-16T05:16:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 5 monclient: authenticate success, global_id 74325 2024-09-16T05:16:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 4 set_mon_vals no callback set 2024-09-16T05:16:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f90f84640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f90f84640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f90f84640 10 monclient: dump: 2024-09-16T05:16:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.404 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 10 monclient: shutdown 2024-09-16T05:16:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 -- 172.21.15.114:0/3466632328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f8c0a17e0 msgr2=0x7f1f8c13cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 --2- 172.21.15.114:0/3466632328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f8c0a17e0 0x7f1f8c13cd70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1f84009e50 tx=0x7f1f8401c8b0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 -- 172.21.15.114:0/3466632328 shutdown_connections 2024-09-16T05:16:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 --2- 172.21.15.114:0/3466632328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f8c0a17e0 0x7f1f8c13cd70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 --2- 172.21.15.114:0/3466632328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f8c09b6c0 0x7f1f8c0a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 -- 172.21.15.114:0/3466632328 >> 172.21.15.114:0/3466632328 conn(0x7f1f8c0a1fb0 msgr2=0x7f1f8c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 -- 172.21.15.114:0/3466632328 shutdown_connections 2024-09-16T05:16:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 -- 172.21.15.114:0/3466632328 wait complete. 2024-09-16T05:16:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 10 monclient: build_initial_monmap 2024-09-16T05:16:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 10 monclient: monmap: 2024-09-16T05:16:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:31.407 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 librados: starting msgr at 2024-09-16T05:16:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 librados: starting objecter 2024-09-16T05:16:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 Processor -- start 2024-09-16T05:16:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 -- start start 2024-09-16T05:16:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 librados: setting wanted keys 2024-09-16T05:16:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 librados: calling monclient init 2024-09-16T05:16:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 10 monclient: init 2024-09-16T05:16:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:16:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f8c09b6c0 0x7f1f8c13ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8c09b6c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c0a1400 0x7f1f8c091d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 10 monclient(hunting): picked mon.noname-b con 0x7f1f8c0a1400 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:31.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f8c0a17e0 0x7f1f8c092290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 10 monclient(hunting): picked mon.noname-c con 0x7f1f8c0a17e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:31.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c0a1c50 con 0x7f1f8c09b6c0 2024-09-16T05:16:31.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c09cad0 con 0x7f1f8c0a1400 2024-09-16T05:16:31.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1f8c13fad0 con 0x7f1f8c0a17e0 2024-09-16T05:16:31.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:31.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f941ea640 10 monclient(hunting): authenticate will time out at 1476.023459s 2024-09-16T05:16:31.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f8c09b6c0 0x7f1f8c13ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f8c09b6c0 0x7f1f8c13ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32938/0 (socket says 172.21.15.114:32938) 2024-09-16T05:16:31.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 1 -- 172.21.15.114:0/4001496777 learned_addr learned my addr 172.21.15.114:0/4001496777 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 10 monclient(hunting): get_auth_request con 0x7f1f8c09b6c0 auth_method 0 2024-09-16T05:16:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f92787640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f8c0a17e0 0x7f1f8c092290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f92787640 10 monclient(hunting): get_auth_request con 0x7f1f8c0a17e0 auth_method 0 2024-09-16T05:16:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f92787640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f92787640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f92787640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f92787640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f92787640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f92787640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 10 monclient(hunting): handle_auth_done global_id 74328 payload 824 2024-09-16T05:16:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 1 -- 172.21.15.114:0/4001496777 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f8c0a17e0 msgr2=0x7f1f8c092290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f8c0a17e0 0x7f1f8c092290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 1 -- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c0a1400 msgr2=0x7f1f8c091d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91785640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c0a1400 0x7f1f8c091d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c0a1400 0x7f1f8c091d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 10 monclient: _finish_hunting 0 2024-09-16T05:16:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 1 monclient: found mon.noname-a 2024-09-16T05:16:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.391+0000 7f1f91f86640 1 -- 172.21.15.114:0/4001496777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8c072de0 con 0x7f1f8c09b6c0 2024-09-16T05:16:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f91f86640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f8c09b6c0 0x7f1f8c13ca30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00e980 tx=0x7f1f7c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 client.?.objecter ms_handle_connect 0x7f1f8c09b6c0 2024-09-16T05:16:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 1 -- 172.21.15.114:0/4001496777 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c017070 con 0x7f1f8c09b6c0 2024-09-16T05:16:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 monclient: dump: 2024-09-16T05:16:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.412 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.412 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.412 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 monclient: _finish_auth 0 2024-09-16T05:16:31.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 monclient: _check_auth_tickets 2024-09-16T05:16:31.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:31.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 1 -- 172.21.15.114:0/4001496777 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f7c00c9a0 con 0x7f1f8c09b6c0 2024-09-16T05:16:31.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:31.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 1 -- 172.21.15.114:0/4001496777 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c00cd70 con 0x7f1f8c09b6c0 2024-09-16T05:16:31.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:31.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 monclient: dump: 2024-09-16T05:16:31.413 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.413 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.413 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.413 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.413 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.413 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.413 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.413 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.413 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.413 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f83fff640 4 set_mon_vals no callback set 2024-09-16T05:16:31.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f941ea640 5 monclient: authenticate success, global_id 74328 2024-09-16T05:16:31.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f941ea640 10 monclient: _renew_subs 2024-09-16T05:16:31.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f941ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:31.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f941ea640 1 -- 172.21.15.114:0/4001496777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f8c095ce0 con 0x7f1f8c09b6c0 2024-09-16T05:16:31.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f941ea640 10 client.74328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:31.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f941ea640 10 monclient: _renew_subs 2024-09-16T05:16:31.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f941ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:31.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f941ea640 1 -- 172.21.15.114:0/4001496777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f8c095720 con 0x7f1f8c09b6c0 2024-09-16T05:16:31.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f941ea640 1 librados: init done 2024-09-16T05:16:31.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f941ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:31.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f941ea640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c02d7d0 0x7f1f8c0927d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f941ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f941ea640 1 -- 172.21.15.114:0/4001496777 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c069390 con 0x7f1f8c02d7d0 2024-09-16T05:16:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 1 -- 172.21.15.114:0/4001496777 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1f7c01a070 con 0x7f1f8c09b6c0 2024-09-16T05:16:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f91785640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c02d7d0 0x7f1f8c0927d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 1 --2- 172.21.15.114:0/4001496777 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1f70037810 0x7f1f70039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 1 -- 172.21.15.114:0/4001496777 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(357..357 src has 1..357) ==== 5406+0+0 (secure 0 0 0) 0x7f1f7c048d10 con 0x7f1f8c09b6c0 2024-09-16T05:16:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f91785640 10 monclient: get_auth_request con 0x7f1f8c02d7d0 auth_method 0 2024-09-16T05:16:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f91785640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f91785640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 client.74328.objecter ms_dispatch 0x7f1f8c000ba0 osd_map(357..357 src has 1..357) 2024-09-16T05:16:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f91785640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 3 client.74328.objecter handle_osd_map got epochs [357,357] > 0 2024-09-16T05:16:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 3 client.74328.objecter handle_osd_map decoding full epoch 357 2024-09-16T05:16:31.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 20 client.74328.objecter dump_active .. 0 homeless 2024-09-16T05:16:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f91785640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f91785640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f91785640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f92787640 1 --2- 172.21.15.114:0/4001496777 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1f70037810 0x7f1f70039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f92787640 10 monclient: get_auth_request con 0x7f1f70037810 auth_method 0 2024-09-16T05:16:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f91785640 10 monclient(hunting): handle_auth_done global_id 64456 payload 824 2024-09-16T05:16:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f91785640 1 -- 172.21.15.114:0/4001496777 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1f8c02dbb0 con 0x7f1f8c02d7d0 2024-09-16T05:16:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 client.74328.objecter ms_handle_connect 0x7f1f8c02d7d0 2024-09-16T05:16:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 client.74328.objecter resend_mon_ops 2024-09-16T05:16:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f91785640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c02d7d0 0x7f1f8c0927d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c13d280 tx=0x7f1f8800d810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 1 -- 172.21.15.114:0/4001496777 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f88018070 con 0x7f1f8c02d7d0 2024-09-16T05:16:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f92787640 1 --2- 172.21.15.114:0/4001496777 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1f70037810 0x7f1f70039cd0 secure :-1 s=READY pgs=3526 cs=0 l=1 rev1=1 crypto rx=0x7f1f840ae000 tx=0x7f1f840a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:16:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 monclient: dump: 2024-09-16T05:16:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.418 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 client.74328.objecter ms_handle_connect 0x7f1f70037810 2024-09-16T05:16:31.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 1 -- 172.21.15.114:0/4001496777 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1f88003650 con 0x7f1f8c02d7d0 2024-09-16T05:16:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 1 -- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c02d7d0 msgr2=0x7f1f8c0927d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c02d7d0 0x7f1f8c0927d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c13d280 tx=0x7f1f8800d810 comp rx=0 tx=0).stop 2024-09-16T05:16:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 1 -- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c02d7d0 msgr2=0x7f1f8c0927d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.395+0000 7f1f827fc640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c02d7d0 0x7f1f8c0927d0 secure :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c13d280 tx=0x7f1f8800d810 comp rx=0 tx=0).stop 2024-09-16T05:16:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:16:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c062ea0 0x7f1f8c063280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 1 -- 172.21.15.114:0/4001496777 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c09c6a0 con 0x7f1f8c062ea0 2024-09-16T05:16:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f91785640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c062ea0 0x7f1f8c063280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f91785640 10 monclient: get_auth_request con 0x7f1f8c062ea0 auth_method 0 2024-09-16T05:16:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f91785640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f91785640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f91785640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f91785640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f91785640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f91785640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f91785640 10 monclient(hunting): handle_auth_done global_id 64459 payload 824 2024-09-16T05:16:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f91785640 1 -- 172.21.15.114:0/4001496777 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1f8c0a0ec0 con 0x7f1f8c062ea0 2024-09-16T05:16:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f91785640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c062ea0 0x7f1f8c063280 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c13d280 tx=0x7f1f88001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f827fc640 10 client.74328.objecter ms_handle_connect 0x7f1f8c062ea0 2024-09-16T05:16:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f827fc640 10 client.74328.objecter resend_mon_ops 2024-09-16T05:16:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f827fc640 1 -- 172.21.15.114:0/4001496777 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f88018030 con 0x7f1f8c062ea0 2024-09-16T05:16:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:16:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f827fc640 10 monclient: dump: 2024-09-16T05:16:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.423 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f827fc640 1 -- 172.21.15.114:0/4001496777 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f1f88003e40 con 0x7f1f8c062ea0 2024-09-16T05:16:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f827fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:16:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f827fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f827fc640 1 -- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c062ea0 msgr2=0x7f1f8c063280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f827fc640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c062ea0 0x7f1f8c063280 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c13d280 tx=0x7f1f88001080 comp rx=0 tx=0).stop 2024-09-16T05:16:31.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f827fc640 1 -- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c062ea0 msgr2=0x7f1f8c063280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:31.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f827fc640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1f8c062ea0 0x7f1f8c063280 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c13d280 tx=0x7f1f88001080 comp rx=0 tx=0).stop 2024-09-16T05:16:31.424 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:16:31.424 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:16:31.424 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:16:31.424 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:16:31.424 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 40, 2024-09-16T05:16:31.424 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 275536, 2024-09-16T05:16:31.424 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:16:31.424 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:16:31.425 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:16:31.425 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:16:31.425 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:31.425 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 4, 2024-09-16T05:16:31.425 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:31.425 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:16:31.425 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:16:31.425 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:31.425 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:31.425 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:31.425 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:31.425 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:31.425 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:31.425 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:31.426 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:31.426 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:31.426 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:31.426 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:31.426 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:31.426 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:16:31.426 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:16:31.426 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:31.426 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:31.426 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:31.426 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:31.426 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:31.426 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:31.426 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:31.427 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:31.427 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:31.427 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:31.427 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:31.427 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.427 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:16:31.427 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:16:31.427 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:16:31.427 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:16:31.427 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:16:31.428 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:16:31.428 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:31.428 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:31.428 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:16:31.428 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:16:31.428 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:16:31.428 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:16:31.428 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:16:31.428 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:16:31.428 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:16:31.428 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:31.428 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:16:31.428 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:31.429 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:31.429 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:31.429 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:31.429 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:31.429 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:31.429 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:31.429 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:31.429 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:31.429 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:31.429 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:31.429 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:31.429 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:16:31.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.429 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:16:31.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.430 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:16:31.430 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:16:31.430 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:31.430 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:31.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.430 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:31.430 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:16:31.430 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.431 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:31.431 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:16:31.431 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.431 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.431 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.431 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:16:31.431 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:16:31.431 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:31.431 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:31.431 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:31.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.432 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:16:31.432 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:16:31.432 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:31.432 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:31.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.432 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:31.432 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:16:31.432 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.432 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:31.432 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:16:31.432 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.432 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.433 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.433 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:16:31.433 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:16:31.433 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:31.433 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:31.433 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:31.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.433 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:16:31.433 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:16:31.433 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:31.433 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:31.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.434 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:31.434 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:16:31.434 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.434 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:31.434 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:16:31.434 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:31.434 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.434 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.434 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:16:31.434 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:16:31.434 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:31.435 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:31.435 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:31.435 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.435 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.435 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:16:31.435 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:16:31.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.435 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:31.435 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:31.435 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:16:31.435 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.435 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:31.435 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:16:31.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.436 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:31.436 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:31.436 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:16:31.436 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.436 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:31.436 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:16:31.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.436 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:31.436 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:31.436 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:16:31.436 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.436 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:31.436 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:16:31.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:31.437 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:31.437 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:31.437 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:16:31.437 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:31.437 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:31.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:31.437 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:16:31.437 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:16:31.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 10 librados: watch_flush enter 2024-09-16T05:16:31.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 10 librados: watch_flush exit 2024-09-16T05:16:31.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 20 client.74328.objecter shutdown clearing up homeless session... 2024-09-16T05:16:31.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 10 client.74328.objecter successfully canceled tick 2024-09-16T05:16:31.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 1 -- 172.21.15.114:0/4001496777 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1f70037810 msgr2=0x7f1f70039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 1 --2- 172.21.15.114:0/4001496777 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1f70037810 0x7f1f70039cd0 secure :-1 s=READY pgs=3526 cs=0 l=1 rev1=1 crypto rx=0x7f1f840ae000 tx=0x7f1f840a7070 comp rx=0 tx=0).stop 2024-09-16T05:16:31.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 10 monclient: shutdown 2024-09-16T05:16:31.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:31.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 1 -- 172.21.15.114:0/4001496777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f8c09b6c0 msgr2=0x7f1f8c13ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f92787640 1 -- 172.21.15.114:0/4001496777 reap_dead start 2024-09-16T05:16:31.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f8c09b6c0 0x7f1f8c13ca30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00e980 tx=0x7f1f7c00ee40 comp rx=0 tx=0).stop 2024-09-16T05:16:31.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 1 -- 172.21.15.114:0/4001496777 shutdown_connections 2024-09-16T05:16:31.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 1 --2- 172.21.15.114:0/4001496777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f8c09b6c0 0x7f1f8c13ca30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 1 -- 172.21.15.114:0/4001496777 >> 172.21.15.114:0/4001496777 conn(0x7f1f8c0a1fb0 msgr2=0x7f1f8c0a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:31.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 1 -- 172.21.15.114:0/4001496777 shutdown_connections 2024-09-16T05:16:31.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 1 -- 172.21.15.114:0/4001496777 wait complete. 2024-09-16T05:16:31.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.403+0000 7f1f941ea640 1 librados: shutdown 2024-09-16T05:16:31.440 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:16:31.441 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T05:16:31.441 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T05:16:31.441 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T05:16:31.441 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T05:16:31.441 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T05:16:31.441 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:16:31.451 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T05:16:31.451 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:16:31.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:16:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 10 monclient: get_monmap_and_config 2024-09-16T05:16:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 10 monclient: build_initial_monmap 2024-09-16T05:16:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 10 monclient: monmap: 2024-09-16T05:16:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:31.609625+0000 2024-09-16T05:16:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:31.609625+0000 2024-09-16T05:16:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:31.607 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:31.607 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:31.607 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:31.607 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 1 Processor -- start 2024-09-16T05:16:31.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 1 -- start start 2024-09-16T05:16:31.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 10 monclient: init 2024-09-16T05:16:31.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:31.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-16T05:16:31.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f32fc0a0ec0 0x7f32fc09dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 10 monclient(hunting): picked mon.noname-a con 0x7f32fc0a0ec0 addr v2:172.21.15.114:3300/0 2024-09-16T05:16:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f32fc09e000 0x7f32fc09e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 10 monclient(hunting): picked mon.noname-c con 0x7f32fc09e000 addr v2:172.21.15.149:3300/0 2024-09-16T05:16:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32fc138630 0x7f32fc13aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 10 monclient(hunting): picked mon.noname-b con 0x7f32fc138630 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f32fc0a12d0 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f32fc0994b0 con 0x7f32fc09e000 2024-09-16T05:16:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f32fc09e950 con 0x7f32fc138630 2024-09-16T05:16:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f3300d5f640 10 monclient(hunting): authenticate will time out at 1476.238771s 2024-09-16T05:16:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fad9d640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32fc138630 msgr2=0x7f32fc13aa20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:16:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fad9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32fc138630 0x7f32fc13aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f32fc0a0ec0 0x7f32fc09dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f32fc0a0ec0 0x7f32fc09dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32952/0 (socket says 172.21.15.114:32952) 2024-09-16T05:16:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 1 -- 172.21.15.114:0/3443747373 learned_addr learned my addr 172.21.15.114:0/3443747373 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 10 monclient(hunting): get_auth_request con 0x7f32fc0a0ec0 auth_method 0 2024-09-16T05:16:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32f1d9b640 1 --2- 172.21.15.114:0/3443747373 >> v2:172.21.15.149:3300/0 conn(0x7f32fc09e000 0x7f32fc09e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32f1d9b640 10 monclient(hunting): get_auth_request con 0x7f32fc09e000 auth_method 0 2024-09-16T05:16:31.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 10 monclient(hunting): handle_auth_done global_id 74331 payload 274 2024-09-16T05:16:31.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 1 -- 172.21.15.114:0/3443747373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32fc138630 msgr2=0x7f32fc13aa20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:16:31.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 1 --2- 172.21.15.114:0/3443747373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32fc138630 0x7f32fc13aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 1 -- 172.21.15.114:0/3443747373 >> v2:172.21.15.149:3300/0 conn(0x7f32fc09e000 msgr2=0x7f32fc09e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:31.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 1 --2- 172.21.15.114:0/3443747373 >> v2:172.21.15.149:3300/0 conn(0x7f32fc09e000 0x7f32fc09e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 10 monclient: _finish_hunting 0 2024-09-16T05:16:31.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 1 monclient: found mon.noname-a 2024-09-16T05:16:31.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:31.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:31.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.607+0000 7f32fa59c640 1 -- 172.21.15.114:0/3443747373 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32fc09a760 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f32fa59c640 1 --2- 172.21.15.114:0/3443747373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32fc0a0ec0 0x7f32fc09dac0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f32fc099150 tx=0x7f32ec01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1431d3a955b741dd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f32f9d9b640 1 -- 172.21.15.114:0/3443747373 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec0b0070 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f32f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f32f9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:31.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f32f9d9b640 10 monclient: dump: 2024-09-16T05:16:31.618 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.618 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.618 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.618 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.618 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.618 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.618 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.618 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.618 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.618 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f32f9d9b640 10 monclient: _finish_auth 0 2024-09-16T05:16:31.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f32f9d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:16:31.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f32f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:31.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f32f9d9b640 1 -- 172.21.15.114:0/3443747373 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32ec0ab600 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 5 monclient: authenticate success, global_id 74331 2024-09-16T05:16:31.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:16:31.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f32f9d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:31.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f32f9d9b640 1 -- 172.21.15.114:0/3443747373 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec0ab9d0 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:31.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 4 set_mon_vals no callback set 2024-09-16T05:16:31.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f32f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f32f9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:31.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f32f9d9b640 10 monclient: dump: 2024-09-16T05:16:31.619 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.619 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.619 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.620 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.620 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.620 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.620 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.620 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.620 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.620 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 10 monclient: shutdown 2024-09-16T05:16:31.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:31.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 -- 172.21.15.114:0/3443747373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32fc0a0ec0 msgr2=0x7f32fc09dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 --2- 172.21.15.114:0/3443747373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32fc0a0ec0 0x7f32fc09dac0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f32fc099150 tx=0x7f32ec01c490 comp rx=0 tx=0).stop 2024-09-16T05:16:31.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 -- 172.21.15.114:0/3443747373 shutdown_connections 2024-09-16T05:16:31.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 --2- 172.21.15.114:0/3443747373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32fc138630 0x7f32fc13aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 --2- 172.21.15.114:0/3443747373 >> v2:172.21.15.149:3300/0 conn(0x7f32fc09e000 0x7f32fc09e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 --2- 172.21.15.114:0/3443747373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32fc0a0ec0 0x7f32fc09dac0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 -- 172.21.15.114:0/3443747373 >> 172.21.15.114:0/3443747373 conn(0x7f32fc0a1fb0 msgr2=0x7f32fc0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:31.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 -- 172.21.15.114:0/3443747373 shutdown_connections 2024-09-16T05:16:31.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 -- 172.21.15.114:0/3443747373 wait complete. 2024-09-16T05:16:31.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 10 monclient: build_initial_monmap 2024-09-16T05:16:31.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 10 monclient: monmap: 2024-09-16T05:16:31.621 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:31.621 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:31.621 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:31.621 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:31.621 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:31.621 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.621 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:31.622 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:31.622 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:31.622 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 librados: starting msgr at 2024-09-16T05:16:31.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 librados: starting objecter 2024-09-16T05:16:31.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 Processor -- start 2024-09-16T05:16:31.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 -- start start 2024-09-16T05:16:31.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 librados: setting wanted keys 2024-09-16T05:16:31.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 librados: calling monclient init 2024-09-16T05:16:31.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 10 monclient: init 2024-09-16T05:16:31.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:31.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:16:31.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32fc09e000 0x7f32fc1cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 10 monclient(hunting): picked mon.noname-c con 0x7f32fc09e000 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:31.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32fc0a0ec0 0x7f32fc1cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 10 monclient(hunting): picked mon.noname-a con 0x7f32fc0a0ec0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:31.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f32fc138630 0x7f32fc0704c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 10 monclient(hunting): picked mon.noname-b con 0x7f32fc138630 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:31.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f32fc09c3b0 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f32fc09c230 con 0x7f32fc138630 2024-09-16T05:16:31.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:31.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f32fc09c9b0 con 0x7f32fc09e000 2024-09-16T05:16:31.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:31.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f32fa59c640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32fc09e000 msgr2=0x7f32fc1cf7c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:16:31.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.611+0000 7f3300d5f640 10 monclient(hunting): authenticate will time out at 1476.243985s 2024-09-16T05:16:31.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32fa59c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32fc09e000 0x7f32fc1cf7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:31.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32fc0a0ec0 0x7f32fc1cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32fc0a0ec0 0x7f32fc1cfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32964/0 (socket says 172.21.15.114:32964) 2024-09-16T05:16:31.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 1 -- 172.21.15.114:0/2291683542 learned_addr learned my addr 172.21.15.114:0/2291683542 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:31.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 10 monclient(hunting): get_auth_request con 0x7f32fc0a0ec0 auth_method 0 2024-09-16T05:16:31.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:31.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:31.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:31.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:31.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:31.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:31.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 10 monclient(hunting): handle_auth_done global_id 74334 payload 824 2024-09-16T05:16:31.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 1 -- 172.21.15.114:0/2291683542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32fc09e000 msgr2=0x7f32fc1cf7c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:16:31.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 1 --2- 172.21.15.114:0/2291683542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32fc09e000 0x7f32fc1cf7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 1 -- 172.21.15.114:0/2291683542 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f32fc138630 msgr2=0x7f32fc0704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 1 --2- 172.21.15.114:0/2291683542 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f32fc138630 0x7f32fc0704c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:16:31.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 1 monclient: found mon.noname-a 2024-09-16T05:16:31.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:31.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:31.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 1 -- 172.21.15.114:0/2291683542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32fc12ec60 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f1d9b640 1 --2- 172.21.15.114:0/2291683542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32fc0a0ec0 0x7f32fc1cfd00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f32e400d7d0 tx=0x7f32e400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 10 client.?.objecter ms_handle_connect 0x7f32fc0a0ec0 2024-09-16T05:16:31.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:31.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 1 -- 172.21.15.114:0/2291683542 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e400b6c0 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 10 monclient: dump: 2024-09-16T05:16:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.626 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 10 monclient: _finish_auth 0 2024-09-16T05:16:31.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:16:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 1 -- 172.21.15.114:0/2291683542 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32e4019070 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f3300d5f640 5 monclient: authenticate success, global_id 74334 2024-09-16T05:16:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f3300d5f640 10 monclient: _renew_subs 2024-09-16T05:16:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f3300d5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f3300d5f640 1 -- 172.21.15.114:0/2291683542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32fc12ef40 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 1 -- 172.21.15.114:0/2291683542 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e400baa0 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 10 monclient: dump: 2024-09-16T05:16:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:31.628 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.628 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:31.628 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:31.628 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:31.628 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:31.628 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:31.628 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:31.628 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:31.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f959a640 4 set_mon_vals no callback set 2024-09-16T05:16:31.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f3300d5f640 10 client.74334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:31.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f3300d5f640 10 monclient: _renew_subs 2024-09-16T05:16:31.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f3300d5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:31.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f3300d5f640 1 -- 172.21.15.114:0/2291683542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32fc0a3fd0 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f3300d5f640 1 librados: init done 2024-09-16T05:16:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f3300d5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f3300d5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 1 -- 172.21.15.114:0/2291683542 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f32e400bc40 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f3300d5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f3300d5f640 1 -- 172.21.15.114:0/2291683542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32fc09e410 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 1 --2- 172.21.15.114:0/2291683542 >> v2:172.21.15.149:6812/1185531135 conn(0x7f32dc0377c0 0x7f32dc039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 1 -- 172.21.15.114:0/2291683542 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(357..357 src has 1..357) ==== 5406+0+0 (secure 0 0 0) 0x7f32e40491d0 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 10 client.74334.objecter ms_dispatch 0x7f32fc000ba0 osd_map(357..357 src has 1..357) 2024-09-16T05:16:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 3 client.74334.objecter handle_osd_map got epochs [357,357] > 0 2024-09-16T05:16:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 3 client.74334.objecter handle_osd_map decoding full epoch 357 2024-09-16T05:16:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 20 client.74334.objecter dump_active .. 0 homeless 2024-09-16T05:16:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32fa59c640 1 --2- 172.21.15.114:0/2291683542 >> v2:172.21.15.149:6812/1185531135 conn(0x7f32dc0377c0 0x7f32dc039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32fa59c640 10 monclient: get_auth_request con 0x7f32dc0377c0 auth_method 0 2024-09-16T05:16:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32f37fe640 10 client.74334.objecter ms_handle_connect 0x7f32dc0377c0 2024-09-16T05:16:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.615+0000 7f32fa59c640 1 --2- 172.21.15.114:0/2291683542 >> v2:172.21.15.149:6812/1185531135 conn(0x7f32dc0377c0 0x7f32dc039c80 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 crypto rx=0x7f32fc0a2fd0 tx=0x7f32ec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.619+0000 7f32f37fe640 1 -- 172.21.15.114:0/2291683542 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f32e401e840 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.619+0000 7f32f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.619+0000 7f32f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.847+0000 7f3300d5f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:16:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.847+0000 7f3300d5f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.847+0000 7f3300d5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.847+0000 7f3300d5f640 1 -- 172.21.15.114:0/2291683542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f32fc0726d0 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.847+0000 7f32f37fe640 1 -- 172.21.15.114:0/2291683542 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f32e401e460 con 0x7f32fc0a0ec0 2024-09-16T05:16:31.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.847+0000 7f32f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:31.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.847+0000 7f32f37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:31.847 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:16:31.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 10 librados: watch_flush enter 2024-09-16T05:16:31.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 10 librados: watch_flush exit 2024-09-16T05:16:31.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 20 client.74334.objecter shutdown clearing up homeless session... 2024-09-16T05:16:31.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 10 client.74334.objecter successfully canceled tick 2024-09-16T05:16:31.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 1 -- 172.21.15.114:0/2291683542 >> v2:172.21.15.149:6812/1185531135 conn(0x7f32dc0377c0 msgr2=0x7f32dc039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 1 --2- 172.21.15.114:0/2291683542 >> v2:172.21.15.149:6812/1185531135 conn(0x7f32dc0377c0 0x7f32dc039c80 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 crypto rx=0x7f32fc0a2fd0 tx=0x7f32ec008040 comp rx=0 tx=0).stop 2024-09-16T05:16:31.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 10 monclient: shutdown 2024-09-16T05:16:31.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:31.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 1 -- 172.21.15.114:0/2291683542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32fc0a0ec0 msgr2=0x7f32fc1cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 1 --2- 172.21.15.114:0/2291683542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32fc0a0ec0 0x7f32fc1cfd00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f32e400d7d0 tx=0x7f32e400dc90 comp rx=0 tx=0).stop 2024-09-16T05:16:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 1 -- 172.21.15.114:0/2291683542 shutdown_connections 2024-09-16T05:16:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 1 --2- 172.21.15.114:0/2291683542 >> v2:172.21.15.149:6812/1185531135 conn(0x7f32dc0377c0 0x7f32dc039c80 unknown :-1 s=CLOSED pgs=3527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 1 --2- 172.21.15.114:0/2291683542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32fc0a0ec0 0x7f32fc1cfd00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 1 --2- 172.21.15.114:0/2291683542 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f32fc138630 0x7f32fc0704c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 1 --2- 172.21.15.114:0/2291683542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32fc09e000 0x7f32fc1cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 1 -- 172.21.15.114:0/2291683542 >> 172.21.15.114:0/2291683542 conn(0x7f32fc0a1fb0 msgr2=0x7f32fc137d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 1 -- 172.21.15.114:0/2291683542 shutdown_connections 2024-09-16T05:16:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 1 -- 172.21.15.114:0/2291683542 wait complete. 2024-09-16T05:16:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:31.851+0000 7f3300d5f640 1 librados: shutdown 2024-09-16T05:16:34.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:16:34.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 10 monclient: get_monmap_and_config 2024-09-16T05:16:34.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 10 monclient: build_initial_monmap 2024-09-16T05:16:34.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 10 monclient: monmap: 2024-09-16T05:16:34.994 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:34.994 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:34.995 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:34.997968+0000 2024-09-16T05:16:34.995 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:34.997968+0000 2024-09-16T05:16:34.995 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:34.995 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:34.995 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:34.995 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:34.995 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:34.995 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:34.995 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:34.995 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 1 Processor -- start 2024-09-16T05:16:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 1 -- start start 2024-09-16T05:16:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 10 monclient: init 2024-09-16T05:16:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-16T05:16:34.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fde1409b6c0 0x7fde140a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:34.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 10 monclient(hunting): picked mon.noname-a con 0x7fde1409b6c0 addr v2:172.21.15.114:3300/0 2024-09-16T05:16:34.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fde140a1400 0x7fde140a1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:34.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 10 monclient(hunting): picked mon.noname-c con 0x7fde140a1400 addr v2:172.21.15.149:3300/0 2024-09-16T05:16:34.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde14139640 0x7fde1413ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:34.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 10 monclient(hunting): picked mon.noname-b con 0x7fde14139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:34.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:34.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fde1409e390 con 0x7fde1409b6c0 2024-09-16T05:16:34.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:34.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fde140a1d50 con 0x7fde140a1400 2024-09-16T05:16:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fde1409ece0 con 0x7fde14139640 2024-09-16T05:16:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde18bdf640 10 monclient(hunting): authenticate will time out at 1479.627004s 2024-09-16T05:16:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fde1409b6c0 0x7fde140a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde12d9d640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde14139640 msgr2=0x7fde1413ba30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:16:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fde1409b6c0 0x7fde140a0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32974/0 (socket says 172.21.15.114:32974) 2024-09-16T05:16:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 1 -- 172.21.15.114:0/1973897487 learned_addr learned my addr 172.21.15.114:0/1973897487 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 10 monclient(hunting): get_auth_request con 0x7fde1409b6c0 auth_method 0 2024-09-16T05:16:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde12d9d640 1 --2- 172.21.15.114:0/1973897487 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde14139640 0x7fde1413ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:35.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:35.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:35.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:35.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 10 monclient(hunting): handle_auth_done global_id 74337 payload 274 2024-09-16T05:16:35.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 1 -- 172.21.15.114:0/1973897487 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde14139640 msgr2=0x7fde1413ba30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:16:35.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 1 --2- 172.21.15.114:0/1973897487 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde14139640 0x7fde1413ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:35.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 1 -- 172.21.15.114:0/1973897487 >> v2:172.21.15.149:3300/0 conn(0x7fde140a1400 msgr2=0x7fde140a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:35.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 1 --2- 172.21.15.114:0/1973897487 >> v2:172.21.15.149:3300/0 conn(0x7fde140a1400 0x7fde140a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:35.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 10 monclient: _finish_hunting 0 2024-09-16T05:16:35.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 1 monclient: found mon.noname-a 2024-09-16T05:16:35.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:35.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:35.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.995+0000 7fde1259c640 1 -- 172.21.15.114:0/1973897487 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1409d4e0 con 0x7fde1409b6c0 2024-09-16T05:16:35.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde1259c640 1 --2- 172.21.15.114:0/1973897487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde1409b6c0 0x7fde140a0ec0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fde1409c630 tx=0x7fde0801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d87e9b915ed177 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:35.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde1159a640 1 -- 172.21.15.114:0/1973897487 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde080b0070 con 0x7fde1409b6c0 2024-09-16T05:16:35.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:35.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde1159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:35.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde1159a640 10 monclient: dump: 2024-09-16T05:16:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:35.007 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde1159a640 10 monclient: _finish_auth 0 2024-09-16T05:16:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde1159a640 10 monclient: _check_auth_tickets 2024-09-16T05:16:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:35.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 5 monclient: authenticate success, global_id 74337 2024-09-16T05:16:35.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:16:35.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde1159a640 1 -- 172.21.15.114:0/1973897487 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde080a4e50 con 0x7fde1409b6c0 2024-09-16T05:16:35.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde1159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:35.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:35.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 4 set_mon_vals no callback set 2024-09-16T05:16:35.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde1159a640 1 -- 172.21.15.114:0/1973897487 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde080ab820 con 0x7fde1409b6c0 2024-09-16T05:16:35.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:35.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde1159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:35.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde1159a640 10 monclient: dump: 2024-09-16T05:16:35.008 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:35.008 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:35.008 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:35.009 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 10 monclient: shutdown 2024-09-16T05:16:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 1 -- 172.21.15.114:0/1973897487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde1409b6c0 msgr2=0x7fde140a0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 1 --2- 172.21.15.114:0/1973897487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde1409b6c0 0x7fde140a0ec0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fde1409c630 tx=0x7fde0801c400 comp rx=0 tx=0).stop 2024-09-16T05:16:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 1 -- 172.21.15.114:0/1973897487 shutdown_connections 2024-09-16T05:16:35.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 1 --2- 172.21.15.114:0/1973897487 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde14139640 0x7fde1413ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 1 --2- 172.21.15.114:0/1973897487 >> v2:172.21.15.149:3300/0 conn(0x7fde140a1400 0x7fde140a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 1 --2- 172.21.15.114:0/1973897487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde1409b6c0 0x7fde140a0ec0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 1 -- 172.21.15.114:0/1973897487 >> 172.21.15.114:0/1973897487 conn(0x7fde140a1fb0 msgr2=0x7fde140a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 1 -- 172.21.15.114:0/1973897487 shutdown_connections 2024-09-16T05:16:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 1 -- 172.21.15.114:0/1973897487 wait complete. 2024-09-16T05:16:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 10 monclient: build_initial_monmap 2024-09-16T05:16:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 10 monclient: monmap: 2024-09-16T05:16:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:35.011 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 1 librados: starting msgr at 2024-09-16T05:16:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 1 librados: starting objecter 2024-09-16T05:16:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 1 Processor -- start 2024-09-16T05:16:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 1 -- start start 2024-09-16T05:16:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 1 librados: setting wanted keys 2024-09-16T05:16:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 1 librados: calling monclient init 2024-09-16T05:16:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:34.999+0000 7fde18bdf640 10 monclient: init 2024-09-16T05:16:35.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:16:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde1409b6c0 0x7fde14096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient(hunting): picked mon.noname-c con 0x7fde1409b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fde140a1400 0x7fde14071400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient(hunting): picked mon.noname-b con 0x7fde140a1400 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde14139640 0x7fde14071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient(hunting): picked mon.noname-a con 0x7fde14139640 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fde1413e790 con 0x7fde14139640 2024-09-16T05:16:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fde1409cad0 con 0x7fde140a1400 2024-09-16T05:16:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:35.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fde1413e910 con 0x7fde1409b6c0 2024-09-16T05:16:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde1259c640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde1409b6c0 msgr2=0x7fde14096a60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:16:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient(hunting): authenticate will time out at 1479.632254s 2024-09-16T05:16:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde1259c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde1409b6c0 0x7fde14096a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde14139640 0x7fde14071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde14139640 0x7fde14071940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32976/0 (socket says 172.21.15.114:32976) 2024-09-16T05:16:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 1 -- 172.21.15.114:0/2385788789 learned_addr learned my addr 172.21.15.114:0/2385788789 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 10 monclient(hunting): get_auth_request con 0x7fde14139640 auth_method 0 2024-09-16T05:16:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:35.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:35.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 10 monclient(hunting): handle_auth_done global_id 74340 payload 824 2024-09-16T05:16:35.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 1 -- 172.21.15.114:0/2385788789 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde1409b6c0 msgr2=0x7fde14096a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:16:35.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 1 --2- 172.21.15.114:0/2385788789 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde1409b6c0 0x7fde14096a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:35.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 1 -- 172.21.15.114:0/2385788789 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fde140a1400 msgr2=0x7fde14071400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:35.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 1 --2- 172.21.15.114:0/2385788789 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fde140a1400 0x7fde14071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:35.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 10 monclient: _finish_hunting 0 2024-09-16T05:16:35.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 1 monclient: found mon.noname-a 2024-09-16T05:16:35.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:35.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:35.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 1 -- 172.21.15.114:0/2385788789 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1409c6a0 con 0x7fde14139640 2024-09-16T05:16:35.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde12d9d640 1 --2- 172.21.15.114:0/2385788789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde14139640 0x7fde14071940 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00be60 tx=0x7fde0c00bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:35.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 10 client.?.objecter ms_handle_connect 0x7fde14139640 2024-09-16T05:16:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 1 -- 172.21.15.114:0/2385788789 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c00cb50 con 0x7fde14139640 2024-09-16T05:16:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 10 monclient: dump: 2024-09-16T05:16:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:35.015 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:35.016 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:35.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 10 monclient: _finish_auth 0 2024-09-16T05:16:35.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:16:35.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:35.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 1 -- 172.21.15.114:0/2385788789 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde0c010030 con 0x7fde14139640 2024-09-16T05:16:35.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 5 monclient: authenticate success, global_id 74340 2024-09-16T05:16:35.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient: _renew_subs 2024-09-16T05:16:35.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:35.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 1 -- 172.21.15.114:0/2385788789 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde14096240 con 0x7fde14139640 2024-09-16T05:16:35.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:35.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 1 -- 172.21.15.114:0/2385788789 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c016430 con 0x7fde14139640 2024-09-16T05:16:35.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:35.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 10 monclient: dump: 2024-09-16T05:16:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:35.017 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 client.74340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient: _renew_subs 2024-09-16T05:16:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde10d99640 4 set_mon_vals no callback set 2024-09-16T05:16:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 1 -- 172.21.15.114:0/2385788789 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde140735f0 con 0x7fde14139640 2024-09-16T05:16:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 1 librados: init done 2024-09-16T05:16:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde18bdf640 1 -- 172.21.15.114:0/2385788789 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde140a0ec0 con 0x7fde14139640 2024-09-16T05:16:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 1 -- 172.21.15.114:0/2385788789 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fde0c008070 con 0x7fde14139640 2024-09-16T05:16:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 1 --2- 172.21.15.114:0/2385788789 >> v2:172.21.15.149:6812/1185531135 conn(0x7fddf40377f0 0x7fddf4039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 1 -- 172.21.15.114:0/2385788789 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(357..357 src has 1..357) ==== 5406+0+0 (secure 0 0 0) 0x7fde0c048b50 con 0x7fde14139640 2024-09-16T05:16:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 10 client.74340.objecter ms_dispatch 0x7fde14000ba0 osd_map(357..357 src has 1..357) 2024-09-16T05:16:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 3 client.74340.objecter handle_osd_map got epochs [357,357] > 0 2024-09-16T05:16:35.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 3 client.74340.objecter handle_osd_map decoding full epoch 357 2024-09-16T05:16:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 20 client.74340.objecter dump_active .. 0 homeless 2024-09-16T05:16:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde1259c640 1 --2- 172.21.15.114:0/2385788789 >> v2:172.21.15.149:6812/1185531135 conn(0x7fddf40377f0 0x7fddf4039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde1259c640 10 monclient: get_auth_request con 0x7fddf40377f0 auth_method 0 2024-09-16T05:16:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde02ffd640 10 client.74340.objecter ms_handle_connect 0x7fddf40377f0 2024-09-16T05:16:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.003+0000 7fde1259c640 1 --2- 172.21.15.114:0/2385788789 >> v2:172.21.15.149:6812/1185531135 conn(0x7fddf40377f0 0x7fddf4039cb0 secure :-1 s=READY pgs=3528 cs=0 l=1 rev1=1 crypto rx=0x7fde080ad000 tx=0x7fde08008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.007+0000 7fde02ffd640 1 -- 172.21.15.114:0/2385788789 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fde0c01d300 con 0x7fde14139640 2024-09-16T05:16:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.007+0000 7fde02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:35.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.007+0000 7fde02ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:35.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:16:35.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:35.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:35.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 1 -- 172.21.15.114:0/2385788789 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fde14139a90 con 0x7fde14139640 2024-09-16T05:16:35.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde02ffd640 1 -- 172.21.15.114:0/2385788789 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fde0c01eda0 con 0x7fde14139640 2024-09-16T05:16:35.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:35.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde02ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:35.233 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:16:35.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 10 librados: watch_flush enter 2024-09-16T05:16:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 10 librados: watch_flush exit 2024-09-16T05:16:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 20 client.74340.objecter shutdown clearing up homeless session... 2024-09-16T05:16:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 10 client.74340.objecter successfully canceled tick 2024-09-16T05:16:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 1 -- 172.21.15.114:0/2385788789 >> v2:172.21.15.149:6812/1185531135 conn(0x7fddf40377f0 msgr2=0x7fddf4039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 1 --2- 172.21.15.114:0/2385788789 >> v2:172.21.15.149:6812/1185531135 conn(0x7fddf40377f0 0x7fddf4039cb0 secure :-1 s=READY pgs=3528 cs=0 l=1 rev1=1 crypto rx=0x7fde080ad000 tx=0x7fde08008040 comp rx=0 tx=0).stop 2024-09-16T05:16:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 10 monclient: shutdown 2024-09-16T05:16:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 1 -- 172.21.15.114:0/2385788789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde14139640 msgr2=0x7fde14071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 1 --2- 172.21.15.114:0/2385788789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde14139640 0x7fde14071940 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00be60 tx=0x7fde0c00bf50 comp rx=0 tx=0).stop 2024-09-16T05:16:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 1 -- 172.21.15.114:0/2385788789 shutdown_connections 2024-09-16T05:16:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 1 --2- 172.21.15.114:0/2385788789 >> v2:172.21.15.149:6812/1185531135 conn(0x7fddf40377f0 0x7fddf4039cb0 unknown :-1 s=CLOSED pgs=3528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 1 --2- 172.21.15.114:0/2385788789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde14139640 0x7fde14071940 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:35.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 1 --2- 172.21.15.114:0/2385788789 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fde140a1400 0x7fde14071400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:35.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 1 --2- 172.21.15.114:0/2385788789 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fde1409b6c0 0x7fde14096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:35.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 1 -- 172.21.15.114:0/2385788789 >> 172.21.15.114:0/2385788789 conn(0x7fde140a1fb0 msgr2=0x7fde140a4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:35.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 1 -- 172.21.15.114:0/2385788789 shutdown_connections 2024-09-16T05:16:35.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 1 -- 172.21.15.114:0/2385788789 wait complete. 2024-09-16T05:16:35.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:35.235+0000 7fde18bdf640 1 librados: shutdown 2024-09-16T05:16:38.253 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:16:38.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 10 monclient: get_monmap_and_config 2024-09-16T05:16:38.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 10 monclient: build_initial_monmap 2024-09-16T05:16:38.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 10 monclient: monmap: 2024-09-16T05:16:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:38.382333+0000 2024-09-16T05:16:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:38.382333+0000 2024-09-16T05:16:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:38.379 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:38.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 1 Processor -- start 2024-09-16T05:16:38.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 1 -- start start 2024-09-16T05:16:38.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 10 monclient: init 2024-09-16T05:16:38.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:38.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-16T05:16:38.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62d809b6c0 addr v1:172.21.15.149:6789/0 2024-09-16T05:16:38.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62d80a15d0 addr v1:172.21.15.114:6789/0 2024-09-16T05:16:38.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f62d80a19e0 0x7f62d813dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:38.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 10 monclient(hunting): picked mon.noname-a con 0x7f62d80a19e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:16:38.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62d809d4e0 con 0x7f62d80a15d0 2024-09-16T05:16:38.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62d809f430 con 0x7f62d809b6c0 2024-09-16T05:16:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f62d809e390 con 0x7f62d80a19e0 2024-09-16T05:16:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62df8ab640 10 monclient(hunting): authenticate will time out at 1483.011523s 2024-09-16T05:16:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62dce46640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f62d80a15d0 0x7f62d813a650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50516/0 (socket says 172.21.15.114:50516) 2024-09-16T05:16:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62dce46640 1 -- 172.21.15.114:0/3672741947 learned_addr learned my addr 172.21.15.114:0/3672741947 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:16:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62dde48640 1 --2- 172.21.15.114:0/3672741947 >> v2:172.21.15.114:3300/0 conn(0x7f62d80a19e0 0x7f62d813dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62dde48640 10 monclient(hunting): get_auth_request con 0x7f62d80a19e0 auth_method 0 2024-09-16T05:16:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62dde48640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62dde48640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62dde48640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62dde48640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62dde48640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:38.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62dde48640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62dde48640 10 monclient(hunting): handle_auth_done global_id 74343 payload 274 2024-09-16T05:16:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.379+0000 7f62dde48640 1 -- 172.21.15.114:0/3672741947 >> v1:172.21.15.149:6789/0 conn(0x7f62d809b6c0 legacy=0x7f62d80a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62dde48640 1 -- 172.21.15.114:0/3672741947 >> v1:172.21.15.114:6789/0 conn(0x7f62d80a15d0 legacy=0x7f62d813a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62dde48640 10 monclient: _finish_hunting 0 2024-09-16T05:16:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62dde48640 1 monclient: found mon.noname-a 2024-09-16T05:16:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62dde48640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62dde48640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62dde48640 1 -- 172.21.15.114:0/3672741947 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d809c2f0 con 0x7f62d80a19e0 2024-09-16T05:16:38.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62dde48640 1 --2- 172.21.15.114:0/3672741947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62d80a19e0 0x7f62d813dd80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f62d809d220 tx=0x7f62d001c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ff20d1478c12f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:38.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62cffff640 1 -- 172.21.15.114:0/3672741947 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d00a6710 con 0x7f62d80a19e0 2024-09-16T05:16:38.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:38.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:38.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62cffff640 10 monclient: dump: 2024-09-16T05:16:38.388 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:38.388 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:38.389 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62cffff640 10 monclient: _finish_auth 0 2024-09-16T05:16:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62cffff640 10 monclient: _check_auth_tickets 2024-09-16T05:16:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 5 monclient: authenticate success, global_id 74343 2024-09-16T05:16:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:16:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62cffff640 1 -- 172.21.15.114:0/3672741947 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62d00a68b0 con 0x7f62d80a19e0 2024-09-16T05:16:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62cffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62cffff640 1 -- 172.21.15.114:0/3672741947 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d00a6c80 con 0x7f62d80a19e0 2024-09-16T05:16:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62cffff640 10 monclient: dump: 2024-09-16T05:16:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:38.391 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:38.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:38.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 4 set_mon_vals no callback set 2024-09-16T05:16:38.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 10 monclient: shutdown 2024-09-16T05:16:38.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:38.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 1 -- 172.21.15.114:0/3672741947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62d80a19e0 msgr2=0x7f62d813dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:38.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 1 --2- 172.21.15.114:0/3672741947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62d80a19e0 0x7f62d813dd80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f62d809d220 tx=0x7f62d001c840 comp rx=0 tx=0).stop 2024-09-16T05:16:38.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 1 -- 172.21.15.114:0/3672741947 shutdown_connections 2024-09-16T05:16:38.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 1 --2- 172.21.15.114:0/3672741947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62d80a19e0 0x7f62d813dd80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:38.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 1 -- 172.21.15.114:0/3672741947 >> 172.21.15.114:0/3672741947 conn(0x7f62d80a1fb0 msgr2=0x7f62d80a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:38.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 1 -- 172.21.15.114:0/3672741947 shutdown_connections 2024-09-16T05:16:38.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 1 -- 172.21.15.114:0/3672741947 wait complete. 2024-09-16T05:16:38.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 10 monclient: build_initial_monmap 2024-09-16T05:16:38.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 10 monclient: monmap: 2024-09-16T05:16:38.392 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:38.392 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:38.392 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:38.392 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:38.392 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:38.392 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:38.392 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:38.392 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:38.392 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:38.392 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:38.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 1 librados: starting msgr at 2024-09-16T05:16:38.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.383+0000 7f62df8ab640 1 librados: starting objecter 2024-09-16T05:16:38.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 1 Processor -- start 2024-09-16T05:16:38.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 1 -- start start 2024-09-16T05:16:38.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 1 librados: setting wanted keys 2024-09-16T05:16:38.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 1 librados: calling monclient init 2024-09-16T05:16:38.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient: init 2024-09-16T05:16:38.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:38.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:16:38.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62d809b6c0 0x7f62d8098430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:38.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient(hunting): picked mon.noname-a con 0x7f62d809b6c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:38.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f62d80a15d0 0x7f62d8091b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:38.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient(hunting): picked mon.noname-b con 0x7f62d80a15d0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:38.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62d80a19e0 0x7f62d80920c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:38.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient(hunting): picked mon.noname-c con 0x7f62d80a19e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:38.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:38.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f62d8140c60 con 0x7f62d809b6c0 2024-09-16T05:16:38.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:38.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f62d8140ae0 con 0x7f62d80a15d0 2024-09-16T05:16:38.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:38.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f62d8140de0 con 0x7f62d80a19e0 2024-09-16T05:16:38.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:38.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient(hunting): authenticate will time out at 1483.016653s 2024-09-16T05:16:38.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dde48640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62d80a19e0 msgr2=0x7f62d80920c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:16:38.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dde48640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62d80a19e0 0x7f62d80920c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:38.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62d809b6c0 0x7f62d8098430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:38.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62d809b6c0 0x7f62d8098430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49922/0 (socket says 172.21.15.114:49922) 2024-09-16T05:16:38.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 1 -- 172.21.15.114:0/4107945951 learned_addr learned my addr 172.21.15.114:0/4107945951 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:38.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 10 monclient(hunting): get_auth_request con 0x7f62d809b6c0 auth_method 0 2024-09-16T05:16:38.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:38.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:38.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 10 monclient(hunting): handle_auth_done global_id 74346 payload 824 2024-09-16T05:16:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 1 -- 172.21.15.114:0/4107945951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62d80a19e0 msgr2=0x7f62d80920c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:16:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 1 --2- 172.21.15.114:0/4107945951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62d80a19e0 0x7f62d80920c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 1 -- 172.21.15.114:0/4107945951 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f62d80a15d0 msgr2=0x7f62d8091b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 1 --2- 172.21.15.114:0/4107945951 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f62d80a15d0 0x7f62d8091b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 10 monclient: _finish_hunting 0 2024-09-16T05:16:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 1 monclient: found mon.noname-a 2024-09-16T05:16:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 1 -- 172.21.15.114:0/4107945951 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d8095830 con 0x7f62d809b6c0 2024-09-16T05:16:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dd647640 1 --2- 172.21.15.114:0/4107945951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62d809b6c0 0x7f62d8098430 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f62d400b410 tx=0x7f62d400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 10 client.?.objecter ms_handle_connect 0x7f62d809b6c0 2024-09-16T05:16:38.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:38.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 1 -- 172.21.15.114:0/4107945951 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d4015030 con 0x7f62d809b6c0 2024-09-16T05:16:38.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:38.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:38.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 10 monclient: dump: 2024-09-16T05:16:38.396 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:38.396 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:38.396 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:38.396 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:38.396 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:38.396 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:38.396 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:38.396 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:38.396 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:38.397 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:38.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 10 monclient: _finish_auth 0 2024-09-16T05:16:38.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 10 monclient: _check_auth_tickets 2024-09-16T05:16:38.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:38.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 1 -- 172.21.15.114:0/4107945951 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62d4004020 con 0x7f62d809b6c0 2024-09-16T05:16:38.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:38.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 1 -- 172.21.15.114:0/4107945951 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d4004450 con 0x7f62d809b6c0 2024-09-16T05:16:38.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 5 monclient: authenticate success, global_id 74346 2024-09-16T05:16:38.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient: _renew_subs 2024-09-16T05:16:38.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:38.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 1 -- 172.21.15.114:0/4107945951 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d8072bd0 con 0x7f62d809b6c0 2024-09-16T05:16:38.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cf7fe640 4 set_mon_vals no callback set 2024-09-16T05:16:38.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:38.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:38.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 10 monclient: dump: 2024-09-16T05:16:38.398 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:38.398 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:38.398 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:38.398 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:38.398 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:38.398 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:38.398 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:38.398 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:38.398 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:38.398 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:38.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 client.74346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:38.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient: _renew_subs 2024-09-16T05:16:38.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:38.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 1 -- 172.21.15.114:0/4107945951 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d8094100 con 0x7f62d809b6c0 2024-09-16T05:16:38.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 1 librados: init done 2024-09-16T05:16:38.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:38.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:38.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:38.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62df8ab640 1 -- 172.21.15.114:0/4107945951 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d80a0ec0 con 0x7f62d809b6c0 2024-09-16T05:16:38.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 1 -- 172.21.15.114:0/4107945951 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f62d4018070 con 0x7f62d809b6c0 2024-09-16T05:16:38.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 1 --2- 172.21.15.114:0/4107945951 >> v2:172.21.15.149:6812/1185531135 conn(0x7f62bc0377c0 0x7f62bc039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:38.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 1 -- 172.21.15.114:0/4107945951 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(357..357 src has 1..357) ==== 5406+0+0 (secure 0 0 0) 0x7f62d4048a00 con 0x7f62d809b6c0 2024-09-16T05:16:38.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 10 client.74346.objecter ms_dispatch 0x7f62d8000ba0 osd_map(357..357 src has 1..357) 2024-09-16T05:16:38.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 3 client.74346.objecter handle_osd_map got epochs [357,357] > 0 2024-09-16T05:16:38.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 3 client.74346.objecter handle_osd_map decoding full epoch 357 2024-09-16T05:16:38.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 20 client.74346.objecter dump_active .. 0 homeless 2024-09-16T05:16:38.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dce46640 1 --2- 172.21.15.114:0/4107945951 >> v2:172.21.15.149:6812/1185531135 conn(0x7f62bc0377c0 0x7f62bc039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:38.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dce46640 10 monclient: get_auth_request con 0x7f62bc0377c0 auth_method 0 2024-09-16T05:16:38.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62dce46640 1 --2- 172.21.15.114:0/4107945951 >> v2:172.21.15.149:6812/1185531135 conn(0x7f62bc0377c0 0x7f62bc039c80 secure :-1 s=READY pgs=3529 cs=0 l=1 rev1=1 crypto rx=0x7f62c8007fd0 tx=0x7f62c8007900 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:38.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.387+0000 7f62cdffb640 10 client.74346.objecter ms_handle_connect 0x7f62bc0377c0 2024-09-16T05:16:38.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.391+0000 7f62cdffb640 1 -- 172.21.15.114:0/4107945951 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f62d40197b0 con 0x7f62d809b6c0 2024-09-16T05:16:38.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.391+0000 7f62cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:38.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.391+0000 7f62cdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:38.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.607+0000 7f62df8ab640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:16:38.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.607+0000 7f62df8ab640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:38.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.607+0000 7f62df8ab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:38.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.607+0000 7f62df8ab640 1 -- 172.21.15.114:0/4107945951 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f62d806dfb0 con 0x7f62d809b6c0 2024-09-16T05:16:38.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62cdffb640 1 -- 172.21.15.114:0/4107945951 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f62d4019a10 con 0x7f62d809b6c0 2024-09-16T05:16:38.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:38.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62cdffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:38.609 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:16:38.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 10 librados: watch_flush enter 2024-09-16T05:16:38.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 10 librados: watch_flush exit 2024-09-16T05:16:38.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 20 client.74346.objecter shutdown clearing up homeless session... 2024-09-16T05:16:38.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 10 client.74346.objecter successfully canceled tick 2024-09-16T05:16:38.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 1 -- 172.21.15.114:0/4107945951 >> v2:172.21.15.149:6812/1185531135 conn(0x7f62bc0377c0 msgr2=0x7f62bc039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:38.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 1 --2- 172.21.15.114:0/4107945951 >> v2:172.21.15.149:6812/1185531135 conn(0x7f62bc0377c0 0x7f62bc039c80 secure :-1 s=READY pgs=3529 cs=0 l=1 rev1=1 crypto rx=0x7f62c8007fd0 tx=0x7f62c8007900 comp rx=0 tx=0).stop 2024-09-16T05:16:38.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 10 monclient: shutdown 2024-09-16T05:16:38.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:38.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 1 -- 172.21.15.114:0/4107945951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62d809b6c0 msgr2=0x7f62d8098430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:38.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 1 --2- 172.21.15.114:0/4107945951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62d809b6c0 0x7f62d8098430 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f62d400b410 tx=0x7f62d400b8d0 comp rx=0 tx=0).stop 2024-09-16T05:16:38.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 1 -- 172.21.15.114:0/4107945951 shutdown_connections 2024-09-16T05:16:38.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 1 --2- 172.21.15.114:0/4107945951 >> v2:172.21.15.149:6812/1185531135 conn(0x7f62bc0377c0 0x7f62bc039c80 unknown :-1 s=CLOSED pgs=3529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:38.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 1 --2- 172.21.15.114:0/4107945951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62d80a19e0 0x7f62d80920c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:38.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 1 --2- 172.21.15.114:0/4107945951 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f62d80a15d0 0x7f62d8091b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:38.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 1 --2- 172.21.15.114:0/4107945951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62d809b6c0 0x7f62d8098430 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:38.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 1 -- 172.21.15.114:0/4107945951 >> 172.21.15.114:0/4107945951 conn(0x7f62d80a1fb0 msgr2=0x7f62d80724b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:38.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 1 -- 172.21.15.114:0/4107945951 shutdown_connections 2024-09-16T05:16:38.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 1 -- 172.21.15.114:0/4107945951 wait complete. 2024-09-16T05:16:38.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:38.611+0000 7f62df8ab640 1 librados: shutdown 2024-09-16T05:16:41.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:16:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 10 monclient: get_monmap_and_config 2024-09-16T05:16:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 10 monclient: build_initial_monmap 2024-09-16T05:16:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 10 monclient: monmap: 2024-09-16T05:16:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:41.757812+0000 2024-09-16T05:16:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:41.757812+0000 2024-09-16T05:16:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:41.755 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:41.755 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:41.755 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:41.755 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:41.755 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:41.755 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:41.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 1 Processor -- start 2024-09-16T05:16:41.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 1 -- start start 2024-09-16T05:16:41.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 10 monclient: init 2024-09-16T05:16:41.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:41.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-16T05:16:41.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff43409dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:41.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 10 monclient(hunting): picked mon.noname-c con 0x7ff4340a0ec0 addr v2:172.21.15.149:3300/0 2024-09-16T05:16:41.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff43409e000 addr v1:172.21.15.149:6789/0 2024-09-16T05:16:41.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff434139640 addr v1:172.21.15.114:6789/0 2024-09-16T05:16:41.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff43409a760 con 0x7ff434139640 2024-09-16T05:16:41.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4340a1870 con 0x7ff43409e000 2024-09-16T05:16:41.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:41.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7ff4340994b0 con 0x7ff4340a0ec0 2024-09-16T05:16:41.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:41.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43c662640 10 monclient(hunting): authenticate will time out at 1486.387450s 2024-09-16T05:16:41.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43abff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff434139640 0x7ff43413ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50522/0 (socket says 172.21.15.114:50522) 2024-09-16T05:16:41.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43abff640 1 -- 172.21.15.114:0/750572346 learned_addr learned my addr 172.21.15.114:0/750572346 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:16:41.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff43409dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:41.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43a3fe640 10 monclient(hunting): get_auth_request con 0x7ff4340a0ec0 auth_method 0 2024-09-16T05:16:41.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43a3fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:41.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43a3fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:41.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.755+0000 7ff43a3fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:41.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.759+0000 7ff43a3fe640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 msgr2=0x7ff43409dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:16:41.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.759+0000 7ff43a3fe640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 msgr2=0x7ff43409dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:16:41.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.759+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff43409dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:16:41.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.759+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff43409dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:41.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.963+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff43409dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:41.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.963+0000 7ff43a3fe640 10 monclient(hunting): get_auth_request con 0x7ff4340a0ec0 auth_method 0 2024-09-16T05:16:41.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.963+0000 7ff43a3fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:41.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.963+0000 7ff43a3fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:41.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.963+0000 7ff43a3fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:16:41.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.963+0000 7ff43a3fe640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 msgr2=0x7ff43409dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:16:41.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.963+0000 7ff43a3fe640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 msgr2=0x7ff43409dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:16:41.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.963+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff43409dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:16:41.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:41.963+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff43409dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:16:42.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:42.367+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff43409dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:42.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:42.367+0000 7ff43a3fe640 10 monclient(hunting): get_auth_request con 0x7ff4340a0ec0 auth_method 0 2024-09-16T05:16:42.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:42.367+0000 7ff43a3fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:42.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:42.367+0000 7ff43a3fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:42.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:42.367+0000 7ff43a3fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:16:42.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:42.367+0000 7ff43a3fe640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 msgr2=0x7ff43409dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:16:42.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:42.367+0000 7ff43a3fe640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 msgr2=0x7ff43409dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:16:42.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:42.367+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff43409dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:16:42.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:42.367+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff43409dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:16:43.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:43.171+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff43409dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:43.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:43.171+0000 7ff43a3fe640 10 monclient(hunting): get_auth_request con 0x7ff4340a0ec0 auth_method 0 2024-09-16T05:16:43.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:43.171+0000 7ff43a3fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:43.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:43.171+0000 7ff43a3fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:43.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:43.171+0000 7ff43a3fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:16:43.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:43.171+0000 7ff43a3fe640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 msgr2=0x7ff43409dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:16:43.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:43.171+0000 7ff43a3fe640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 msgr2=0x7ff43409dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:16:43.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:43.171+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff43409dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:16:43.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:43.171+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff43409dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:16:44.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 10 monclient(hunting): tick 2024-09-16T05:16:44.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:16:44.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 1 monclient(hunting): continuing hunt 2024-09-16T05:16:44.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:16:44.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 msgr2=0x7ff43409dac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:16:44.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff43409dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:44.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 1 -- 172.21.15.114:0/750572346 >> v1:172.21.15.149:6789/0 conn(0x7ff43409e000 legacy=0x7ff43409e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:44.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 1 -- 172.21.15.114:0/750572346 >> v1:172.21.15.114:6789/0 conn(0x7ff434139640 legacy=0x7ff43413ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:44.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 10 monclient: _add_conns ranks=[2,3] 2024-09-16T05:16:44.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 1 --2- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003740 0x7ff418005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:44.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 10 monclient(hunting): picked mon.noname-b con 0x7ff418003740 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:44.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 0x7ff4180086c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 10 monclient(hunting): picked mon.noname-a con 0x7ff418006200 addr v2:172.21.15.114:3300/0 2024-09-16T05:16:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 1 -- 172.21.15.114:0/750572346 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff4340994b0 con 0x7ff418006200 2024-09-16T05:16:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 1 -- 172.21.15.114:0/750572346 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff418008d80 con 0x7ff418003740 2024-09-16T05:16:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff42bfff640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff43a3fe640 1 -- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003740 msgr2=0x7ff418005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:16:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.755+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003740 0x7ff418005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.759+0000 7ff439bfd640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 0x7ff4180086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.759+0000 7ff439bfd640 10 monclient(hunting): get_auth_request con 0x7ff418006200 auth_method 0 2024-09-16T05:16:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.759+0000 7ff439bfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.759+0000 7ff439bfd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.759+0000 7ff439bfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.759+0000 7ff439bfd640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 msgr2=0x7ff4180086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:16:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.759+0000 7ff439bfd640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 msgr2=0x7ff4180086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:16:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.759+0000 7ff439bfd640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 0x7ff4180086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:16:44.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.759+0000 7ff439bfd640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 0x7ff4180086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:44.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.959+0000 7ff43a3fe640 1 -- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003740 msgr2=0x7ff418005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:16:44.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.959+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003740 0x7ff418005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:16:44.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.963+0000 7ff439bfd640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 0x7ff4180086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:44.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.963+0000 7ff439bfd640 10 monclient(hunting): get_auth_request con 0x7ff418006200 auth_method 0 2024-09-16T05:16:44.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.963+0000 7ff439bfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:44.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.963+0000 7ff439bfd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:44.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.963+0000 7ff439bfd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:16:44.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.963+0000 7ff439bfd640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 msgr2=0x7ff4180086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:16:44.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.963+0000 7ff439bfd640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 msgr2=0x7ff4180086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:16:44.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.963+0000 7ff439bfd640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 0x7ff4180086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:16:44.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:44.963+0000 7ff439bfd640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 0x7ff4180086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:16:45.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:45.363+0000 7ff43a3fe640 1 -- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003740 msgr2=0x7ff418005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:16:45.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:45.363+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003740 0x7ff418005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:16:45.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:45.367+0000 7ff439bfd640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 0x7ff4180086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:45.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:45.367+0000 7ff439bfd640 10 monclient(hunting): get_auth_request con 0x7ff418006200 auth_method 0 2024-09-16T05:16:45.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:45.367+0000 7ff439bfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:45.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:45.367+0000 7ff439bfd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:45.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:45.367+0000 7ff439bfd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:16:45.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:45.367+0000 7ff439bfd640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 msgr2=0x7ff4180086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:16:45.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:45.367+0000 7ff439bfd640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 msgr2=0x7ff4180086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:16:45.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:45.367+0000 7ff439bfd640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 0x7ff4180086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:16:45.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:45.367+0000 7ff439bfd640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 0x7ff4180086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:16:46.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:46.167+0000 7ff43a3fe640 1 -- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003740 msgr2=0x7ff418005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:16:46.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:46.167+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003740 0x7ff418005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:16:46.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:46.171+0000 7ff439bfd640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 0x7ff4180086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:46.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:46.171+0000 7ff439bfd640 10 monclient(hunting): get_auth_request con 0x7ff418006200 auth_method 0 2024-09-16T05:16:46.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:46.171+0000 7ff439bfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:46.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:46.171+0000 7ff439bfd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:46.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:46.171+0000 7ff439bfd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:16:46.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:46.171+0000 7ff439bfd640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 msgr2=0x7ff4180086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:16:46.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:46.171+0000 7ff439bfd640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 msgr2=0x7ff4180086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:16:46.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:46.171+0000 7ff439bfd640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 0x7ff4180086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:16:46.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:46.171+0000 7ff439bfd640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 0x7ff4180086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:16:47.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.755+0000 7ff42bfff640 10 monclient(hunting): tick 2024-09-16T05:16:47.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.755+0000 7ff42bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:16:47.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.755+0000 7ff42bfff640 1 monclient(hunting): continuing hunt 2024-09-16T05:16:47.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.755+0000 7ff42bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:16:47.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.755+0000 7ff42bfff640 1 -- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003740 msgr2=0x7ff418005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:16:47.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.755+0000 7ff42bfff640 1 --2- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003740 0x7ff418005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:47.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.755+0000 7ff42bfff640 1 -- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 msgr2=0x7ff4180086c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:16:47.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.755+0000 7ff42bfff640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.114:3300/0 conn(0x7ff418006200 0x7ff4180086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:47.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.755+0000 7ff42bfff640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-16T05:16:47.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.755+0000 7ff43abff640 1 -- 172.21.15.114:0/750572346 reap_dead start 2024-09-16T05:16:47.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff42bfff640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff41800b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:47.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff42bfff640 10 monclient(hunting): picked mon.noname-c con 0x7ff4340a0ec0 addr v2:172.21.15.149:3300/0 2024-09-16T05:16:47.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff42bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff43409e000 addr v1:172.21.15.149:6789/0 2024-09-16T05:16:47.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff42bfff640 1 --2- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003b60 0x7ff418006690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:47.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff42bfff640 10 monclient(hunting): picked mon.noname-b con 0x7ff418003b60 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:47.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff42bfff640 1 -- 172.21.15.114:0/750572346 --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff434099790 con 0x7ff43409e000 2024-09-16T05:16:47.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff42bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:47.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff42bfff640 1 -- 172.21.15.114:0/750572346 --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7ff4340994b0 con 0x7ff4340a0ec0 2024-09-16T05:16:47.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff42bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:47.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff42bfff640 1 -- 172.21.15.114:0/750572346 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff418008d80 con 0x7ff418003b60 2024-09-16T05:16:47.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff42bfff640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:47.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43abff640 1 -- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003b60 msgr2=0x7ff418006690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:16:47.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43abff640 1 --2- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003b60 0x7ff418006690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:47.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> v2:172.21.15.149:3300/0 conn(0x7ff4340a0ec0 0x7ff41800b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:47.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 10 monclient(hunting): get_auth_request con 0x7ff4340a0ec0 auth_method 0 2024-09-16T05:16:47.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:47.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:47.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:47.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:47.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:47.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:47.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 10 monclient(hunting): handle_auth_done global_id 64465 payload 274 2024-09-16T05:16:47.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 1 -- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003b60 msgr2=0x7ff418006690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:16:47.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003b60 0x7ff418006690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:47.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 1 -- 172.21.15.114:0/750572346 >> v1:172.21.15.149:6789/0 conn(0x7ff43409e000 legacy=0x7ff41800d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:47.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 10 monclient: _finish_hunting 0 2024-09-16T05:16:47.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 1 monclient: found mon.noname-c 2024-09-16T05:16:47.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:47.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:16:47.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff43a3fe640 1 -- 172.21.15.114:0/750572346 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff418007d10 con 0x7ff4340a0ec0 2024-09-16T05:16:47.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff4393fc640 1 -- 172.21.15.114:0/750572346 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2429467973 0 0) 0x7ff434099790 con 0x7ff43409e000 2024-09-16T05:16:47.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.759+0000 7ff4393fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:16:47.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.763+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/750572346 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4340a0ec0 0x7ff41800b260 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff430009610 tx=0x7ff43001d7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eee11bd5be541361 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:47.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.763+0000 7ff4393fc640 1 -- 172.21.15.114:0/750572346 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4300af070 con 0x7ff4340a0ec0 2024-09-16T05:16:47.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff4393fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:47.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff4393fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:16:47.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff4393fc640 10 monclient: dump: 2024-09-16T05:16:47.764 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:47.765 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:47.765 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:47.765 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:47.765 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:47.765 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:47.765 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:47.765 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:47.765 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:47.765 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:47.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff4393fc640 10 monclient: _finish_auth 0 2024-09-16T05:16:47.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff4393fc640 10 monclient: _check_auth_tickets 2024-09-16T05:16:47.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff4393fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:47.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 5 monclient: authenticate success, global_id 64465 2024-09-16T05:16:47.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff4393fc640 1 -- 172.21.15.114:0/750572346 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4300aa6a0 con 0x7ff4340a0ec0 2024-09-16T05:16:47.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff4393fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:47.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:47.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 4 set_mon_vals no callback set 2024-09-16T05:16:47.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff4393fc640 1 -- 172.21.15.114:0/750572346 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4300aaa90 con 0x7ff4340a0ec0 2024-09-16T05:16:47.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff4393fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:47.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff4393fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:16:47.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff4393fc640 10 monclient: dump: 2024-09-16T05:16:47.766 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:47.766 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:47.766 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:47.766 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:47.766 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:47.766 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:47.767 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:47.767 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:47.767 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:47.767 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:47.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 10 monclient: shutdown 2024-09-16T05:16:47.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:47.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 1 -- 172.21.15.114:0/750572346 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4340a0ec0 msgr2=0x7ff41800b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:47.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 1 --2- 172.21.15.114:0/750572346 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4340a0ec0 0x7ff41800b260 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff430009610 tx=0x7ff43001d7c0 comp rx=0 tx=0).stop 2024-09-16T05:16:47.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 1 -- 172.21.15.114:0/750572346 shutdown_connections 2024-09-16T05:16:47.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 1 --2- 172.21.15.114:0/750572346 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff418003b60 0x7ff418006690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:47.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 1 --2- 172.21.15.114:0/750572346 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4340a0ec0 0x7ff41800b260 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:47.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 1 -- 172.21.15.114:0/750572346 >> 172.21.15.114:0/750572346 conn(0x7ff4340a1fb0 msgr2=0x7ff4340a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:47.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 1 -- 172.21.15.114:0/750572346 shutdown_connections 2024-09-16T05:16:47.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 1 -- 172.21.15.114:0/750572346 wait complete. 2024-09-16T05:16:47.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 10 monclient: build_initial_monmap 2024-09-16T05:16:47.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 10 monclient: monmap: 2024-09-16T05:16:47.769 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:47.769 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:47.769 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:47.769 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:47.769 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:47.769 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:47.769 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:47.770 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:47.770 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:47.770 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:47.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 1 librados: starting msgr at 2024-09-16T05:16:47.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 1 librados: starting objecter 2024-09-16T05:16:47.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 1 Processor -- start 2024-09-16T05:16:47.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.767+0000 7ff43c662640 1 -- start start 2024-09-16T05:16:47.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 1 librados: setting wanted keys 2024-09-16T05:16:47.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 1 librados: calling monclient init 2024-09-16T05:16:47.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient: init 2024-09-16T05:16:47.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:47.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:16:47.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff418003b60 0x7ff4341c3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:47.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient(hunting): picked mon.noname-a con 0x7ff418003b60 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:47.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff43409e000 0x7ff434133830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:47.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient(hunting): picked mon.noname-c con 0x7ff43409e000 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:47.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4340a0ec0 0x7ff434133d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:47.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient(hunting): picked mon.noname-b con 0x7ff4340a0ec0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:47.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:47.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff43409c3b0 con 0x7ff418003b60 2024-09-16T05:16:47.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:47.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7ff43409c230 con 0x7ff4340a0ec0 2024-09-16T05:16:47.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:47.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff434136e30 con 0x7ff43409e000 2024-09-16T05:16:47.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:47.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient(hunting): authenticate will time out at 1492.400900s 2024-09-16T05:16:47.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff439bfd640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff43409e000 msgr2=0x7ff434133830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:16:47.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff439bfd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff43409e000 0x7ff434133830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:47.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff418003b60 0x7ff4341c3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:47.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff418003b60 0x7ff4341c3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53606/0 (socket says 172.21.15.114:53606) 2024-09-16T05:16:47.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 1 -- 172.21.15.114:0/1089080813 learned_addr learned my addr 172.21.15.114:0/1089080813 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:47.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 10 monclient(hunting): get_auth_request con 0x7ff418003b60 auth_method 0 2024-09-16T05:16:47.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:47.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:47.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:47.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43abff640 1 --2- 172.21.15.114:0/1089080813 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4340a0ec0 0x7ff434133d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:47.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43abff640 10 monclient(hunting): get_auth_request con 0x7ff4340a0ec0 auth_method 0 2024-09-16T05:16:47.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43abff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43abff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43abff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43abff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43abff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43abff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 10 monclient(hunting): handle_auth_done global_id 74349 payload 824 2024-09-16T05:16:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 1 -- 172.21.15.114:0/1089080813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff43409e000 msgr2=0x7ff434133830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:16:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/1089080813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff43409e000 0x7ff434133830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 1 -- 172.21.15.114:0/1089080813 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4340a0ec0 msgr2=0x7ff434133d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/1089080813 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4340a0ec0 0x7ff434133d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 10 monclient: _finish_hunting 0 2024-09-16T05:16:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 1 monclient: found mon.noname-a 2024-09-16T05:16:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 1 -- 172.21.15.114:0/1089080813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4341cff80 con 0x7ff418003b60 2024-09-16T05:16:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43a3fe640 1 --2- 172.21.15.114:0/1089080813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff418003b60 0x7ff4341c3710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff43001d230 tx=0x7ff43001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 10 client.?.objecter ms_handle_connect 0x7ff418003b60 2024-09-16T05:16:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 1 -- 172.21.15.114:0/1089080813 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4300af040 con 0x7ff418003b60 2024-09-16T05:16:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 10 monclient: dump: 2024-09-16T05:16:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:47.780 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:47.780 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:47.780 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:47.780 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:47.780 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:47.780 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:47.780 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:47.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 10 monclient: _finish_auth 0 2024-09-16T05:16:47.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 10 monclient: _check_auth_tickets 2024-09-16T05:16:47.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:47.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 5 monclient: authenticate success, global_id 74349 2024-09-16T05:16:47.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 1 -- 172.21.15.114:0/1089080813 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4300aaa60 con 0x7ff418003b60 2024-09-16T05:16:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient: _renew_subs 2024-09-16T05:16:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 1 -- 172.21.15.114:0/1089080813 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4300aae10 con 0x7ff418003b60 2024-09-16T05:16:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 1 -- 172.21.15.114:0/1089080813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff43412d2b0 con 0x7ff418003b60 2024-09-16T05:16:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42bfff640 4 set_mon_vals no callback set 2024-09-16T05:16:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 10 monclient: dump: 2024-09-16T05:16:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:47.782 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:47.782 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:47.782 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:47.782 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:47.782 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:47.782 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:47.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 client.74349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:47.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient: _renew_subs 2024-09-16T05:16:47.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:47.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 1 -- 172.21.15.114:0/1089080813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff43412cfd0 con 0x7ff418003b60 2024-09-16T05:16:47.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 1 librados: init done 2024-09-16T05:16:47.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:47.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:47.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:47.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff43c662640 1 -- 172.21.15.114:0/1089080813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4341371c0 con 0x7ff418003b60 2024-09-16T05:16:47.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 1 -- 172.21.15.114:0/1089080813 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff4300b2070 con 0x7ff418003b60 2024-09-16T05:16:47.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 1 --2- 172.21.15.114:0/1089080813 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff418038d80 0x7ff41803b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:47.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 1 -- 172.21.15.114:0/1089080813 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(358..358 src has 1..358) ==== 5021+0+0 (secure 0 0 0) 0x7ff4300e45e0 con 0x7ff418003b60 2024-09-16T05:16:47.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 10 client.74349.objecter ms_dispatch 0x7ff434000ba0 osd_map(358..358 src has 1..358) 2024-09-16T05:16:47.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 3 client.74349.objecter handle_osd_map got epochs [358,358] > 0 2024-09-16T05:16:47.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 3 client.74349.objecter handle_osd_map decoding full epoch 358 2024-09-16T05:16:47.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.771+0000 7ff42affd640 20 client.74349.objecter dump_active .. 0 homeless 2024-09-16T05:16:47.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.775+0000 7ff439bfd640 1 --2- 172.21.15.114:0/1089080813 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff418038d80 0x7ff41803b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:47.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.775+0000 7ff439bfd640 10 monclient: get_auth_request con 0x7ff418038d80 auth_method 0 2024-09-16T05:16:47.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.775+0000 7ff42affd640 10 client.74349.objecter ms_handle_connect 0x7ff418038d80 2024-09-16T05:16:47.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.775+0000 7ff439bfd640 1 --2- 172.21.15.114:0/1089080813 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff418038d80 0x7ff41803b240 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7ff424000eb0 tx=0x7ff424009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.779+0000 7ff42affd640 1 -- 172.21.15.114:0/1089080813 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ff4341cff80 con 0x7ff418003b60 2024-09-16T05:16:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.779+0000 7ff42affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:47.779+0000 7ff42affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:48.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:16:48.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:48.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:48.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 1 -- 172.21.15.114:0/1089080813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff4340686f0 con 0x7ff418003b60 2024-09-16T05:16:48.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff42affd640 1 -- 172.21.15.114:0/1089080813 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7ff4300174d0 con 0x7ff418003b60 2024-09-16T05:16:48.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff42affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:16:48.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff42affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:48.010 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":44,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:16:48.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 10 librados: watch_flush enter 2024-09-16T05:16:48.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 10 librados: watch_flush exit 2024-09-16T05:16:48.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 20 client.74349.objecter shutdown clearing up homeless session... 2024-09-16T05:16:48.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 10 client.74349.objecter successfully canceled tick 2024-09-16T05:16:48.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 1 -- 172.21.15.114:0/1089080813 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff418038d80 msgr2=0x7ff41803b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:48.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 1 --2- 172.21.15.114:0/1089080813 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff418038d80 0x7ff41803b240 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7ff424000eb0 tx=0x7ff424009040 comp rx=0 tx=0).stop 2024-09-16T05:16:48.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 10 monclient: shutdown 2024-09-16T05:16:48.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:48.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 1 -- 172.21.15.114:0/1089080813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff418003b60 msgr2=0x7ff4341c3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:48.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 1 --2- 172.21.15.114:0/1089080813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff418003b60 0x7ff4341c3710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff43001d230 tx=0x7ff43001c000 comp rx=0 tx=0).stop 2024-09-16T05:16:48.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 1 -- 172.21.15.114:0/1089080813 shutdown_connections 2024-09-16T05:16:48.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 1 --2- 172.21.15.114:0/1089080813 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff418038d80 0x7ff41803b240 unknown :-1 s=CLOSED pgs=3530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 1 --2- 172.21.15.114:0/1089080813 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4340a0ec0 0x7ff434133d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 1 --2- 172.21.15.114:0/1089080813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff43409e000 0x7ff434133830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 1 --2- 172.21.15.114:0/1089080813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff418003b60 0x7ff4341c3710 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.011+0000 7ff43c662640 1 -- 172.21.15.114:0/1089080813 >> 172.21.15.114:0/1089080813 conn(0x7ff4340a1fb0 msgr2=0x7ff43409c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:48.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.015+0000 7ff43c662640 1 -- 172.21.15.114:0/1089080813 shutdown_connections 2024-09-16T05:16:48.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.015+0000 7ff43c662640 1 -- 172.21.15.114:0/1089080813 wait complete. 2024-09-16T05:16:48.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.015+0000 7ff43c662640 1 librados: shutdown 2024-09-16T05:16:48.029 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:16:48.029 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T05:16:48.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:16:48.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 10 monclient: get_monmap_and_config 2024-09-16T05:16:48.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 10 monclient: build_initial_monmap 2024-09-16T05:16:48.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 10 monclient: monmap: 2024-09-16T05:16:48.149 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:48.149 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:48.149 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:48.152423+0000 2024-09-16T05:16:48.149 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:48.152423+0000 2024-09-16T05:16:48.149 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:48.149 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.149 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:48.149 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:48.149 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:48.149 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:48.149 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:48.149 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 1 Processor -- start 2024-09-16T05:16:48.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 1 -- start start 2024-09-16T05:16:48.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 10 monclient: init 2024-09-16T05:16:48.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:48.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-16T05:16:48.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7e01376e0 addr v1:172.21.15.114:6789/0 2024-09-16T05:16:48.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fa7e01370a0 0x7fa7e0137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 10 monclient(hunting): picked mon.noname-c con 0x7fa7e01370a0 addr v2:172.21.15.149:3300/0 2024-09-16T05:16:48.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa7e00a5d00 0x7fa7e00a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 10 monclient(hunting): picked mon.noname-a con 0x7fa7e00a5d00 addr v2:172.21.15.114:3300/0 2024-09-16T05:16:48.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e009bd20 con 0x7fa7e01376e0 2024-09-16T05:16:48.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:48.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa7e0138000 con 0x7fa7e00a5d00 2024-09-16T05:16:48.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:48.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fa7e009c5c0 con 0x7fa7e01370a0 2024-09-16T05:16:48.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:48.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7e6190640 10 monclient(hunting): authenticate will time out at 1492.781659s 2024-09-16T05:16:48.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7df7fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa7e01376e0 0x7fa7e00a4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50334/0 (socket says 172.21.15.114:50334) 2024-09-16T05:16:48.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7df7fe640 1 -- 172.21.15.114:0/263010851 learned_addr learned my addr 172.21.15.114:0/263010851 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:16:48.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 1 --2- 172.21.15.114:0/263010851 >> v2:172.21.15.114:3300/0 conn(0x7fa7e00a5d00 0x7fa7e00a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:48.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 10 monclient(hunting): get_auth_request con 0x7fa7e00a5d00 auth_method 0 2024-09-16T05:16:48.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:48.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:48.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:48.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:48.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:48.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:48.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7deffd640 1 --2- 172.21.15.114:0/263010851 >> v2:172.21.15.149:3300/0 conn(0x7fa7e01370a0 0x7fa7e0137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:48.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7deffd640 10 monclient(hunting): get_auth_request con 0x7fa7e01370a0 auth_method 0 2024-09-16T05:16:48.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:48.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:48.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:48.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 10 monclient(hunting): handle_auth_done global_id 74352 payload 274 2024-09-16T05:16:48.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 1 -- 172.21.15.114:0/263010851 >> v2:172.21.15.149:3300/0 conn(0x7fa7e01370a0 msgr2=0x7fa7e0137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:48.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 1 --2- 172.21.15.114:0/263010851 >> v2:172.21.15.149:3300/0 conn(0x7fa7e01370a0 0x7fa7e0137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 1 -- 172.21.15.114:0/263010851 >> v1:172.21.15.114:6789/0 conn(0x7fa7e01376e0 legacy=0x7fa7e00a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:48.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 10 monclient: _finish_hunting 0 2024-09-16T05:16:48.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 1 monclient: found mon.noname-a 2024-09-16T05:16:48.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:48.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:48.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.151+0000 7fa7dffff640 1 -- 172.21.15.114:0/263010851 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e009c000 con 0x7fa7e00a5d00 2024-09-16T05:16:48.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7dffff640 1 --2- 172.21.15.114:0/263010851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e00a5d00 0x7fa7e00a4980 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0073760 tx=0x7fa7d001c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11fd4484cd46318d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:48.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7de7fc640 1 -- 172.21.15.114:0/263010851 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d00b0070 con 0x7fa7e00a5d00 2024-09-16T05:16:48.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:48.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:48.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7de7fc640 10 monclient: dump: 2024-09-16T05:16:48.161 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:48.162 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:48.162 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.162 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.162 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:48.162 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.162 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:48.162 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:48.162 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:48.162 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7de7fc640 10 monclient: _finish_auth 0 2024-09-16T05:16:48.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7de7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:16:48.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:48.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7de7fc640 1 -- 172.21.15.114:0/263010851 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7d00ab600 con 0x7fa7e00a5d00 2024-09-16T05:16:48.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7de7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:48.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7de7fc640 1 -- 172.21.15.114:0/263010851 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d00ab9d0 con 0x7fa7e00a5d00 2024-09-16T05:16:48.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 5 monclient: authenticate success, global_id 74352 2024-09-16T05:16:48.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:48.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 4 set_mon_vals no callback set 2024-09-16T05:16:48.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:48.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:48.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7de7fc640 10 monclient: dump: 2024-09-16T05:16:48.163 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:48.163 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:48.163 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.163 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.163 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:48.163 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.163 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:48.164 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:48.164 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:48.164 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient: shutdown 2024-09-16T05:16:48.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:48.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 -- 172.21.15.114:0/263010851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e00a5d00 msgr2=0x7fa7e00a4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:48.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 --2- 172.21.15.114:0/263010851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e00a5d00 0x7fa7e00a4980 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0073760 tx=0x7fa7d001c340 comp rx=0 tx=0).stop 2024-09-16T05:16:48.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 -- 172.21.15.114:0/263010851 shutdown_connections 2024-09-16T05:16:48.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 --2- 172.21.15.114:0/263010851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e00a5d00 0x7fa7e00a4980 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 --2- 172.21.15.114:0/263010851 >> v2:172.21.15.149:3300/0 conn(0x7fa7e01370a0 0x7fa7e0137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 -- 172.21.15.114:0/263010851 >> 172.21.15.114:0/263010851 conn(0x7fa7e00a9a40 msgr2=0x7fa7e00a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:48.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 -- 172.21.15.114:0/263010851 shutdown_connections 2024-09-16T05:16:48.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 -- 172.21.15.114:0/263010851 wait complete. 2024-09-16T05:16:48.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient: build_initial_monmap 2024-09-16T05:16:48.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient: monmap: 2024-09-16T05:16:48.165 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:48.165 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:48.165 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:48.165 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:48.165 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:48.165 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.165 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:48.165 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:48.165 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:48.165 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 librados: starting msgr at 2024-09-16T05:16:48.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 librados: starting objecter 2024-09-16T05:16:48.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 Processor -- start 2024-09-16T05:16:48.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 -- start start 2024-09-16T05:16:48.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 librados: setting wanted keys 2024-09-16T05:16:48.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 librados: calling monclient init 2024-09-16T05:16:48.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient: init 2024-09-16T05:16:48.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:48.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:16:48.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e00a5d00 0x7fa7e01cf030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient(hunting): picked mon.noname-a con 0x7fa7e00a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:48.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa7e01370a0 0x7fa7e01cf760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient(hunting): picked mon.noname-b con 0x7fa7e01370a0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:48.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e01376e0 0x7fa7e013c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient(hunting): picked mon.noname-c con 0x7fa7e01376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e0141d80 con 0x7fa7e00a5d00 2024-09-16T05:16:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e0141c00 con 0x7fa7e01370a0 2024-09-16T05:16:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa7e0141f00 con 0x7fa7e01376e0 2024-09-16T05:16:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient(hunting): authenticate will time out at 1492.786514s 2024-09-16T05:16:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7dffff640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e01376e0 msgr2=0x7fa7e013c070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:16:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7dffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e01376e0 0x7fa7e013c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e00a5d00 0x7fa7e01cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e00a5d00 0x7fa7e01cf030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53636/0 (socket says 172.21.15.114:53636) 2024-09-16T05:16:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 1 -- 172.21.15.114:0/2646065716 learned_addr learned my addr 172.21.15.114:0/2646065716 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 10 monclient(hunting): get_auth_request con 0x7fa7e00a5d00 auth_method 0 2024-09-16T05:16:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7deffd640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa7e01370a0 0x7fa7e01cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 10 monclient(hunting): handle_auth_done global_id 74358 payload 824 2024-09-16T05:16:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 1 -- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e01376e0 msgr2=0x7fa7e013c070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:16:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e01376e0 0x7fa7e013c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 1 -- 172.21.15.114:0/2646065716 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa7e01370a0 msgr2=0x7fa7e01cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa7e01370a0 0x7fa7e01cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:16:48.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 1 monclient: found mon.noname-a 2024-09-16T05:16:48.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:48.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:48.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 1 -- 172.21.15.114:0/2646065716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e0120860 con 0x7fa7e00a5d00 2024-09-16T05:16:48.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7deffd640 10 monclient: get_auth_request con 0x7fa7e01370a0 auth_method 0 2024-09-16T05:16:48.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7deffd640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa7e01370a0 0x7fa7e01cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:16:48.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7df7fe640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e00a5d00 0x7fa7e01cf030 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800cb40 tx=0x7fa7d800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:48.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7c7fff640 10 client.?.objecter ms_handle_connect 0x7fa7e00a5d00 2024-09-16T05:16:48.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7c7fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:48.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7c7fff640 1 -- 172.21.15.114:0/2646065716 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8017070 con 0x7fa7e00a5d00 2024-09-16T05:16:48.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:48.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:48.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7c7fff640 10 monclient: dump: 2024-09-16T05:16:48.170 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:48.170 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:48.170 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.170 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.170 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:48.170 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.170 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:48.170 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:48.170 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:48.170 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7c7fff640 10 monclient: _finish_auth 0 2024-09-16T05:16:48.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7c7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:16:48.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:48.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7c7fff640 1 -- 172.21.15.114:0/2646065716 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7d800aca0 con 0x7fa7e00a5d00 2024-09-16T05:16:48.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 5 monclient: authenticate success, global_id 74358 2024-09-16T05:16:48.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient: _renew_subs 2024-09-16T05:16:48.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:48.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 -- 172.21.15.114:0/2646065716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e0120b40 con 0x7fa7e00a5d00 2024-09-16T05:16:48.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7c7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:48.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7c7fff640 1 -- 172.21.15.114:0/2646065716 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8012620 con 0x7fa7e00a5d00 2024-09-16T05:16:48.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:48.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:48.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7c7fff640 10 monclient: dump: 2024-09-16T05:16:48.171 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:48.171 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:48.171 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.171 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.171 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:48.171 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.172 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:48.172 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:48.172 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:48.172 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7ddffb640 4 set_mon_vals no callback set 2024-09-16T05:16:48.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 client.74358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:48.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient: _renew_subs 2024-09-16T05:16:48.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:48.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 -- 172.21.15.114:0/2646065716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e0073d20 con 0x7fa7e00a5d00 2024-09-16T05:16:48.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 librados: init done 2024-09-16T05:16:48.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:48.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e01cfca0 0x7fa7e013c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:48.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.155+0000 7fa7e6190640 1 -- 172.21.15.114:0/2646065716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e0137ac0 con 0x7fa7e01cfca0 2024-09-16T05:16:48.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7deffd640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e01cfca0 0x7fa7e013c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:48.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7deffd640 10 monclient: get_auth_request con 0x7fa7e01cfca0 auth_method 0 2024-09-16T05:16:48.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:48.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:48.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:48.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 1 -- 172.21.15.114:0/2646065716 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa7d801a070 con 0x7fa7e00a5d00 2024-09-16T05:16:48.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:48.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:48.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:48.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7deffd640 10 monclient(hunting): handle_auth_done global_id 74361 payload 824 2024-09-16T05:16:48.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7deffd640 1 -- 172.21.15.114:0/2646065716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7e0120640 con 0x7fa7e01cfca0 2024-09-16T05:16:48.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 1 --2- 172.21.15.114:0/2646065716 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa7cc0377c0 0x7fa7cc039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 1 -- 172.21.15.114:0/2646065716 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(358..358 src has 1..358) ==== 5021+0+0 (secure 0 0 0) 0x7fa7d8048a90 con 0x7fa7e00a5d00 2024-09-16T05:16:48.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 10 client.74358.objecter ms_dispatch 0x7fa7e0000ba0 osd_map(358..358 src has 1..358) 2024-09-16T05:16:48.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 3 client.74358.objecter handle_osd_map got epochs [358,358] > 0 2024-09-16T05:16:48.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 3 client.74358.objecter handle_osd_map decoding full epoch 358 2024-09-16T05:16:48.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7deffd640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e01cfca0 0x7fa7e013c5b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa7e013d910 tx=0x7fa7d400eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:48.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 20 client.74358.objecter dump_active .. 0 homeless 2024-09-16T05:16:48.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 10 client.74358.objecter ms_handle_connect 0x7fa7e01cfca0 2024-09-16T05:16:48.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 10 client.74358.objecter resend_mon_ops 2024-09-16T05:16:48.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 1 -- 172.21.15.114:0/2646065716 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4017020 con 0x7fa7e01cfca0 2024-09-16T05:16:48.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:48.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:48.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 10 monclient: dump: 2024-09-16T05:16:48.174 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:48.174 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:48.175 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7dffff640 1 --2- 172.21.15.114:0/2646065716 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa7cc0377c0 0x7fa7cc039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7dffff640 10 monclient: get_auth_request con 0x7fa7cc0377c0 auth_method 0 2024-09-16T05:16:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 1 -- 172.21.15.114:0/2646065716 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa7d4004030 con 0x7fa7e01cfca0 2024-09-16T05:16:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 1 -- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e01cfca0 msgr2=0x7fa7e013c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e01cfca0 0x7fa7e013c5b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa7e013d910 tx=0x7fa7d400eba0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 1 -- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e01cfca0 msgr2=0x7fa7e013c5b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e01cfca0 0x7fa7e013c5b0 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa7e013d910 tx=0x7fa7d400eba0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7c7fff640 10 client.74358.objecter ms_handle_connect 0x7fa7cc0377c0 2024-09-16T05:16:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.159+0000 7fa7dffff640 1 --2- 172.21.15.114:0/2646065716 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa7cc0377c0 0x7fa7cc039c80 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00ad000 tx=0x7fa7d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7e6190640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:16:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7e6190640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e013caf0 0x7fa7e0138190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7e6190640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7e6190640 1 -- 172.21.15.114:0/2646065716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e013cf00 con 0x7fa7e013caf0 2024-09-16T05:16:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7deffd640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e013caf0 0x7fa7e0138190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7deffd640 10 monclient: get_auth_request con 0x7fa7e013caf0 auth_method 0 2024-09-16T05:16:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7deffd640 10 monclient(hunting): handle_auth_done global_id 74364 payload 824 2024-09-16T05:16:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7deffd640 1 -- 172.21.15.114:0/2646065716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa7e013e5f0 con 0x7fa7e013caf0 2024-09-16T05:16:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7c7fff640 10 client.74358.objecter ms_handle_connect 0x7fa7e013caf0 2024-09-16T05:16:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7c7fff640 10 client.74358.objecter resend_mon_ops 2024-09-16T05:16:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7deffd640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e013caf0 0x7fa7e0138190 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4000c10 tx=0x7fa7d4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7c7fff640 1 -- 172.21.15.114:0/2646065716 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4017070 con 0x7fa7e013caf0 2024-09-16T05:16:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.163+0000 7fa7c7fff640 10 monclient: dump: 2024-09-16T05:16:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:48.180 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7c7fff640 1 -- 172.21.15.114:0/2646065716 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fa7d400c9c0 con 0x7fa7e013caf0 2024-09-16T05:16:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7c7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:16:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7c7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7c7fff640 1 -- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e013caf0 msgr2=0x7fa7e0138190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7c7fff640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e013caf0 0x7fa7e0138190 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4000c10 tx=0x7fa7d4004740 comp rx=0 tx=0).stop 2024-09-16T05:16:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7c7fff640 1 -- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e013caf0 msgr2=0x7fa7e0138190 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7c7fff640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e013caf0 0x7fa7e0138190 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4000c10 tx=0x7fa7d4004740 comp rx=0 tx=0).stop 2024-09-16T05:16:48.181 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:16:48.181 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:16:48.181 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:16:48.181 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:16:48.181 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 44, 2024-09-16T05:16:48.181 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 167218, 2024-09-16T05:16:48.181 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:16:48.181 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:16:48.181 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-16T05:16:48.182 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:48.182 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 1, 2024-09-16T05:16:48.182 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:48.182 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:16:48.182 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:16:48.182 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:48.182 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:48.182 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:48.182 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:48.182 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:48.182 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:48.182 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:48.182 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:48.182 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:48.183 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:48.183 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:48.183 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:48.183 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:16:48.183 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:16:48.183 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:48.183 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:48.183 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:48.183 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:48.183 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:48.183 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:48.183 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:48.183 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:48.183 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:48.183 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:48.184 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:48.184 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:48.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.184 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:16:48.184 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:16:48.184 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:16:48.184 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:16:48.184 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:16:48.184 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:16:48.184 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:48.184 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:48.184 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:16:48.185 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:16:48.185 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:16:48.185 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:16:48.185 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:16:48.185 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:16:48.185 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:16:48.185 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:48.185 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:16:48.185 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:48.185 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:48.185 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:48.185 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:48.185 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:48.186 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:48.186 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:48.186 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:48.186 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:48.186 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:48.186 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:48.186 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:48.186 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:16:48.186 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.186 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:16:48.186 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.186 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:16:48.186 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:16:48.186 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:48.186 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:48.187 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.187 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:48.187 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:16:48.187 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:48.187 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.187 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.187 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:48.187 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:16:48.187 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:48.187 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:48.187 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:48.187 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.187 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:16:48.188 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:16:48.188 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:48.188 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:48.188 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:48.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.188 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.188 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:16:48.188 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:16:48.188 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:48.188 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:48.188 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.188 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:48.189 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:16:48.189 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:48.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.189 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.189 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:48.189 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:16:48.189 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:48.189 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:48.189 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:48.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.189 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:16:48.189 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:16:48.189 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:48.189 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:48.190 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:48.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.190 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.190 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:16:48.190 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:16:48.190 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:48.190 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:48.190 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.190 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:48.190 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:16:48.190 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:48.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.190 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.191 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:48.191 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:16:48.191 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:48.191 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:48.191 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:48.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.191 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:16:48.191 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:16:48.191 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:48.191 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:48.191 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:48.191 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:48.191 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:48.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.192 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:16:48.192 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:16:48.192 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.192 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:48.192 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:48.192 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:16:48.192 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:48.192 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:48.192 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:16:48.192 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.192 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:48.192 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:48.193 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:16:48.193 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:48.193 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:48.193 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:16:48.193 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.193 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:48.193 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:48.193 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:16:48.193 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:48.193 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:48.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.193 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:16:48.193 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:16:48.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7e6190640 10 librados: watch_flush enter 2024-09-16T05:16:48.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7e6190640 10 librados: watch_flush exit 2024-09-16T05:16:48.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7e6190640 20 client.74358.objecter shutdown clearing up homeless session... 2024-09-16T05:16:48.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7e6190640 10 client.74358.objecter successfully canceled tick 2024-09-16T05:16:48.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7e6190640 1 -- 172.21.15.114:0/2646065716 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa7cc0377c0 msgr2=0x7fa7cc039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:48.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7e6190640 1 --2- 172.21.15.114:0/2646065716 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa7cc0377c0 0x7fa7cc039c80 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00ad000 tx=0x7fa7d0008040 comp rx=0 tx=0).stop 2024-09-16T05:16:48.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7e6190640 10 monclient: shutdown 2024-09-16T05:16:48.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7e6190640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:48.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7e6190640 1 -- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e00a5d00 msgr2=0x7fa7e01cf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:48.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7e6190640 1 --2- 172.21.15.114:0/2646065716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e00a5d00 0x7fa7e01cf030 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800cb40 tx=0x7fa7d800a060 comp rx=0 tx=0).stop 2024-09-16T05:16:48.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7dffff640 1 -- 172.21.15.114:0/2646065716 reap_dead start 2024-09-16T05:16:48.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7dffff640 1 -- 172.21.15.114:0/2646065716 reap_dead start 2024-09-16T05:16:48.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7e6190640 1 -- 172.21.15.114:0/2646065716 shutdown_connections 2024-09-16T05:16:48.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7e6190640 1 -- 172.21.15.114:0/2646065716 >> 172.21.15.114:0/2646065716 conn(0x7fa7e00a9a40 msgr2=0x7fa7e00a7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:48.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7e6190640 1 -- 172.21.15.114:0/2646065716 shutdown_connections 2024-09-16T05:16:48.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7e6190640 1 -- 172.21.15.114:0/2646065716 wait complete. 2024-09-16T05:16:48.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.167+0000 7fa7e6190640 1 librados: shutdown 2024-09-16T05:16:48.197 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:16:48.197 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T05:16:48.197 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:16:48.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 10 monclient: get_monmap_and_config 2024-09-16T05:16:48.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 10 monclient: build_initial_monmap 2024-09-16T05:16:48.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 10 monclient: monmap: 2024-09-16T05:16:48.313 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:48.313 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:48.313 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:16:48.316414+0000 2024-09-16T05:16:48.313 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:16:48.316414+0000 2024-09-16T05:16:48.313 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:48.313 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.313 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:16:48.313 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:16:48.314 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:16:48.314 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:16:48.314 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:16:48.314 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 1 Processor -- start 2024-09-16T05:16:48.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 1 -- start start 2024-09-16T05:16:48.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 10 monclient: init 2024-09-16T05:16:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-16T05:16:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0330068250 0x7f033009d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 10 monclient(hunting): picked mon.noname-b con 0x7f0330068250 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f033009f370 0x7f033009d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 10 monclient(hunting): picked mon.noname-a con 0x7f033009f370 addr v2:172.21.15.114:3300/0 2024-09-16T05:16:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f033009dc80 0x7f033013a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 10 monclient(hunting): picked mon.noname-c con 0x7f033009dc80 addr v2:172.21.15.149:3300/0 2024-09-16T05:16:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f033009fef0 con 0x7f033009f370 2024-09-16T05:16:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f0330068630 con 0x7f033009dc80 2024-09-16T05:16:48.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:48.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f03300a3720 con 0x7f0330068250 2024-09-16T05:16:48.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:48.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f03362f9640 10 monclient(hunting): authenticate will time out at 1492.945764s 2024-09-16T05:16:48.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032ffff640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0330068250 msgr2=0x7f033009d200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:16:48.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f033009f370 0x7f033009d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:48.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032ffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0330068250 0x7f033009d200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:48.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f033009f370 0x7f033009d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53660/0 (socket says 172.21.15.114:53660) 2024-09-16T05:16:48.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 1 -- 172.21.15.114:0/4102897666 learned_addr learned my addr 172.21.15.114:0/4102897666 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:48.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 10 monclient(hunting): get_auth_request con 0x7f033009f370 auth_method 0 2024-09-16T05:16:48.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:48.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:48.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:48.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:48.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:48.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:48.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f0334896640 1 --2- 172.21.15.114:0/4102897666 >> v2:172.21.15.149:3300/0 conn(0x7f033009dc80 0x7f033013a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:48.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f0334896640 10 monclient(hunting): get_auth_request con 0x7f033009dc80 auth_method 0 2024-09-16T05:16:48.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f0334896640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:48.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f0334896640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:48.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f0334896640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:48.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 10 monclient(hunting): handle_auth_done global_id 74367 payload 274 2024-09-16T05:16:48.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 1 -- 172.21.15.114:0/4102897666 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0330068250 msgr2=0x7f033009d200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:16:48.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 1 --2- 172.21.15.114:0/4102897666 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0330068250 0x7f033009d200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 1 -- 172.21.15.114:0/4102897666 >> v2:172.21.15.149:3300/0 conn(0x7f033009dc80 msgr2=0x7f033013a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:16:48.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 1 --2- 172.21.15.114:0/4102897666 >> v2:172.21.15.149:3300/0 conn(0x7f033009dc80 0x7f033013a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:16:48.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 1 monclient: found mon.noname-a 2024-09-16T05:16:48.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:48.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:48.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.315+0000 7f032f7fe640 1 -- 172.21.15.114:0/4102897666 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03300a1c40 con 0x7f033009f370 2024-09-16T05:16:48.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032f7fe640 1 --2- 172.21.15.114:0/4102897666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f033009f370 0x7f033009d740 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f033006be20 tx=0x7f031801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3048eac548d8b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:48.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032effd640 1 -- 172.21.15.114:0/4102897666 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03180b0070 con 0x7f033009f370 2024-09-16T05:16:48.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:48.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:48.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032effd640 10 monclient: dump: 2024-09-16T05:16:48.321 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:48.321 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:48.321 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.321 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.322 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:48.322 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.322 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:48.322 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:48.322 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:48.322 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032effd640 10 monclient: _finish_auth 0 2024-09-16T05:16:48.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032effd640 10 monclient: _check_auth_tickets 2024-09-16T05:16:48.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:48.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032effd640 1 -- 172.21.15.114:0/4102897666 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03180a4e50 con 0x7f033009f370 2024-09-16T05:16:48.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 5 monclient: authenticate success, global_id 74367 2024-09-16T05:16:48.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:16:48.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:48.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient: get_monmap_and_config success 2024-09-16T05:16:48.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032effd640 1 -- 172.21.15.114:0/4102897666 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03180ab820 con 0x7f033009f370 2024-09-16T05:16:48.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:48.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 4 set_mon_vals no callback set 2024-09-16T05:16:48.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:48.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032effd640 10 monclient: dump: 2024-09-16T05:16:48.323 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:48.323 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:48.323 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.323 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.323 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:48.323 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.324 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:48.324 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:48.324 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:48.324 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient: shutdown 2024-09-16T05:16:48.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:48.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 -- 172.21.15.114:0/4102897666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f033009f370 msgr2=0x7f033009d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:48.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 --2- 172.21.15.114:0/4102897666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f033009f370 0x7f033009d740 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f033006be20 tx=0x7f031801c400 comp rx=0 tx=0).stop 2024-09-16T05:16:48.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 -- 172.21.15.114:0/4102897666 shutdown_connections 2024-09-16T05:16:48.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 --2- 172.21.15.114:0/4102897666 >> v2:172.21.15.149:3300/0 conn(0x7f033009dc80 0x7f033013a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 --2- 172.21.15.114:0/4102897666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f033009f370 0x7f033009d740 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 --2- 172.21.15.114:0/4102897666 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0330068250 0x7f033009d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 -- 172.21.15.114:0/4102897666 >> 172.21.15.114:0/4102897666 conn(0x7f03300a6360 msgr2=0x7f03300a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:48.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 -- 172.21.15.114:0/4102897666 shutdown_connections 2024-09-16T05:16:48.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 -- 172.21.15.114:0/4102897666 wait complete. 2024-09-16T05:16:48.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient: build_initial_monmap 2024-09-16T05:16:48.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient: monmap: 2024-09-16T05:16:48.331 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:16:48.331 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:16:48.331 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:16:48.332 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:16:48.332 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:16:48.332 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.332 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:16:48.332 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:16:48.332 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:16:48.332 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 librados: starting msgr at 2024-09-16T05:16:48.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 librados: starting objecter 2024-09-16T05:16:48.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 Processor -- start 2024-09-16T05:16:48.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 -- start start 2024-09-16T05:16:48.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 librados: setting wanted keys 2024-09-16T05:16:48.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 librados: calling monclient init 2024-09-16T05:16:48.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient: init 2024-09-16T05:16:48.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient: _reopen_session rank -1 2024-09-16T05:16:48.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:16:48.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0330068250 0x7f0330139290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient(hunting): picked mon.noname-a con 0x7f0330068250 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:16:48.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033009dc80 0x7f03301397d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient(hunting): picked mon.noname-b con 0x7f033009dc80 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:16:48.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033009f370 0x7f033013d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient(hunting): picked mon.noname-c con 0x7f033009f370 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:16:48.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:48.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f03300a0a90 con 0x7f0330068250 2024-09-16T05:16:48.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:48.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f03300a0910 con 0x7f033009dc80 2024-09-16T05:16:48.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:48.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f03300a0c10 con 0x7f033009f370 2024-09-16T05:16:48.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient(hunting): _renew_subs 2024-09-16T05:16:48.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient(hunting): authenticate will time out at 1492.950812s 2024-09-16T05:16:48.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f0334896640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033009f370 msgr2=0x7f033013d6a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:16:48.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f0334896640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033009f370 0x7f033013d6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:16:48.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0330068250 0x7f0330139290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:48.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0330068250 0x7f0330139290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53670/0 (socket says 172.21.15.114:53670) 2024-09-16T05:16:48.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 1 -- 172.21.15.114:0/3830930355 learned_addr learned my addr 172.21.15.114:0/3830930355 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:16:48.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 10 monclient(hunting): get_auth_request con 0x7f0330068250 auth_method 0 2024-09-16T05:16:48.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:48.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032f7fe640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033009dc80 0x7f03301397d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 10 monclient(hunting): handle_auth_done global_id 74370 payload 824 2024-09-16T05:16:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 1 -- 172.21.15.114:0/3830930355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033009f370 msgr2=0x7f033013d6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:16:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f033009f370 0x7f033013d6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 1 -- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033009dc80 msgr2=0x7f03301397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033009dc80 0x7f03301397d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:16:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 1 monclient: found mon.noname-a 2024-09-16T05:16:48.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:16:48.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:16:48.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 1 -- 172.21.15.114:0/3830930355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0330126f40 con 0x7f0330068250 2024-09-16T05:16:48.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032f7fe640 10 monclient: get_auth_request con 0x7f033009dc80 auth_method 0 2024-09-16T05:16:48.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032f7fe640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033009dc80 0x7f03301397d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:16:48.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032ffff640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0330068250 0x7f0330139290 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f032800b9e0 tx=0x7f032800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:48.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032cff9640 10 client.?.objecter ms_handle_connect 0x7f0330068250 2024-09-16T05:16:48.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:16:48.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032cff9640 1 -- 172.21.15.114:0/3830930355 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032800cba0 con 0x7f0330068250 2024-09-16T05:16:48.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:48.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:16:48.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032cff9640 10 monclient: dump: 2024-09-16T05:16:48.336 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:48.337 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:48.337 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.337 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.337 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:48.337 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.337 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:48.337 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:48.337 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:48.337 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032cff9640 10 monclient: _finish_auth 0 2024-09-16T05:16:48.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032cff9640 10 monclient: _check_auth_tickets 2024-09-16T05:16:48.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:16:48.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032cff9640 1 -- 172.21.15.114:0/3830930355 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0328010070 con 0x7f0330068250 2024-09-16T05:16:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:16:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032cff9640 1 -- 172.21.15.114:0/3830930355 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0328015470 con 0x7f0330068250 2024-09-16T05:16:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 5 monclient: authenticate success, global_id 74370 2024-09-16T05:16:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient: _renew_subs 2024-09-16T05:16:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 -- 172.21.15.114:0/3830930355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03300a55e0 con 0x7f0330068250 2024-09-16T05:16:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032dffb640 4 set_mon_vals no callback set 2024-09-16T05:16:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:16:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f032cff9640 10 monclient: dump: 2024-09-16T05:16:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:48.338 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:48.339 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 client.74370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:16:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient: _renew_subs 2024-09-16T05:16:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:16:48.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 -- 172.21.15.114:0/3830930355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f033013a690 con 0x7f0330068250 2024-09-16T05:16:48.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.319+0000 7f03362f9640 1 librados: init done 2024-09-16T05:16:48.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f03362f9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:48.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f03362f9640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033013e3a0 0x7f033002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f03362f9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:48.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f03362f9640 1 -- 172.21.15.114:0/3830930355 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f03300a1070 con 0x7f033013e3a0 2024-09-16T05:16:48.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 1 -- 172.21.15.114:0/3830930355 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f032801e3f0 con 0x7f0330068250 2024-09-16T05:16:48.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 1 --2- 172.21.15.114:0/3830930355 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03100377c0 0x7f0310039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032f7fe640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033013e3a0 0x7f033002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:48.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 1 -- 172.21.15.114:0/3830930355 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(358..358 src has 1..358) ==== 5021+0+0 (secure 0 0 0) 0x7f0328048e90 con 0x7f0330068250 2024-09-16T05:16:48.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 10 client.74370.objecter ms_dispatch 0x7f0330000ba0 osd_map(358..358 src has 1..358) 2024-09-16T05:16:48.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 3 client.74370.objecter handle_osd_map got epochs [358,358] > 0 2024-09-16T05:16:48.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 3 client.74370.objecter handle_osd_map decoding full epoch 358 2024-09-16T05:16:48.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032f7fe640 10 monclient: get_auth_request con 0x7f033013e3a0 auth_method 0 2024-09-16T05:16:48.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:48.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:48.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:48.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 20 client.74370.objecter dump_active .. 0 homeless 2024-09-16T05:16:48.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:48.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:48.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:48.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f0334896640 1 --2- 172.21.15.114:0/3830930355 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03100377c0 0x7f0310039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:48.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f0334896640 10 monclient: get_auth_request con 0x7f03100377c0 auth_method 0 2024-09-16T05:16:48.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032f7fe640 10 monclient(hunting): handle_auth_done global_id 64480 payload 824 2024-09-16T05:16:48.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032f7fe640 1 -- 172.21.15.114:0/3830930355 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f033011da60 con 0x7f033013e3a0 2024-09-16T05:16:48.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 10 client.74370.objecter ms_handle_connect 0x7f033013e3a0 2024-09-16T05:16:48.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 10 client.74370.objecter resend_mon_ops 2024-09-16T05:16:48.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032f7fe640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033013e3a0 0x7f033002c650 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f033009cd10 tx=0x7f03180191a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:48.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 10 client.74370.objecter ms_handle_connect 0x7f03100377c0 2024-09-16T05:16:48.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f0334896640 1 --2- 172.21.15.114:0/3830930355 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03100377c0 0x7f0310039c80 secure :-1 s=READY pgs=3532 cs=0 l=1 rev1=1 crypto rx=0x7f0320006e90 tx=0x7f0320006e20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:48.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 1 -- 172.21.15.114:0/3830930355 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03180b9030 con 0x7f033013e3a0 2024-09-16T05:16:48.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:48.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:16:48.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 10 monclient: dump: 2024-09-16T05:16:48.343 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:48.343 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:48.343 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.343 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.343 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:48.343 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.343 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:48.343 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:48.343 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:48.343 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 1 -- 172.21.15.114:0/3830930355 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f03180b1040 con 0x7f033013e3a0 2024-09-16T05:16:48.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:16:48.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:16:48.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 1 -- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033013e3a0 msgr2=0x7f033002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:48.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033013e3a0 0x7f033002c650 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f033009cd10 tx=0x7f03180191a0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 1 -- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033013e3a0 msgr2=0x7f033002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:48.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.323+0000 7f032cff9640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033013e3a0 0x7f033002c650 secure :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f033009cd10 tx=0x7f03180191a0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f03362f9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:16:48.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f03362f9640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033013e780 0x7f033002cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:16:48.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f03362f9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:16:48.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f03362f9640 1 -- 172.21.15.114:0/3830930355 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f03300008e0 con 0x7f033013e780 2024-09-16T05:16:48.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f032f7fe640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033013e780 0x7f033002cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:16:48.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f032f7fe640 10 monclient: get_auth_request con 0x7f033013e780 auth_method 0 2024-09-16T05:16:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f032f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:16:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f032f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:16:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f032f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:16:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f032f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:16:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f032f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:16:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f032f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:16:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f032f7fe640 10 monclient(hunting): handle_auth_done global_id 64483 payload 824 2024-09-16T05:16:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f032f7fe640 1 -- 172.21.15.114:0/3830930355 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0330126650 con 0x7f033013e780 2024-09-16T05:16:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f032f7fe640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033013e780 0x7f033002cb90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0318000c10 tx=0x7f03180a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:16:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f032cff9640 10 client.74370.objecter ms_handle_connect 0x7f033013e780 2024-09-16T05:16:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.331+0000 7f032cff9640 10 client.74370.objecter resend_mon_ops 2024-09-16T05:16:48.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f032cff9640 1 -- 172.21.15.114:0/3830930355 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03180c5070 con 0x7f033013e780 2024-09-16T05:16:48.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f032cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:16:48.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f032cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:16:48.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f032cff9640 10 monclient: dump: 2024-09-16T05:16:48.347 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:16:48.347 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:16:48.347 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.347 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:16:48.347 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:16:48.347 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:16:48.347 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:16:48.347 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:16:48.348 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:16:48.348 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:16:48.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f032cff9640 1 -- 172.21.15.114:0/3830930355 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f03180a4e50 con 0x7f033013e780 2024-09-16T05:16:48.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f032cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:16:48.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f032cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:16:48.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f032cff9640 1 -- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033013e780 msgr2=0x7f033002cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:48.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f032cff9640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033013e780 0x7f033002cb90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0318000c10 tx=0x7f03180a8000 comp rx=0 tx=0).stop 2024-09-16T05:16:48.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f032cff9640 1 -- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033013e780 msgr2=0x7f033002cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:16:48.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f032cff9640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f033013e780 0x7f033002cb90 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0318000c10 tx=0x7f03180a8000 comp rx=0 tx=0).stop 2024-09-16T05:16:48.349 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:16:48.349 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:16:48.349 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:16:48.349 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:16:48.349 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 44, 2024-09-16T05:16:48.349 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 292468, 2024-09-16T05:16:48.349 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:16:48.349 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:16:48.349 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-16T05:16:48.349 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:48.349 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:16:48.350 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:48.350 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:16:48.350 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:16:48.350 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:48.350 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:48.350 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:48.350 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:48.350 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:48.350 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:48.350 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:48.350 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:48.350 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:48.351 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:48.351 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:48.351 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:48.351 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:16:48.351 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:16:48.351 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:48.351 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:48.351 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:48.351 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:48.351 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:48.351 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:48.351 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:48.351 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:48.352 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:48.352 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:48.352 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:48.352 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:48.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.352 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:16:48.352 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:16:48.352 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:16:48.352 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:16:48.352 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:16:48.352 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:16:48.352 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:48.353 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:16:48.353 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:16:48.353 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:16:48.353 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:16:48.353 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:16:48.353 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:16:48.353 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:16:48.353 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:16:48.353 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:16:48.353 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:16:48.353 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:16:48.353 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:16:48.354 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:16:48.354 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:16:48.354 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:16:48.354 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:16:48.354 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:16:48.354 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:16:48.354 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:16:48.354 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:16:48.354 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:16:48.354 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:48.354 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:16:48.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.355 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:16:48.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.355 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:16:48.355 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:16:48.355 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:48.355 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:48.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.355 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:48.355 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:16:48.355 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:48.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.355 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:48.355 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:16:48.356 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:48.356 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:48.356 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:48.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.356 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:16:48.356 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:16:48.356 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:48.356 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:48.356 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:48.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.356 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:16:48.357 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:16:48.357 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:48.357 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:48.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.357 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:48.357 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:16:48.357 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:48.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.357 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:48.357 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:16:48.357 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:48.358 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:48.358 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:48.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.358 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:16:48.358 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:16:48.358 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:48.358 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:48.358 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:48.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.358 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:16:48.358 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:16:48.359 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:16:48.359 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:16:48.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.359 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:16:48.359 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:16:48.359 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:48.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.359 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:16:48.359 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:16:48.359 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:16:48.359 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:48.359 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:48.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.360 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:16:48.360 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:16:48.360 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:16:48.360 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:16:48.360 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:16:48.360 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:48.360 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:48.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.360 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:16:48.360 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:16:48.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.360 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:48.360 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:48.360 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:16:48.361 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:48.361 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:48.361 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:16:48.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.361 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:48.361 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:48.361 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:16:48.361 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:48.361 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:48.361 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:16:48.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.361 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:48.361 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:48.362 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:16:48.362 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:48.362 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:16:48.362 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:16:48.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:16:48.362 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:16:48.362 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:16:48.362 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:16:48.362 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:16:48.362 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:16:48.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:16:48.362 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:16:48.362 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:16:48.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f03362f9640 10 librados: watch_flush enter 2024-09-16T05:16:48.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f03362f9640 10 librados: watch_flush exit 2024-09-16T05:16:48.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f03362f9640 20 client.74370.objecter shutdown clearing up homeless session... 2024-09-16T05:16:48.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f03362f9640 10 client.74370.objecter successfully canceled tick 2024-09-16T05:16:48.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f03362f9640 1 -- 172.21.15.114:0/3830930355 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03100377c0 msgr2=0x7f0310039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:48.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f03362f9640 1 --2- 172.21.15.114:0/3830930355 >> v2:172.21.15.149:6812/1185531135 conn(0x7f03100377c0 0x7f0310039c80 secure :-1 s=READY pgs=3532 cs=0 l=1 rev1=1 crypto rx=0x7f0320006e90 tx=0x7f0320006e20 comp rx=0 tx=0).stop 2024-09-16T05:16:48.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f03362f9640 10 monclient: shutdown 2024-09-16T05:16:48.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f03362f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:16:48.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f03362f9640 1 -- 172.21.15.114:0/3830930355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0330068250 msgr2=0x7f0330139290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:16:48.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f03362f9640 1 --2- 172.21.15.114:0/3830930355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0330068250 0x7f0330139290 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f032800b9e0 tx=0x7f032800bea0 comp rx=0 tx=0).stop 2024-09-16T05:16:48.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f0334896640 1 -- 172.21.15.114:0/3830930355 reap_dead start 2024-09-16T05:16:48.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f0334896640 1 -- 172.21.15.114:0/3830930355 reap_dead start 2024-09-16T05:16:48.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f03362f9640 1 -- 172.21.15.114:0/3830930355 shutdown_connections 2024-09-16T05:16:48.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f03362f9640 1 -- 172.21.15.114:0/3830930355 >> 172.21.15.114:0/3830930355 conn(0x7f03300a6360 msgr2=0x7f03300a4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:16:48.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f03362f9640 1 -- 172.21.15.114:0/3830930355 shutdown_connections 2024-09-16T05:16:48.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f03362f9640 1 -- 172.21.15.114:0/3830930355 wait complete. 2024-09-16T05:16:48.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:16:48.335+0000 7f03362f9640 1 librados: shutdown 2024-09-16T05:16:48.366 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:16:48.366 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:17:35.106 INFO:tasks.ceph.osd.1.smithi114.stderr:2024-09-16T05:17:35.106+0000 7f3f502c0640 -1 snap_mapper.add_oid found existing snaps mapped on 301:18d27916:test-rados-api-smithi149-29623-80::foo:2, removing 2024-09-16T05:17:35.106 INFO:tasks.ceph.osd.0.smithi114.stderr:2024-09-16T05:17:35.106+0000 7f445340a640 -1 snap_mapper.add_oid found existing snaps mapped on 301:18d27916:test-rados-api-smithi149-29623-80::foo:2, removing 2024-09-16T05:18:05.129 INFO:tasks.workunit.client.0.smithi149.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (17997 ms) 2024-09-16T05:18:05.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-09-16T05:18:05.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (51064 ms) 2024-09-16T05:18:05.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-09-16T05:18:05.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3016 ms) 2024-09-16T05:18:05.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-09-16T05:18:05.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:18:05.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3043 ms) 2024-09-16T05:18:05.130 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-09-16T05:18:05.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (2909 ms) 2024-09-16T05:18:05.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-09-16T05:18:05.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:18:05.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3030 ms) 2024-09-16T05:18:05.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-09-16T05:18:05.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:18:05.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4037 ms) 2024-09-16T05:18:05.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-09-16T05:18:05.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:18:05.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40059 ms) 2024-09-16T05:18:05.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-09-16T05:18:05.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:18:05.131 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16261 ms) 2024-09-16T05:18:05.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-09-16T05:18:05.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:18:05.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (3662 ms) 2024-09-16T05:18:05.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-09-16T05:18:05.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:18:05.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3028 ms) 2024-09-16T05:18:05.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-09-16T05:18:05.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:18:05.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25278 ms) 2024-09-16T05:18:05.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-09-16T05:18:05.132 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:18:05.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15051 ms) 2024-09-16T05:18:05.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-09-16T05:18:05.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:18:05.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5037 ms) 2024-09-16T05:18:05.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-09-16T05:18:05.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:18:05.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (3591 ms) 2024-09-16T05:18:05.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-09-16T05:18:05.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:18:05.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: waiting for scrubs... 2024-09-16T05:18:05.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: done waiting 2024-09-16T05:18:05.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (24403 ms) 2024-09-16T05:18:05.133 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-09-16T05:18:05.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:18:05.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9689 ms) 2024-09-16T05:18:05.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-09-16T05:18:05.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:18:05.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (8978 ms) 2024-09-16T05:18:05.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-09-16T05:18:05.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:18:05.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9010 ms) 2024-09-16T05:18:05.134 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-09-16T05:18:18.367 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T05:18:18.367 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T05:18:18.367 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T05:18:18.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T05:18:18.371 INFO:tasks.ceph.mon.c:Started 2024-09-16T05:18:18.371 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:18:18.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:18:18.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 10 monclient: get_monmap_and_config 2024-09-16T05:18:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 10 monclient: build_initial_monmap 2024-09-16T05:18:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 10 monclient: monmap: 2024-09-16T05:18:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:18.508764+0000 2024-09-16T05:18:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:18.508764+0000 2024-09-16T05:18:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:18.504 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:18.505 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:18.505 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:18.505 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:18.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 1 Processor -- start 2024-09-16T05:18:18.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 1 -- start start 2024-09-16T05:18:18.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 10 monclient: init 2024-09-16T05:18:18.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:18.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-16T05:18:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78d809b6c0 addr v1:172.21.15.149:6789/0 2024-09-16T05:18:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f78d80a15d0 0x7f78d813a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 10 monclient(hunting): picked mon.noname-a con 0x7f78d80a15d0 addr v2:172.21.15.114:3300/0 2024-09-16T05:18:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f78d80a19e0 0x7f78d813cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 10 monclient(hunting): picked mon.noname-c con 0x7f78d80a19e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:18:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d809d4e0 con 0x7f78d809b6c0 2024-09-16T05:18:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f78d809e390 con 0x7f78d80a15d0 2024-09-16T05:18:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f78d809ece0 con 0x7f78d80a19e0 2024-09-16T05:18:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:18.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78dd5c4640 10 monclient(hunting): authenticate will time out at 1583.137636s 2024-09-16T05:18:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f78d80a15d0 0x7f78d813a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f78d80a15d0 0x7f78d813a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36362/0 (socket says 172.21.15.114:36362) 2024-09-16T05:18:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 1 -- 172.21.15.114:0/1469790074 learned_addr learned my addr 172.21.15.114:0/1469790074 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 10 monclient(hunting): get_auth_request con 0x7f78d80a15d0 auth_method 0 2024-09-16T05:18:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d77fe640 1 --2- 172.21.15.114:0/1469790074 >> v2:172.21.15.149:3300/0 conn(0x7f78d80a19e0 0x7f78d813cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 10 monclient(hunting): handle_auth_done global_id 74373 payload 274 2024-09-16T05:18:18.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 1 -- 172.21.15.114:0/1469790074 >> v2:172.21.15.149:3300/0 conn(0x7f78d80a19e0 msgr2=0x7f78d813cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 1 --2- 172.21.15.114:0/1469790074 >> v2:172.21.15.149:3300/0 conn(0x7f78d80a19e0 0x7f78d813cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 1 -- 172.21.15.114:0/1469790074 >> v1:172.21.15.149:6789/0 conn(0x7f78d809b6c0 legacy=0x7f78d80a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 10 monclient: _finish_hunting 0 2024-09-16T05:18:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 1 monclient: found mon.noname-a 2024-09-16T05:18:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.506+0000 7f78d67fc640 1 -- 172.21.15.114:0/1469790074 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d809f430 con 0x7f78d80a15d0 2024-09-16T05:18:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d67fc640 1 --2- 172.21.15.114:0/1469790074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d80a15d0 0x7f78d813a650 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f78d809d220 tx=0x7f78c801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcbcba7021f844cf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d5ffb640 1 -- 172.21.15.114:0/1469790074 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c80a5710 con 0x7f78d80a15d0 2024-09-16T05:18:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:18.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d5ffb640 10 monclient: dump: 2024-09-16T05:18:18.512 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:18.512 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:18.512 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:18.512 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:18.512 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:18.512 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:18.512 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:18.512 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:18.512 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:18.512 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:18.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:18:18.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:18:18.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:18.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d5ffb640 1 -- 172.21.15.114:0/1469790074 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78c80a58b0 con 0x7f78d80a15d0 2024-09-16T05:18:18.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 5 monclient: authenticate success, global_id 74373 2024-09-16T05:18:18.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:18.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:18.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:18.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d5ffb640 1 -- 172.21.15.114:0/1469790074 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c80a5c80 con 0x7f78d80a15d0 2024-09-16T05:18:18.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 4 set_mon_vals no callback set 2024-09-16T05:18:18.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:18.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:18.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d5ffb640 10 monclient: dump: 2024-09-16T05:18:18.514 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:18.514 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:18.514 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:18.514 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:18.514 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:18.514 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:18.514 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:18.514 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:18.514 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:18.514 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:18.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient: shutdown 2024-09-16T05:18:18.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:18.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 -- 172.21.15.114:0/1469790074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d80a15d0 msgr2=0x7f78d813a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:18.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 --2- 172.21.15.114:0/1469790074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d80a15d0 0x7f78d813a650 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f78d809d220 tx=0x7f78c801c960 comp rx=0 tx=0).stop 2024-09-16T05:18:18.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 -- 172.21.15.114:0/1469790074 shutdown_connections 2024-09-16T05:18:18.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 --2- 172.21.15.114:0/1469790074 >> v2:172.21.15.149:3300/0 conn(0x7f78d80a19e0 0x7f78d813cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:18.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 --2- 172.21.15.114:0/1469790074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d80a15d0 0x7f78d813a650 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:18.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 -- 172.21.15.114:0/1469790074 >> 172.21.15.114:0/1469790074 conn(0x7f78d80a1fb0 msgr2=0x7f78d80a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:18.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 -- 172.21.15.114:0/1469790074 shutdown_connections 2024-09-16T05:18:18.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 -- 172.21.15.114:0/1469790074 wait complete. 2024-09-16T05:18:18.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient: build_initial_monmap 2024-09-16T05:18:18.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient: monmap: 2024-09-16T05:18:18.520 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:18.520 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:18.520 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:18.520 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:18.521 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:18.521 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:18.521 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:18.521 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:18.521 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:18.521 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:18.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 librados: starting msgr at 2024-09-16T05:18:18.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 librados: starting objecter 2024-09-16T05:18:18.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 Processor -- start 2024-09-16T05:18:18.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 -- start start 2024-09-16T05:18:18.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 librados: setting wanted keys 2024-09-16T05:18:18.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 librados: calling monclient init 2024-09-16T05:18:18.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient: init 2024-09-16T05:18:18.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:18.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:18:18.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d809b6c0 0x7f78d8098450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:18.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient(hunting): picked mon.noname-a con 0x7f78d809b6c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:18.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f78d80a15d0 0x7f78d8091b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:18.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient(hunting): picked mon.noname-b con 0x7f78d80a15d0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:18.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78d80a19e0 0x7f78d8092070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:18.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient(hunting): picked mon.noname-c con 0x7f78d80a19e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:18.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:18.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f78d813f900 con 0x7f78d809b6c0 2024-09-16T05:18:18.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:18.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f78d809cad0 con 0x7f78d80a15d0 2024-09-16T05:18:18.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:18.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f78d813fa80 con 0x7f78d80a19e0 2024-09-16T05:18:18.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:18.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient(hunting): authenticate will time out at 1583.142578s 2024-09-16T05:18:18.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d809b6c0 0x7f78d8098450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:18.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d809b6c0 0x7f78d8098450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36374/0 (socket says 172.21.15.114:36374) 2024-09-16T05:18:18.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 1 -- 172.21.15.114:0/1959838863 learned_addr learned my addr 172.21.15.114:0/1959838863 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:18.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 10 monclient(hunting): get_auth_request con 0x7f78d809b6c0 auth_method 0 2024-09-16T05:18:18.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d77fe640 1 --2- 172.21.15.114:0/1959838863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78d80a19e0 0x7f78d8092070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:18.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:18.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:18.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:18.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d77fe640 10 monclient(hunting): get_auth_request con 0x7f78d80a19e0 auth_method 0 2024-09-16T05:18:18.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:18.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:18.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:18.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:18.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:18.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:18.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d77fe640 1 -- 172.21.15.114:0/1959838863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78d80a19e0 msgr2=0x7f78d8092070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T05:18:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d77fe640 1 -- 172.21.15.114:0/1959838863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78d80a19e0 msgr2=0x7f78d8092070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:18:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d77fe640 1 --2- 172.21.15.114:0/1959838863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78d80a19e0 0x7f78d8092070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d67fc640 1 --2- 172.21.15.114:0/1959838863 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f78d80a15d0 0x7f78d8091b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d77fe640 1 --2- 172.21.15.114:0/1959838863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78d80a19e0 0x7f78d8092070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d67fc640 10 monclient(hunting): get_auth_request con 0x7f78d80a15d0 auth_method 0 2024-09-16T05:18:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 10 monclient(hunting): handle_auth_done global_id 74376 payload 824 2024-09-16T05:18:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 1 -- 172.21.15.114:0/1959838863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78d80a19e0 msgr2=0x7f78d8092070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:18:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 1 --2- 172.21.15.114:0/1959838863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78d80a19e0 0x7f78d8092070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 1 -- 172.21.15.114:0/1959838863 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f78d80a15d0 msgr2=0x7f78d8091b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 1 --2- 172.21.15.114:0/1959838863 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f78d80a15d0 0x7f78d8091b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:18:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 1 monclient: found mon.noname-a 2024-09-16T05:18:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 1 -- 172.21.15.114:0/1959838863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d8098990 con 0x7f78d809b6c0 2024-09-16T05:18:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d67fc640 1 --2- 172.21.15.114:0/1959838863 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f78d80a15d0 0x7f78d8091b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:18:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d6ffd640 1 --2- 172.21.15.114:0/1959838863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d809b6c0 0x7f78d8098450 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f78cc00b410 tx=0x7f78cc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:18.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78bf7fe640 10 client.?.objecter ms_handle_connect 0x7f78d809b6c0 2024-09-16T05:18:18.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78bf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:18.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78bf7fe640 1 -- 172.21.15.114:0/1959838863 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc015030 con 0x7f78d809b6c0 2024-09-16T05:18:18.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:18.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78bf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:18.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78bf7fe640 10 monclient: dump: 2024-09-16T05:18:18.528 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:18.528 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:18.528 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:18.528 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:18.528 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:18.528 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:18.528 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:18.529 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:18.529 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:18.529 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:18.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78bf7fe640 10 monclient: _finish_auth 0 2024-09-16T05:18:18.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78bf7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:18:18.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:18.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78bf7fe640 1 -- 172.21.15.114:0/1959838863 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78cc004020 con 0x7f78d809b6c0 2024-09-16T05:18:18.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78bf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:18.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78bf7fe640 1 -- 172.21.15.114:0/1959838863 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc004410 con 0x7f78d809b6c0 2024-09-16T05:18:18.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:18.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:18.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78bf7fe640 10 monclient: dump: 2024-09-16T05:18:18.530 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:18.531 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:18.531 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:18.531 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:18.531 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:18.531 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:18.531 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:18.531 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:18.532 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:18.532 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:18.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 5 monclient: authenticate success, global_id 74376 2024-09-16T05:18:18.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient: _renew_subs 2024-09-16T05:18:18.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:18.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78d57fa640 4 set_mon_vals no callback set 2024-09-16T05:18:18.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 -- 172.21.15.114:0/1959838863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78d80957e0 con 0x7f78d809b6c0 2024-09-16T05:18:18.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 client.74376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:18.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient: _renew_subs 2024-09-16T05:18:18.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:18.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 -- 172.21.15.114:0/1959838863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78d809c6a0 con 0x7f78d809b6c0 2024-09-16T05:18:18.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.510+0000 7f78dd5c4640 1 librados: init done 2024-09-16T05:18:18.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.514+0000 7f78dd5c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:18.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.514+0000 7f78dd5c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:18.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.514+0000 7f78dd5c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:18.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.514+0000 7f78dd5c4640 1 -- 172.21.15.114:0/1959838863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78d80940b0 con 0x7f78d809b6c0 2024-09-16T05:18:18.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.514+0000 7f78bf7fe640 1 -- 172.21.15.114:0/1959838863 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f78cc018070 con 0x7f78d809b6c0 2024-09-16T05:18:18.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.514+0000 7f78bf7fe640 1 --2- 172.21.15.114:0/1959838863 >> v2:172.21.15.149:6812/1185531135 conn(0x7f78c4037810 0x7f78c4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:18.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.514+0000 7f78bf7fe640 1 -- 172.21.15.114:0/1959838863 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(421..421 src has 1..421) ==== 5021+0+0 (secure 0 0 0) 0x7f78cc048850 con 0x7f78d809b6c0 2024-09-16T05:18:18.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.514+0000 7f78bf7fe640 10 client.74376.objecter ms_dispatch 0x7f78d8000ba0 osd_map(421..421 src has 1..421) 2024-09-16T05:18:18.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.514+0000 7f78bf7fe640 3 client.74376.objecter handle_osd_map got epochs [421,421] > 0 2024-09-16T05:18:18.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.514+0000 7f78bf7fe640 3 client.74376.objecter handle_osd_map decoding full epoch 421 2024-09-16T05:18:18.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.514+0000 7f78bf7fe640 20 client.74376.objecter dump_active .. 0 homeless 2024-09-16T05:18:18.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.514+0000 7f78d67fc640 1 --2- 172.21.15.114:0/1959838863 >> v2:172.21.15.149:6812/1185531135 conn(0x7f78c4037810 0x7f78c4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:18.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.514+0000 7f78d67fc640 10 monclient: get_auth_request con 0x7f78c4037810 auth_method 0 2024-09-16T05:18:18.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.514+0000 7f78bf7fe640 10 client.74376.objecter ms_handle_connect 0x7f78c4037810 2024-09-16T05:18:18.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.514+0000 7f78d67fc640 1 --2- 172.21.15.114:0/1959838863 >> v2:172.21.15.149:6812/1185531135 conn(0x7f78c4037810 0x7f78c4039cd0 secure :-1 s=READY pgs=3534 cs=0 l=1 rev1=1 crypto rx=0x7f78c80ae000 tx=0x7f78c80a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:18.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.518+0000 7f78bf7fe640 1 -- 172.21.15.114:0/1959838863 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f78d8098990 con 0x7f78d809b6c0 2024-09-16T05:18:18.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.518+0000 7f78bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:18.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.518+0000 7f78bf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:18.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:18:18.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:18.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:18.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 1 -- 172.21.15.114:0/1959838863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f78d80a0e40 con 0x7f78d809b6c0 2024-09-16T05:18:18.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78bf7fe640 1 -- 172.21.15.114:0/1959838863 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f78cc0199f0 con 0x7f78d809b6c0 2024-09-16T05:18:18.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:18.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78bf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:18.741 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":44,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":92,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:18:18.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 10 librados: watch_flush enter 2024-09-16T05:18:18.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 10 librados: watch_flush exit 2024-09-16T05:18:18.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 20 client.74376.objecter shutdown clearing up homeless session... 2024-09-16T05:18:18.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 10 client.74376.objecter successfully canceled tick 2024-09-16T05:18:18.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 1 -- 172.21.15.114:0/1959838863 >> v2:172.21.15.149:6812/1185531135 conn(0x7f78c4037810 msgr2=0x7f78c4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:18.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 1 --2- 172.21.15.114:0/1959838863 >> v2:172.21.15.149:6812/1185531135 conn(0x7f78c4037810 0x7f78c4039cd0 secure :-1 s=READY pgs=3534 cs=0 l=1 rev1=1 crypto rx=0x7f78c80ae000 tx=0x7f78c80a7070 comp rx=0 tx=0).stop 2024-09-16T05:18:18.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 10 monclient: shutdown 2024-09-16T05:18:18.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:18.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 1 -- 172.21.15.114:0/1959838863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d809b6c0 msgr2=0x7f78d8098450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:18.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 1 --2- 172.21.15.114:0/1959838863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d809b6c0 0x7f78d8098450 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f78cc00b410 tx=0x7f78cc00b8d0 comp rx=0 tx=0).stop 2024-09-16T05:18:18.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 1 -- 172.21.15.114:0/1959838863 shutdown_connections 2024-09-16T05:18:18.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 1 --2- 172.21.15.114:0/1959838863 >> v2:172.21.15.149:6812/1185531135 conn(0x7f78c4037810 0x7f78c4039cd0 unknown :-1 s=CLOSED pgs=3534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:18.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 1 --2- 172.21.15.114:0/1959838863 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78d80a19e0 0x7f78d8092070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:18.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 1 --2- 172.21.15.114:0/1959838863 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f78d80a15d0 0x7f78d8091b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:18.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 1 --2- 172.21.15.114:0/1959838863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d809b6c0 0x7f78d8098450 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:18.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 1 -- 172.21.15.114:0/1959838863 >> 172.21.15.114:0/1959838863 conn(0x7f78d80a1fb0 msgr2=0x7f78d8071330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:18.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 1 -- 172.21.15.114:0/1959838863 shutdown_connections 2024-09-16T05:18:18.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 1 -- 172.21.15.114:0/1959838863 wait complete. 2024-09-16T05:18:18.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:18.742+0000 7f78dd5c4640 1 librados: shutdown 2024-09-16T05:18:21.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:18:21.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 10 monclient: get_monmap_and_config 2024-09-16T05:18:21.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 10 monclient: build_initial_monmap 2024-09-16T05:18:21.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 10 monclient: monmap: 2024-09-16T05:18:21.885 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:21.885 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:21.885 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:21.890288+0000 2024-09-16T05:18:21.886 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:21.890288+0000 2024-09-16T05:18:21.886 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:21.886 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:21.886 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:21.886 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:21.886 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:21.886 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:21.886 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:21.886 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:21.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 1 Processor -- start 2024-09-16T05:18:21.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 1 -- start start 2024-09-16T05:18:21.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 10 monclient: init 2024-09-16T05:18:21.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:21.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-16T05:18:21.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcbe40a0ec0 0x7fcbe409dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:21.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 10 monclient(hunting): picked mon.noname-a con 0x7fcbe40a0ec0 addr v2:172.21.15.114:3300/0 2024-09-16T05:18:21.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe409e000 0x7fcbe409e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:21.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 10 monclient(hunting): picked mon.noname-b con 0x7fcbe409e000 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:21.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcbe4138630 addr v1:172.21.15.149:6789/0 2024-09-16T05:18:21.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbe409e950 con 0x7fcbe4138630 2024-09-16T05:18:21.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:21.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fcbe40a12a0 con 0x7fcbe40a0ec0 2024-09-16T05:18:21.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:21.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe40994b0 con 0x7fcbe409e000 2024-09-16T05:18:21.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:21.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe9b89640 10 monclient(hunting): authenticate will time out at 1586.519585s 2024-09-16T05:18:21.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe37fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcbe40a0ec0 0x7fcbe409dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:21.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe37fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcbe40a0ec0 0x7fcbe409dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36412/0 (socket says 172.21.15.114:36412) 2024-09-16T05:18:21.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe37fe640 1 -- 172.21.15.114:0/3569439825 learned_addr learned my addr 172.21.15.114:0/3569439825 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:21.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe37fe640 10 monclient(hunting): get_auth_request con 0x7fcbe40a0ec0 auth_method 0 2024-09-16T05:18:21.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:21.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:21.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:21.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe2ffd640 1 --2- 172.21.15.114:0/3569439825 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe409e000 0x7fcbe409e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:21.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe2ffd640 10 monclient(hunting): get_auth_request con 0x7fcbe409e000 auth_method 0 2024-09-16T05:18:21.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:21.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:21.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.886+0000 7fcbe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:21.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:21.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:21.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:21.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe2ffd640 1 -- 172.21.15.114:0/3569439825 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe409e000 msgr2=0x7fcbe409e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:18:21.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe2ffd640 1 -- 172.21.15.114:0/3569439825 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe409e000 msgr2=0x7fcbe409e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:21.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe2ffd640 1 --2- 172.21.15.114:0/3569439825 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe409e000 0x7fcbe409e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:21.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe2ffd640 1 --2- 172.21.15.114:0/3569439825 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe409e000 0x7fcbe409e410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:21.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe37fe640 10 monclient(hunting): handle_auth_done global_id 74379 payload 274 2024-09-16T05:18:21.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe37fe640 1 -- 172.21.15.114:0/3569439825 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe409e000 msgr2=0x7fcbe409e410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:18:21.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe37fe640 1 --2- 172.21.15.114:0/3569439825 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe409e000 0x7fcbe409e410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:21.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe37fe640 1 -- 172.21.15.114:0/3569439825 >> v1:172.21.15.149:6789/0 conn(0x7fcbe4138630 legacy=0x7fcbe413aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:21.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:18:21.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe37fe640 1 monclient: found mon.noname-a 2024-09-16T05:18:21.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:21.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe37fe640 1 -- 172.21.15.114:0/3569439825 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe409a760 con 0x7fcbe40a0ec0 2024-09-16T05:18:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe37fe640 1 --2- 172.21.15.114:0/3569439825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbe40a0ec0 0x7fcbe409dac0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400d150 tx=0x7fcbd401eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c5a7c20dfe9564 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe27fc640 1 -- 172.21.15.114:0/3569439825 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400b2d0 con 0x7fcbe40a0ec0 2024-09-16T05:18:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe27fc640 10 monclient: dump: 2024-09-16T05:18:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:21.894 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe27fc640 10 monclient: _finish_auth 0 2024-09-16T05:18:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:18:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe27fc640 1 -- 172.21.15.114:0/3569439825 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbd400b470 con 0x7fcbe40a0ec0 2024-09-16T05:18:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 5 monclient: authenticate success, global_id 74379 2024-09-16T05:18:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe27fc640 1 -- 172.21.15.114:0/3569439825 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd40a7a60 con 0x7fcbe40a0ec0 2024-09-16T05:18:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 4 set_mon_vals no callback set 2024-09-16T05:18:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:21.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:21.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe27fc640 10 monclient: dump: 2024-09-16T05:18:21.895 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:21.895 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:21.895 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:21.895 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:21.895 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:21.895 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:21.895 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:21.895 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:21.895 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:21.895 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:21.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 10 monclient: shutdown 2024-09-16T05:18:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 1 -- 172.21.15.114:0/3569439825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbe40a0ec0 msgr2=0x7fcbe409dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 1 --2- 172.21.15.114:0/3569439825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbe40a0ec0 0x7fcbe409dac0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400d150 tx=0x7fcbd401eb80 comp rx=0 tx=0).stop 2024-09-16T05:18:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 1 -- 172.21.15.114:0/3569439825 shutdown_connections 2024-09-16T05:18:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 1 --2- 172.21.15.114:0/3569439825 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe409e000 0x7fcbe409e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 1 --2- 172.21.15.114:0/3569439825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbe40a0ec0 0x7fcbe409dac0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 1 -- 172.21.15.114:0/3569439825 >> 172.21.15.114:0/3569439825 conn(0x7fcbe40a1fb0 msgr2=0x7fcbe40a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 1 -- 172.21.15.114:0/3569439825 shutdown_connections 2024-09-16T05:18:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 1 -- 172.21.15.114:0/3569439825 wait complete. 2024-09-16T05:18:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 10 monclient: build_initial_monmap 2024-09-16T05:18:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.890+0000 7fcbe9b89640 10 monclient: monmap: 2024-09-16T05:18:21.901 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:21.902 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 librados: starting msgr at 2024-09-16T05:18:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 librados: starting objecter 2024-09-16T05:18:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 Processor -- start 2024-09-16T05:18:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 -- start start 2024-09-16T05:18:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 librados: setting wanted keys 2024-09-16T05:18:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 librados: calling monclient init 2024-09-16T05:18:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient: init 2024-09-16T05:18:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:18:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbe409e000 0x7fcbe4137850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient(hunting): picked mon.noname-a con 0x7fcbe409e000 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe40a0ec0 0x7fcbe4137d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient(hunting): picked mon.noname-c con 0x7fcbe40a0ec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcbe4138630 0x7fcbe408d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient(hunting): picked mon.noname-b con 0x7fcbe4138630 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe409c3b0 con 0x7fcbe409e000 2024-09-16T05:18:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:21.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe409c230 con 0x7fcbe4138630 2024-09-16T05:18:21.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:21.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe409d1c0 con 0x7fcbe40a0ec0 2024-09-16T05:18:21.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:21.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient(hunting): authenticate will time out at 1586.524813s 2024-09-16T05:18:21.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbe409e000 0x7fcbe4137850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:21.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbe409e000 0x7fcbe4137850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36426/0 (socket says 172.21.15.114:36426) 2024-09-16T05:18:21.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe2ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe40a0ec0 0x7fcbe4137d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:21.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 1 -- 172.21.15.114:0/1483542237 learned_addr learned my addr 172.21.15.114:0/1483542237 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:21.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 10 monclient(hunting): get_auth_request con 0x7fcbe409e000 auth_method 0 2024-09-16T05:18:21.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:21.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:21.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:21.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe2ffd640 10 monclient(hunting): get_auth_request con 0x7fcbe40a0ec0 auth_method 0 2024-09-16T05:18:21.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:21.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:21.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:21.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:21.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:21.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:21.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe2ffd640 1 -- 172.21.15.114:0/1483542237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe40a0ec0 msgr2=0x7fcbe4137d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:18:21.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe2ffd640 1 -- 172.21.15.114:0/1483542237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe40a0ec0 msgr2=0x7fcbe4137d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:18:21.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe2ffd640 1 --2- 172.21.15.114:0/1483542237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe40a0ec0 0x7fcbe4137d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:21.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe2ffd640 1 --2- 172.21.15.114:0/1483542237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe40a0ec0 0x7fcbe4137d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:21.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 10 monclient(hunting): handle_auth_done global_id 74382 payload 824 2024-09-16T05:18:21.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 1 -- 172.21.15.114:0/1483542237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe40a0ec0 msgr2=0x7fcbe4137d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:18:21.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 1 --2- 172.21.15.114:0/1483542237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe40a0ec0 0x7fcbe4137d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:21.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe3fff640 1 --2- 172.21.15.114:0/1483542237 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcbe4138630 0x7fcbe408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:21.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 1 -- 172.21.15.114:0/1483542237 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcbe4138630 msgr2=0x7fcbe408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:21.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 1 --2- 172.21.15.114:0/1483542237 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcbe4138630 0x7fcbe408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:21.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:18:21.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 1 monclient: found mon.noname-a 2024-09-16T05:18:21.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:21.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:21.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 1 -- 172.21.15.114:0/1483542237 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe40900e0 con 0x7fcbe409e000 2024-09-16T05:18:21.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe3fff640 10 monclient: get_auth_request con 0x7fcbe4138630 auth_method 0 2024-09-16T05:18:21.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe3fff640 1 --2- 172.21.15.114:0/1483542237 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcbe4138630 0x7fcbe408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:18:21.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe37fe640 1 --2- 172.21.15.114:0/1483542237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbe409e000 0x7fcbe4137850 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcbd401c7b0 tx=0x7fcbd401b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:21.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 10 client.?.objecter ms_handle_connect 0x7fcbe409e000 2024-09-16T05:18:21.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:21.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 1 -- 172.21.15.114:0/1483542237 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd40be020 con 0x7fcbe409e000 2024-09-16T05:18:21.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:21.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:21.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 10 monclient: dump: 2024-09-16T05:18:21.907 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:21.907 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:21.907 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:21.907 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:21.907 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:21.907 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:21.907 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:21.908 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 10 monclient: _finish_auth 0 2024-09-16T05:18:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 10 monclient: _check_auth_tickets 2024-09-16T05:18:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 1 -- 172.21.15.114:0/1483542237 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbd4009840 con 0x7fcbe409e000 2024-09-16T05:18:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 1 -- 172.21.15.114:0/1483542237 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4009c30 con 0x7fcbe409e000 2024-09-16T05:18:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 5 monclient: authenticate success, global_id 74382 2024-09-16T05:18:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient: _renew_subs 2024-09-16T05:18:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe1ffb640 4 set_mon_vals no callback set 2024-09-16T05:18:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 -- 172.21.15.114:0/1483542237 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbe40903c0 con 0x7fcbe409e000 2024-09-16T05:18:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 10 monclient: dump: 2024-09-16T05:18:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:21.910 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:21.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 client.74382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:21.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient: _renew_subs 2024-09-16T05:18:21.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:21.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 -- 172.21.15.114:0/1483542237 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbe408f180 con 0x7fcbe409e000 2024-09-16T05:18:21.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 librados: init done 2024-09-16T05:18:21.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:21.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 1 -- 172.21.15.114:0/1483542237 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcbd40c1020 con 0x7fcbe409e000 2024-09-16T05:18:21.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:21.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:21.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe9b89640 1 -- 172.21.15.114:0/1483542237 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbe4076970 con 0x7fcbe409e000 2024-09-16T05:18:21.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 1 --2- 172.21.15.114:0/1483542237 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcbc4037810 0x7fcbc4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:21.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 1 -- 172.21.15.114:0/1483542237 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(424..424 src has 1..424) ==== 5406+0+0 (secure 0 0 0) 0x7fcbd40e6550 con 0x7fcbe409e000 2024-09-16T05:18:21.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 10 client.74382.objecter ms_dispatch 0x7fcbe4000ba0 osd_map(424..424 src has 1..424) 2024-09-16T05:18:21.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 3 client.74382.objecter handle_osd_map got epochs [424,424] > 0 2024-09-16T05:18:21.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 3 client.74382.objecter handle_osd_map decoding full epoch 424 2024-09-16T05:18:21.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbdbfff640 20 client.74382.objecter dump_active .. 0 homeless 2024-09-16T05:18:21.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe2ffd640 1 --2- 172.21.15.114:0/1483542237 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcbc4037810 0x7fcbc4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:21.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.894+0000 7fcbe2ffd640 10 monclient: get_auth_request con 0x7fcbc4037810 auth_method 0 2024-09-16T05:18:21.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.898+0000 7fcbdbfff640 10 client.74382.objecter ms_handle_connect 0x7fcbc4037810 2024-09-16T05:18:21.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.898+0000 7fcbe2ffd640 1 --2- 172.21.15.114:0/1483542237 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcbc4037810 0x7fcbc4039cd0 secure :-1 s=READY pgs=3535 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4098fc0 tx=0x7fcbcc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:21.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.898+0000 7fcbdbfff640 1 -- 172.21.15.114:0/1483542237 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fcbd4018800 con 0x7fcbe409e000 2024-09-16T05:18:21.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.898+0000 7fcbdbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:21.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:21.898+0000 7fcbdbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:22.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.126+0000 7fcbe9b89640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:18:22.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.126+0000 7fcbe9b89640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:22.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.126+0000 7fcbe9b89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:22.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.126+0000 7fcbe9b89640 1 -- 172.21.15.114:0/1483542237 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcbe4090a60 con 0x7fcbe409e000 2024-09-16T05:18:22.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbdbfff640 1 -- 172.21.15.114:0/1483542237 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fcbd40af3f0 con 0x7fcbe409e000 2024-09-16T05:18:22.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbdbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:22.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbdbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:22.127 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":44,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":95,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:18:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 10 librados: watch_flush enter 2024-09-16T05:18:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 10 librados: watch_flush exit 2024-09-16T05:18:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 20 client.74382.objecter shutdown clearing up homeless session... 2024-09-16T05:18:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 10 client.74382.objecter successfully canceled tick 2024-09-16T05:18:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 1 -- 172.21.15.114:0/1483542237 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcbc4037810 msgr2=0x7fcbc4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 1 --2- 172.21.15.114:0/1483542237 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcbc4037810 0x7fcbc4039cd0 secure :-1 s=READY pgs=3535 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4098fc0 tx=0x7fcbcc008040 comp rx=0 tx=0).stop 2024-09-16T05:18:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 10 monclient: shutdown 2024-09-16T05:18:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 1 -- 172.21.15.114:0/1483542237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbe409e000 msgr2=0x7fcbe4137850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 1 --2- 172.21.15.114:0/1483542237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbe409e000 0x7fcbe4137850 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcbd401c7b0 tx=0x7fcbd401b920 comp rx=0 tx=0).stop 2024-09-16T05:18:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 1 -- 172.21.15.114:0/1483542237 shutdown_connections 2024-09-16T05:18:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 1 --2- 172.21.15.114:0/1483542237 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcbc4037810 0x7fcbc4039cd0 unknown :-1 s=CLOSED pgs=3535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:22.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 1 --2- 172.21.15.114:0/1483542237 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcbe4138630 0x7fcbe408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:22.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 1 --2- 172.21.15.114:0/1483542237 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcbe40a0ec0 0x7fcbe4137d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:22.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 1 --2- 172.21.15.114:0/1483542237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbe409e000 0x7fcbe4137850 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:22.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 1 -- 172.21.15.114:0/1483542237 >> 172.21.15.114:0/1483542237 conn(0x7fcbe40a1fb0 msgr2=0x7fcbe406d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:22.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 1 -- 172.21.15.114:0/1483542237 shutdown_connections 2024-09-16T05:18:22.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 1 -- 172.21.15.114:0/1483542237 wait complete. 2024-09-16T05:18:22.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:22.130+0000 7fcbe9b89640 1 librados: shutdown 2024-09-16T05:18:25.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:18:25.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 10 monclient: get_monmap_and_config 2024-09-16T05:18:25.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 10 monclient: build_initial_monmap 2024-09-16T05:18:25.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 10 monclient: monmap: 2024-09-16T05:18:25.294 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:25.294 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:25.295 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:25.278203+0000 2024-09-16T05:18:25.295 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:25.278203+0000 2024-09-16T05:18:25.295 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:25.295 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:25.295 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:25.296 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:25.296 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:25.296 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:25.296 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:25.296 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:25.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 1 Processor -- start 2024-09-16T05:18:25.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 1 -- start start 2024-09-16T05:18:25.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 10 monclient: init 2024-09-16T05:18:25.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:25.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-16T05:18:25.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54ec098500 addr v1:172.21.15.149:6789/0 2024-09-16T05:18:25.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54ec1368a0 addr v1:172.21.15.114:6789/0 2024-09-16T05:18:25.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f54ec13a3b0 0x7f54ec13c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:25.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 10 monclient(hunting): picked mon.noname-c con 0x7f54ec13a3b0 addr v2:172.21.15.149:3300/0 2024-09-16T05:18:25.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54ec09db80 con 0x7f54ec1368a0 2024-09-16T05:18:25.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54ec09e2d0 con 0x7f54ec098500 2024-09-16T05:18:25.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:25.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f54ec098ff0 con 0x7f54ec13a3b0 2024-09-16T05:18:25.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:25.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f3943640 10 monclient(hunting): authenticate will time out at 1589.907226s 2024-09-16T05:18:25.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f0ede640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f54ec1368a0 0x7f54ec138c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46720/0 (socket says 172.21.15.114:46720) 2024-09-16T05:18:25.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f0ede640 1 -- 172.21.15.114:0/3627973887 learned_addr learned my addr 172.21.15.114:0/3627973887 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:18:25.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 1 -- 172.21.15.114:0/3627973887 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1598799881 0 0) 0x7f54ec09db80 con 0x7f54ec1368a0 2024-09-16T05:18:25.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:18:25.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:25.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:25.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 10 monclient(hunting): my global_id is 74385 2024-09-16T05:18:25.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 1 -- 172.21.15.114:0/3627973887 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54d4004640 con 0x7f54ec1368a0 2024-09-16T05:18:25.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f1ee0640 1 --2- 172.21.15.114:0/3627973887 >> v2:172.21.15.149:3300/0 conn(0x7f54ec13a3b0 0x7f54ec13c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:25.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f1ee0640 10 monclient(hunting): get_auth_request con 0x7f54ec13a3b0 auth_method 0 2024-09-16T05:18:25.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f1ee0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:25.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f1ee0640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:25.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54f1ee0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:25.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 1 -- 172.21.15.114:0/3627973887 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 430054291 0 0) 0x7f54d4004640 con 0x7f54ec1368a0 2024-09-16T05:18:25.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 1 -- 172.21.15.114:0/3627973887 >> v2:172.21.15.149:3300/0 conn(0x7f54ec13a3b0 msgr2=0x7f54ec13c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:25.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 1 --2- 172.21.15.114:0/3627973887 >> v2:172.21.15.149:3300/0 conn(0x7f54ec13a3b0 0x7f54ec13c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 1 -- 172.21.15.114:0/3627973887 >> v1:172.21.15.149:6789/0 conn(0x7f54ec098500 legacy=0x7f54ec0988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 10 monclient: _finish_hunting 0 2024-09-16T05:18:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:18:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:18:25.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 1 -- 172.21.15.114:0/3627973887 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54ec09f270 con 0x7f54ec1368a0 2024-09-16T05:18:25.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 10 monclient: _finish_auth 0 2024-09-16T05:18:25.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 10 monclient: _check_auth_tickets 2024-09-16T05:18:25.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:25.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 1 -- 172.21.15.114:0/3627973887 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f54e8001ba0 con 0x7f54ec1368a0 2024-09-16T05:18:25.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:25.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:18:25.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 10 monclient: dump: 2024-09-16T05:18:25.305 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:25.305 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:25.305 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:25.306 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:25.306 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:25.306 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:25.306 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:25.306 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:25.306 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:25.307 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:25.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:18:25.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:25.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 1 -- 172.21.15.114:0/3627973887 >> v1:172.21.15.114:6789/0 conn(0x7f54ec1368a0 legacy=0x7f54ec138c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:25.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:18:25.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.274+0000 7f54e3fff640 1 --2- 172.21.15.114:0/3627973887 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54d4004ad0 0x7f54d4004ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:25.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 10 monclient(hunting): picked mon.c con 0x7f54d4004ad0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:25.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 1 --2- 172.21.15.114:0/3627973887 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f54d4005920 0x7f54d4005da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:25.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 10 monclient(hunting): picked mon.b con 0x7f54d4005920 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:25.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 1 --2- 172.21.15.114:0/3627973887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54d40064e0 0x7f54d400cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:25.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 10 monclient(hunting): picked mon.a con 0x7f54d40064e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:25.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:25.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 1 -- 172.21.15.114:0/3627973887 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f54ec098ff0 con 0x7f54d40064e0 2024-09-16T05:18:25.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:25.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 1 -- 172.21.15.114:0/3627973887 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f54d400d370 con 0x7f54d4005920 2024-09-16T05:18:25.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:25.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 1 -- 172.21.15.114:0/3627973887 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f54d400d570 con 0x7f54d4004ad0 2024-09-16T05:18:25.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:25.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:18:25.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f1ee0640 1 --2- 172.21.15.114:0/3627973887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54d40064e0 0x7f54d400cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:25.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 1 --2- 172.21.15.114:0/3627973887 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54d4004ad0 0x7f54d4004ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:25.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f1ee0640 10 monclient(hunting): get_auth_request con 0x7f54d40064e0 auth_method 0 2024-09-16T05:18:25.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f1ee0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:25.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f1ee0640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:25.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f1ee0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:25.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 10 monclient(hunting): get_auth_request con 0x7f54d4004ad0 auth_method 0 2024-09-16T05:18:25.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:25.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:25.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:25.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f1ee0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:25.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f1ee0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:25.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f1ee0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:18:25.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:25.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:25.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:18:25.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f0ede640 1 --2- 172.21.15.114:0/3627973887 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f54d4005920 0x7f54d4005da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:25.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f0ede640 10 monclient(hunting): get_auth_request con 0x7f54d4005920 auth_method 0 2024-09-16T05:18:25.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f0ede640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:25.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f0ede640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:25.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f0ede640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:25.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 10 monclient(hunting): handle_auth_done global_id 74385 payload 293 2024-09-16T05:18:25.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 1 -- 172.21.15.114:0/3627973887 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f54d4005920 msgr2=0x7f54d4005da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:25.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 1 --2- 172.21.15.114:0/3627973887 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f54d4005920 0x7f54d4005da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:25.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 1 -- 172.21.15.114:0/3627973887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54d40064e0 msgr2=0x7f54d400cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:25.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 1 --2- 172.21.15.114:0/3627973887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54d40064e0 0x7f54d400cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:25.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 10 monclient: _finish_hunting 0 2024-09-16T05:18:25.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 1 monclient: found mon.c 2024-09-16T05:18:25.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:25.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:18:25.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 1 -- 172.21.15.114:0/3627973887 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d4004640 con 0x7f54d4004ad0 2024-09-16T05:18:25.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 10 monclient: _finish_auth 0 2024-09-16T05:18:25.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 10 monclient: _check_auth_tickets 2024-09-16T05:18:25.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:25.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f3943640 5 monclient: authenticate success, global_id 74385 2024-09-16T05:18:25.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f3943640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:25.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f1ee0640 1 --2- 172.21.15.114:0/3627973887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54d40064e0 0x7f54d400cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:18:25.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f1ee0640 1 -- 172.21.15.114:0/3627973887 reap_dead start 2024-09-16T05:18:25.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f16df640 1 --2- 172.21.15.114:0/3627973887 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54d4004ad0 0x7f54d4004ef0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f54e4001780 tx=0x7f54e401c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9d6a6da0f7094d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:25.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 1 -- 172.21.15.114:0/3627973887 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e40013a0 con 0x7f54d4004ad0 2024-09-16T05:18:25.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:25.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:18:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 10 monclient: dump: 2024-09-16T05:18:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:25.318 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f3943640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 1 -- 172.21.15.114:0/3627973887 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54e40052f0 con 0x7f54d4004ad0 2024-09-16T05:18:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f3943640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 1 -- 172.21.15.114:0/3627973887 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e4003840 con 0x7f54d4004ad0 2024-09-16T05:18:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f3943640 4 set_mon_vals no callback set 2024-09-16T05:18:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:18:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54e3fff640 10 monclient: dump: 2024-09-16T05:18:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:25.320 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:25.320 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:25.320 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:25.320 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:25.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f3943640 10 monclient: shutdown 2024-09-16T05:18:25.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f3943640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:25.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f3943640 1 -- 172.21.15.114:0/3627973887 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54d4004ad0 msgr2=0x7f54d4004ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:25.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f3943640 1 --2- 172.21.15.114:0/3627973887 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54d4004ad0 0x7f54d4004ef0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f54e4001780 tx=0x7f54e401c470 comp rx=0 tx=0).stop 2024-09-16T05:18:25.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f3943640 1 -- 172.21.15.114:0/3627973887 shutdown_connections 2024-09-16T05:18:25.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f3943640 1 --2- 172.21.15.114:0/3627973887 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54d4004ad0 0x7f54d4004ef0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:25.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f3943640 1 -- 172.21.15.114:0/3627973887 >> 172.21.15.114:0/3627973887 conn(0x7f54ec0a1fb0 msgr2=0x7f54ec0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:25.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f3943640 1 -- 172.21.15.114:0/3627973887 shutdown_connections 2024-09-16T05:18:25.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.278+0000 7f54f3943640 1 -- 172.21.15.114:0/3627973887 wait complete. 2024-09-16T05:18:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient: build_initial_monmap 2024-09-16T05:18:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient: monmap: 2024-09-16T05:18:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:25.321 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 librados: starting msgr at 2024-09-16T05:18:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 librados: starting objecter 2024-09-16T05:18:25.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 Processor -- start 2024-09-16T05:18:25.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 -- start start 2024-09-16T05:18:25.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 librados: setting wanted keys 2024-09-16T05:18:25.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 librados: calling monclient init 2024-09-16T05:18:25.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient: init 2024-09-16T05:18:25.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:25.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:18:25.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f54d4004ad0 0x7f54ec12e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:25.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient(hunting): picked mon.noname-b con 0x7f54d4004ad0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:25.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54ec12eec0 0x7f54ec12f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:25.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient(hunting): picked mon.noname-c con 0x7f54ec12eec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:25.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54ec1ce590 0x7f54ec1ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:25.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient(hunting): picked mon.noname-a con 0x7f54ec1ce590 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:25.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:25.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f54ec099e00 con 0x7f54ec1ce590 2024-09-16T05:18:25.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:25.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f54ec09f550 con 0x7f54d4004ad0 2024-09-16T05:18:25.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:25.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f54ec099f80 con 0x7f54ec12eec0 2024-09-16T05:18:25.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:25.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient(hunting): authenticate will time out at 1589.913066s 2024-09-16T05:18:25.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f0ede640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54ec12eec0 0x7f54ec12f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:25.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54ec1ce590 0x7f54ec1ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:25.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54ec1ce590 0x7f54ec1ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36448/0 (socket says 172.21.15.114:36448) 2024-09-16T05:18:25.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f0ede640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54ec12eec0 0x7f54ec12f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:58862/0 (socket says 172.21.15.114:58862) 2024-09-16T05:18:25.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 1 -- 172.21.15.114:0/624040347 learned_addr learned my addr 172.21.15.114:0/624040347 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:25.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 10 monclient(hunting): get_auth_request con 0x7f54ec1ce590 auth_method 0 2024-09-16T05:18:25.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:25.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:25.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:25.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f0ede640 10 monclient(hunting): get_auth_request con 0x7f54ec12eec0 auth_method 0 2024-09-16T05:18:25.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f0ede640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:25.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f0ede640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:25.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f0ede640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f0ede640 1 -- 172.21.15.114:0/624040347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54ec12eec0 msgr2=0x7f54ec12f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:18:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f0ede640 1 -- 172.21.15.114:0/624040347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54ec12eec0 msgr2=0x7f54ec12f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:18:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f0ede640 1 --2- 172.21.15.114:0/624040347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54ec12eec0 0x7f54ec12f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f0ede640 1 --2- 172.21.15.114:0/624040347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54ec12eec0 0x7f54ec12f300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f16df640 1 --2- 172.21.15.114:0/624040347 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f54d4004ad0 0x7f54ec12e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f16df640 10 monclient(hunting): get_auth_request con 0x7f54d4004ad0 auth_method 0 2024-09-16T05:18:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f16df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f16df640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f16df640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 10 monclient(hunting): handle_auth_done global_id 74388 payload 824 2024-09-16T05:18:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 1 -- 172.21.15.114:0/624040347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54ec12eec0 msgr2=0x7f54ec12f300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:18:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 1 --2- 172.21.15.114:0/624040347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54ec12eec0 0x7f54ec12f300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 1 -- 172.21.15.114:0/624040347 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f54d4004ad0 msgr2=0x7f54ec12e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 1 --2- 172.21.15.114:0/624040347 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f54d4004ad0 0x7f54ec12e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 10 monclient: _finish_hunting 0 2024-09-16T05:18:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 1 monclient: found mon.noname-a 2024-09-16T05:18:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 1 -- 172.21.15.114:0/624040347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54ec131240 con 0x7f54ec1ce590 2024-09-16T05:18:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f1ee0640 1 --2- 172.21.15.114:0/624040347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54ec1ce590 0x7f54ec1ce9f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f54ec09dec0 tx=0x7f54dc003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 10 client.?.objecter ms_handle_connect 0x7f54ec1ce590 2024-09-16T05:18:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 1 -- 172.21.15.114:0/624040347 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54dc018070 con 0x7f54ec1ce590 2024-09-16T05:18:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 10 monclient: dump: 2024-09-16T05:18:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:25.327 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 10 monclient: _finish_auth 0 2024-09-16T05:18:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:18:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 5 monclient: authenticate success, global_id 74388 2024-09-16T05:18:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 1 -- 172.21.15.114:0/624040347 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54dc003e10 con 0x7f54ec1ce590 2024-09-16T05:18:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient: _renew_subs 2024-09-16T05:18:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 -- 172.21.15.114:0/624040347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54ec09c6a0 con 0x7f54ec1ce590 2024-09-16T05:18:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 1 -- 172.21.15.114:0/624040347 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54dc00b600 con 0x7f54ec1ce590 2024-09-16T05:18:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 10 monclient: dump: 2024-09-16T05:18:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:25.329 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:25.329 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:25.329 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:25.329 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:25.329 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:25.329 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:25.329 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:25.329 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:25.329 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:25.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 client.74388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:25.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient: _renew_subs 2024-09-16T05:18:25.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:25.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e37fe640 4 set_mon_vals no callback set 2024-09-16T05:18:25.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 -- 172.21.15.114:0/624040347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54ec134090 con 0x7f54ec1ce590 2024-09-16T05:18:25.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 librados: init done 2024-09-16T05:18:25.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:25.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:25.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:25.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f3943640 1 -- 172.21.15.114:0/624040347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54ec09ca40 con 0x7f54ec1ce590 2024-09-16T05:18:25.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 1 -- 172.21.15.114:0/624040347 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f54dc01b070 con 0x7f54ec1ce590 2024-09-16T05:18:25.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 1 --2- 172.21.15.114:0/624040347 >> v2:172.21.15.149:6812/1185531135 conn(0x7f54d8037810 0x7f54d8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:25.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 1 -- 172.21.15.114:0/624040347 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(428..428 src has 1..428) ==== 5406+0+0 (secure 0 0 0) 0x7f54dc048d50 con 0x7f54ec1ce590 2024-09-16T05:18:25.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 10 client.74388.objecter ms_dispatch 0x7f54ec000ba0 osd_map(428..428 src has 1..428) 2024-09-16T05:18:25.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 3 client.74388.objecter handle_osd_map got epochs [428,428] > 0 2024-09-16T05:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 3 client.74388.objecter handle_osd_map decoding full epoch 428 2024-09-16T05:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54e1ffb640 20 client.74388.objecter dump_active .. 0 homeless 2024-09-16T05:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f16df640 1 --2- 172.21.15.114:0/624040347 >> v2:172.21.15.149:6812/1185531135 conn(0x7f54d8037810 0x7f54d8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.282+0000 7f54f16df640 10 monclient: get_auth_request con 0x7f54d8037810 auth_method 0 2024-09-16T05:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.286+0000 7f54e1ffb640 10 client.74388.objecter ms_handle_connect 0x7f54d8037810 2024-09-16T05:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.286+0000 7f54f16df640 1 --2- 172.21.15.114:0/624040347 >> v2:172.21.15.149:6812/1185531135 conn(0x7f54d8037810 0x7f54d8039cd0 secure :-1 s=READY pgs=3536 cs=0 l=1 rev1=1 crypto rx=0x7f54e4001750 tx=0x7f54e40011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.290+0000 7f54e1ffb640 1 -- 172.21.15.114:0/624040347 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f54dc014430 con 0x7f54ec1ce590 2024-09-16T05:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.290+0000 7f54e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.290+0000 7f54e1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:25.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.510+0000 7f54f3943640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:18:25.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.510+0000 7f54f3943640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:25.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.510+0000 7f54f3943640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:25.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.510+0000 7f54f3943640 1 -- 172.21.15.114:0/624040347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f54ec0686f0 con 0x7f54ec1ce590 2024-09-16T05:18:25.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.510+0000 7f54e1ffb640 1 -- 172.21.15.114:0/624040347 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f54dc010b50 con 0x7f54ec1ce590 2024-09-16T05:18:25.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.510+0000 7f54e1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:25.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.510+0000 7f54e1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:25.511 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":44,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":99,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:18:25.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 10 librados: watch_flush enter 2024-09-16T05:18:25.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 10 librados: watch_flush exit 2024-09-16T05:18:25.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 20 client.74388.objecter shutdown clearing up homeless session... 2024-09-16T05:18:25.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 10 client.74388.objecter successfully canceled tick 2024-09-16T05:18:25.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 1 -- 172.21.15.114:0/624040347 >> v2:172.21.15.149:6812/1185531135 conn(0x7f54d8037810 msgr2=0x7f54d8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:25.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 1 --2- 172.21.15.114:0/624040347 >> v2:172.21.15.149:6812/1185531135 conn(0x7f54d8037810 0x7f54d8039cd0 secure :-1 s=READY pgs=3536 cs=0 l=1 rev1=1 crypto rx=0x7f54e4001750 tx=0x7f54e40011a0 comp rx=0 tx=0).stop 2024-09-16T05:18:25.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 10 monclient: shutdown 2024-09-16T05:18:25.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:25.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 1 -- 172.21.15.114:0/624040347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54ec1ce590 msgr2=0x7f54ec1ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:25.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 1 --2- 172.21.15.114:0/624040347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54ec1ce590 0x7f54ec1ce9f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f54ec09dec0 tx=0x7f54dc003ce0 comp rx=0 tx=0).stop 2024-09-16T05:18:25.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 1 -- 172.21.15.114:0/624040347 shutdown_connections 2024-09-16T05:18:25.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 1 --2- 172.21.15.114:0/624040347 >> v2:172.21.15.149:6812/1185531135 conn(0x7f54d8037810 0x7f54d8039cd0 unknown :-1 s=CLOSED pgs=3536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:25.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 1 --2- 172.21.15.114:0/624040347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54ec1ce590 0x7f54ec1ce9f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:25.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 1 --2- 172.21.15.114:0/624040347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54ec12eec0 0x7f54ec12f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:25.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 1 --2- 172.21.15.114:0/624040347 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f54d4004ad0 0x7f54ec12e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:25.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 1 -- 172.21.15.114:0/624040347 >> 172.21.15.114:0/624040347 conn(0x7f54ec0a1fb0 msgr2=0x7f54ec0a33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:25.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 1 -- 172.21.15.114:0/624040347 shutdown_connections 2024-09-16T05:18:25.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 1 -- 172.21.15.114:0/624040347 wait complete. 2024-09-16T05:18:25.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:25.514+0000 7f54f3943640 1 librados: shutdown 2024-09-16T05:18:28.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:18:28.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 10 monclient: get_monmap_and_config 2024-09-16T05:18:28.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 10 monclient: build_initial_monmap 2024-09-16T05:18:28.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 10 monclient: monmap: 2024-09-16T05:18:28.660 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:28.660 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:28.660 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:28.665072+0000 2024-09-16T05:18:28.660 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:28.665072+0000 2024-09-16T05:18:28.660 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:28.660 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:28.660 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:28.660 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:28.661 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:28.661 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:28.661 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:28.661 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:28.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 1 Processor -- start 2024-09-16T05:18:28.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 1 -- start start 2024-09-16T05:18:28.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 10 monclient: init 2024-09-16T05:18:28.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:28.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-16T05:18:28.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 0x7f81880988e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:28.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 10 monclient(hunting): picked mon.noname-c con 0x7f8188098500 addr v2:172.21.15.149:3300/0 2024-09-16T05:18:28.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f818809c9b0 addr v1:172.21.15.114:6789/0 2024-09-16T05:18:28.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 0x7f818813b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:28.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 10 monclient(hunting): picked mon.noname-a con 0x7f818809d4d0 addr v2:172.21.15.114:3300/0 2024-09-16T05:18:28.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f818809db80 con 0x7f818809c9b0 2024-09-16T05:18:28.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:28.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f818809e070 con 0x7f818809d4d0 2024-09-16T05:18:28.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:28.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f818809f550 con 0x7f8188098500 2024-09-16T05:18:28.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:28.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f818d45a640 10 monclient(hunting): authenticate will time out at 1593.294369s 2024-09-16T05:18:28.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f81877fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 0x7f818813b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f81877fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 0x7f818813b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57136/0 (socket says 172.21.15.114:57136) 2024-09-16T05:18:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f81867fc640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f818809c9b0 0x7f818809cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50170/0 (socket says 172.21.15.114:50170) 2024-09-16T05:18:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f81877fe640 1 -- 172.21.15.114:0/1817848989 learned_addr learned my addr 172.21.15.114:0/1817848989 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f81877fe640 10 monclient(hunting): get_auth_request con 0x7f818809d4d0 auth_method 0 2024-09-16T05:18:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f81877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f81877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f81877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f81877fe640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 msgr2=0x7f818813b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:18:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f81877fe640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 msgr2=0x7f818813b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f81877fe640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 0x7f818813b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:28.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f81877fe640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 0x7f818813b8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:28.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 0x7f81880988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:28.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f8186ffd640 10 monclient(hunting): get_auth_request con 0x7f8188098500 auth_method 0 2024-09-16T05:18:28.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f8186ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:28.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f8186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:28.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f8186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:28.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f8186ffd640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 msgr2=0x7f81880988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:18:28.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f8186ffd640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 msgr2=0x7f81880988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:28.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 0x7f81880988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:28.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.662+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 0x7f81880988e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f81877fe640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 0x7f818813b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f81877fe640 10 monclient(hunting): get_auth_request con 0x7f818809d4d0 auth_method 0 2024-09-16T05:18:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f81877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f81877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f81877fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f81877fe640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 msgr2=0x7f818813b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:18:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f81877fe640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 msgr2=0x7f818813b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f81877fe640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 0x7f818813b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f81877fe640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 0x7f818813b8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:18:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 0x7f81880988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f8186ffd640 10 monclient(hunting): get_auth_request con 0x7f8188098500 auth_method 0 2024-09-16T05:18:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f8186ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f8186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f8186ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f8186ffd640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 msgr2=0x7f81880988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:18:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f8186ffd640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 msgr2=0x7f81880988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 0x7f81880988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:28.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:28.866+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 0x7f81880988e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:18:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f81877fe640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 0x7f818813b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f81877fe640 10 monclient(hunting): get_auth_request con 0x7f818809d4d0 auth_method 0 2024-09-16T05:18:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f81877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f81877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f81877fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 0x7f81880988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f8186ffd640 10 monclient(hunting): get_auth_request con 0x7f8188098500 auth_method 0 2024-09-16T05:18:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f8186ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f8186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f8186ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f81877fe640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 msgr2=0x7f818813b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:18:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f81877fe640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 msgr2=0x7f818813b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f81877fe640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 0x7f818813b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f81877fe640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 0x7f818813b8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:18:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f8186ffd640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 msgr2=0x7f81880988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:18:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f8186ffd640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 msgr2=0x7f81880988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 0x7f81880988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:29.270+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 0x7f81880988e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:18:30.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f81877fe640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 0x7f818813b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:30.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f81877fe640 10 monclient(hunting): get_auth_request con 0x7f818809d4d0 auth_method 0 2024-09-16T05:18:30.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f81877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:30.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f81877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:30.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f81877fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:30.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f81877fe640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 msgr2=0x7f818813b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:18:30.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f81877fe640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 msgr2=0x7f818813b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:30.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f81877fe640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 0x7f818813b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:30.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 0x7f81880988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:30.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f8186ffd640 10 monclient(hunting): get_auth_request con 0x7f8188098500 auth_method 0 2024-09-16T05:18:30.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f8186ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:30.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f8186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:30.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f81877fe640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 0x7f818813b8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:18:30.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f8186ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:30.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f8186ffd640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 msgr2=0x7f81880988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:18:30.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f8186ffd640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 msgr2=0x7f81880988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:30.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 0x7f81880988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:30.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:30.074+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 0x7f81880988e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:18:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 10 monclient(hunting): tick 2024-09-16T05:18:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:18:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 1 monclient(hunting): continuing hunt 2024-09-16T05:18:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:18:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 msgr2=0x7f81880988e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:18:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.149:3300/0 conn(0x7f8188098500 0x7f81880988e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 1 -- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 msgr2=0x7f818813b8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:18:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 1 --2- 172.21.15.114:0/1817848989 >> v2:172.21.15.114:3300/0 conn(0x7f818809d4d0 0x7f818813b8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 1 -- 172.21.15.114:0/1817848989 >> v1:172.21.15.114:6789/0 conn(0x7f818809c9b0 legacy=0x7f818809cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 10 monclient: _add_conns ranks=[2,1] 2024-09-16T05:18:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680036a0 0x7f8168005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f81680036a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8168006200 addr v1:172.21.15.149:6789/0 2024-09-16T05:18:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 1 -- 172.21.15.114:0/1817848989 --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f818809e2d0 con 0x7f8168006200 2024-09-16T05:18:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 1 -- 172.21.15.114:0/1817848989 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f818809f550 con 0x7f81680036a0 2024-09-16T05:18:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8184ff9640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680036a0 0x7f8168005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8186ffd640 10 monclient(hunting): get_auth_request con 0x7f81680036a0 auth_method 0 2024-09-16T05:18:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8186ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8186ffd640 1 -- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680036a0 msgr2=0x7f8168005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:18:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8186ffd640 1 -- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680036a0 msgr2=0x7f8168005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680036a0 0x7f8168005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.662+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680036a0 0x7f8168005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:31.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.866+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680036a0 0x7f8168005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:31.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.866+0000 7f8186ffd640 10 monclient(hunting): get_auth_request con 0x7f81680036a0 auth_method 0 2024-09-16T05:18:31.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.866+0000 7f8186ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:31.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.866+0000 7f8186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:31.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.866+0000 7f8186ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:31.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.866+0000 7f8186ffd640 1 -- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680036a0 msgr2=0x7f8168005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:18:31.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.866+0000 7f8186ffd640 1 -- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680036a0 msgr2=0x7f8168005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:31.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.866+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680036a0 0x7f8168005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:31.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:31.866+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680036a0 0x7f8168005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:18:32.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 -- 172.21.15.114:0/1817848989 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3481043716 0 0) 0x7f818809e2d0 con 0x7f8168006200 2024-09-16T05:18:32.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:18:32.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:32.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:32.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient(hunting): my global_id is 64501 2024-09-16T05:18:32.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 -- 172.21.15.114:0/1817848989 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8174003530 con 0x7f8168006200 2024-09-16T05:18:32.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 -- 172.21.15.114:0/1817848989 <== mon.1 v1:172.21.15.149:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2926096659 0 0) 0x7f8174003530 con 0x7f8168006200 2024-09-16T05:18:32.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 -- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680036a0 msgr2=0x7f8168005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:18:32.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81680036a0 0x7f8168005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:18:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 monclient: found mon.noname-c-legacy 2024-09-16T05:18:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.149:6789/0 2024-09-16T05:18:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 -- 172.21.15.114:0/1817848989 --> v1:172.21.15.149:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f816800afe0 con 0x7f8168006200 2024-09-16T05:18:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient: _finish_auth 0 2024-09-16T05:18:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:18:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 -- 172.21.15.114:0/1817848989 <== mon.1 v1:172.21.15.149:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f81700017c0 con 0x7f8168006200 2024-09-16T05:18:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T05:18:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient: dump: 2024-09-16T05:18:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:32.161 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] but i'm connected to v1:172.21.15.149:6789/0, reconnecting 2024-09-16T05:18:32.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:32.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 -- 172.21.15.114:0/1817848989 >> v1:172.21.15.149:6789/0 conn(0x7f8168006200 legacy=0x7f81680086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:32.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:18:32.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f81877fe640 1 -- 172.21.15.114:0/1817848989 reap_dead start 2024-09-16T05:18:32.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 0x7f8174003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:32.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient(hunting): picked mon.c con 0x7f81740039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:32.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8174004810 0x7f8174004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:32.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient(hunting): picked mon.a con 0x7f8174004810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:32.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81740052f0 0x7f817400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:32.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient(hunting): picked mon.b con 0x7f81740052f0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 -- 172.21.15.114:0/1817848989 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f818809f550 con 0x7f8174004810 2024-09-16T05:18:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 -- 172.21.15.114:0/1817848989 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f817400c240 con 0x7f81740052f0 2024-09-16T05:18:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 1 -- 172.21.15.114:0/1817848989 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f817400c440 con 0x7f81740039c0 2024-09-16T05:18:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8185ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:18:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 0x7f8174003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8186ffd640 10 monclient(hunting): get_auth_request con 0x7f81740039c0 auth_method 0 2024-09-16T05:18:32.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8186ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:32.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:32.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f8186ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:32.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f81867fc640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8174004810 0x7f8174004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:32.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f81877fe640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81740052f0 0x7f817400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:32.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f81877fe640 10 monclient(hunting): get_auth_request con 0x7f81740052f0 auth_method 0 2024-09-16T05:18:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f81877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f81877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f81877fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f81867fc640 10 monclient(hunting): get_auth_request con 0x7f8174004810 auth_method 0 2024-09-16T05:18:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f81867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f81867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.158+0000 7f81867fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:32.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:18:32.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f81877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:32.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f81877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:32.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f81877fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:18:32.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f81867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:32.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f81867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:32.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f81867fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:18:32.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 10 monclient(hunting): handle_auth_done global_id 64501 payload 293 2024-09-16T05:18:32.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 1 -- 172.21.15.114:0/1817848989 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81740052f0 msgr2=0x7f817400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81740052f0 0x7f817400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 1 -- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8174004810 msgr2=0x7f8174004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8174004810 0x7f8174004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:18:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 1 monclient: found mon.c 2024-09-16T05:18:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:18:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 1 -- 172.21.15.114:0/1817848989 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8174003530 con 0x7f81740039c0 2024-09-16T05:18:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 10 monclient: _finish_auth 0 2024-09-16T05:18:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:18:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f8186ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:32.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f818d45a640 5 monclient: authenticate success, global_id 64501 2024-09-16T05:18:32.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.162+0000 7f818d45a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:32.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 0x7f8174003dc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8178000ee0 tx=0x7f817801d9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=764f011b2068e467 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:32.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f8185ffb640 1 -- 172.21.15.114:0/1817848989 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817800f040 con 0x7f81740039c0 2024-09-16T05:18:32.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f8185ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:32.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f8185ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:18:32.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f8185ffb640 10 monclient: dump: 2024-09-16T05:18:32.175 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:32.175 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:32.175 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:32.175 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:32.175 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:32.176 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:32.176 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:32.176 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:32.176 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:32.176 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:32.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f8185ffb640 1 -- 172.21.15.114:0/1817848989 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81780a6880 con 0x7f81740039c0 2024-09-16T05:18:32.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f8185ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:32.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f8185ffb640 1 -- 172.21.15.114:0/1817848989 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81780a6cb0 con 0x7f81740039c0 2024-09-16T05:18:32.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:32.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 4 set_mon_vals no callback set 2024-09-16T05:18:32.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f8185ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:32.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f8185ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:18:32.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f8185ffb640 10 monclient: dump: 2024-09-16T05:18:32.177 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:32.177 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:32.177 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:32.177 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:32.177 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:32.178 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:32.178 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:32.178 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:32.178 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:32.178 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:32.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 10 monclient: shutdown 2024-09-16T05:18:32.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:32.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 1 -- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 msgr2=0x7f8174003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:32.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 0x7f8174003dc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8178000ee0 tx=0x7f817801d9e0 comp rx=0 tx=0).stop 2024-09-16T05:18:32.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 1 -- 172.21.15.114:0/1817848989 shutdown_connections 2024-09-16T05:18:32.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8174004810 0x7f8174004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:32.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81740052f0 0x7f817400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:32.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 1 --2- 172.21.15.114:0/1817848989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 0x7f8174003dc0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:32.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 1 -- 172.21.15.114:0/1817848989 >> 172.21.15.114:0/1817848989 conn(0x7f81880a1fb0 msgr2=0x7f81880a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:32.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 1 -- 172.21.15.114:0/1817848989 shutdown_connections 2024-09-16T05:18:32.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 1 -- 172.21.15.114:0/1817848989 wait complete. 2024-09-16T05:18:32.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 10 monclient: build_initial_monmap 2024-09-16T05:18:32.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 10 monclient: monmap: 2024-09-16T05:18:32.179 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:32.179 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:32.179 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:32.179 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:32.179 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:32.179 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:32.179 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:32.179 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:32.180 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:32.180 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:32.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 1 librados: starting msgr at 2024-09-16T05:18:32.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 1 librados: starting objecter 2024-09-16T05:18:32.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.166+0000 7f818d45a640 1 Processor -- start 2024-09-16T05:18:32.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 1 -- start start 2024-09-16T05:18:32.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 1 librados: setting wanted keys 2024-09-16T05:18:32.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 1 librados: calling monclient init 2024-09-16T05:18:32.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 monclient: init 2024-09-16T05:18:32.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:32.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:18:32.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 0x7f818812c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:32.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 monclient(hunting): picked mon.noname-c con 0x7f81740039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:32.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8174004810 0x7f818812cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:32.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 monclient(hunting): picked mon.noname-b con 0x7f8174004810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:32.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81740052f0 0x7f818812d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:32.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 monclient(hunting): picked mon.noname-a con 0x7f81740052f0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:32.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:32.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8188073c00 con 0x7f81740052f0 2024-09-16T05:18:32.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:32.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f8188073a80 con 0x7f8174004810 2024-09-16T05:18:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8188099e00 con 0x7f81740039c0 2024-09-16T05:18:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 monclient(hunting): authenticate will time out at 1596.800791s 2024-09-16T05:18:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f8186ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 0x7f818812c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81740052f0 0x7f818812d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f8186ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 0x7f818812c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:50686/0 (socket says 172.21.15.114:50686) 2024-09-16T05:18:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81740052f0 0x7f818812d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57186/0 (socket says 172.21.15.114:57186) 2024-09-16T05:18:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f8186ffd640 1 -- 172.21.15.114:0/1233433913 learned_addr learned my addr 172.21.15.114:0/1233433913 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f8186ffd640 10 monclient(hunting): get_auth_request con 0x7f81740039c0 auth_method 0 2024-09-16T05:18:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f8186ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f8186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f8186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 10 monclient(hunting): get_auth_request con 0x7f81740052f0 auth_method 0 2024-09-16T05:18:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f8186ffd640 1 -- 172.21.15.114:0/1233433913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 msgr2=0x7f818812c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:18:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f8186ffd640 1 -- 172.21.15.114:0/1233433913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 msgr2=0x7f818812c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:18:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1233433913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 0x7f818812c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1233433913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 0x7f818812c830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81867fc640 1 --2- 172.21.15.114:0/1233433913 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8174004810 0x7f818812cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81867fc640 10 monclient(hunting): get_auth_request con 0x7f8174004810 auth_method 0 2024-09-16T05:18:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:32.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:32.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:32.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 10 monclient(hunting): handle_auth_done global_id 74391 payload 824 2024-09-16T05:18:32.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 1 -- 172.21.15.114:0/1233433913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 msgr2=0x7f818812c830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:18:32.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 1 --2- 172.21.15.114:0/1233433913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 0x7f818812c830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:32.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 1 -- 172.21.15.114:0/1233433913 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8174004810 msgr2=0x7f818812cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:32.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 1 --2- 172.21.15.114:0/1233433913 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8174004810 0x7f818812cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:32.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 10 monclient: _finish_hunting 0 2024-09-16T05:18:32.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 1 monclient: found mon.noname-a 2024-09-16T05:18:32.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:32.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:32.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 1 -- 172.21.15.114:0/1233433913 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8188130420 con 0x7f81740052f0 2024-09-16T05:18:32.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81867fc640 1 --2- 172.21.15.114:0/1233433913 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8174004810 0x7f818812cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:18:32.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81877fe640 1 --2- 172.21.15.114:0/1233433913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81740052f0 0x7f818812d2b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f818809d8b0 tx=0x7f8180009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:32.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f817f7fe640 10 client.?.objecter ms_handle_connect 0x7f81740052f0 2024-09-16T05:18:32.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f817f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:32.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f817f7fe640 1 -- 172.21.15.114:0/1233433913 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818001e070 con 0x7f81740052f0 2024-09-16T05:18:32.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f817f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:32.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f817f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:32.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f817f7fe640 10 monclient: dump: 2024-09-16T05:18:32.185 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:32.185 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:32.185 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:32.185 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:32.185 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:32.185 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:32.185 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:32.185 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:32.186 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:32.186 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:32.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f817f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:18:32.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f817f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:18:32.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f817f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:32.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f817f7fe640 1 -- 172.21.15.114:0/1233433913 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8180003bb0 con 0x7f81740052f0 2024-09-16T05:18:32.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 5 monclient: authenticate success, global_id 74391 2024-09-16T05:18:32.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f817f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:32.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f817f7fe640 1 -- 172.21.15.114:0/1233433913 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81800178c0 con 0x7f81740052f0 2024-09-16T05:18:32.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f817f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:32.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f817f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:32.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f817f7fe640 10 monclient: dump: 2024-09-16T05:18:32.186 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:32.186 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:32.187 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:32.187 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:32.187 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:32.187 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:32.187 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:32.187 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:32.187 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:32.187 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:32.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f81857fa640 4 set_mon_vals no callback set 2024-09-16T05:18:32.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 monclient: _renew_subs 2024-09-16T05:18:32.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:32.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 1 -- 172.21.15.114:0/1233433913 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8188130700 con 0x7f81740052f0 2024-09-16T05:18:32.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 client.74391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:32.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 monclient: _renew_subs 2024-09-16T05:18:32.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:32.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 1 -- 172.21.15.114:0/1233433913 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f818809c6a0 con 0x7f81740052f0 2024-09-16T05:18:32.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.170+0000 7f818d45a640 1 librados: init done 2024-09-16T05:18:32.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.174+0000 7f817f7fe640 1 -- 172.21.15.114:0/1233433913 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8180021070 con 0x7f81740052f0 2024-09-16T05:18:32.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.174+0000 7f817f7fe640 1 --2- 172.21.15.114:0/1233433913 >> v2:172.21.15.149:6812/1185531135 conn(0x7f81680380d0 0x7f816803a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:32.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.174+0000 7f818d45a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:32.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.174+0000 7f818d45a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:32.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.174+0000 7f818d45a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:32.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.174+0000 7f817f7fe640 1 -- 172.21.15.114:0/1233433913 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(430..430 src has 1..430) ==== 5406+0+0 (secure 0 0 0) 0x7f818004b8b0 con 0x7f81740052f0 2024-09-16T05:18:32.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.174+0000 7f817f7fe640 10 client.74391.objecter ms_dispatch 0x7f8188000ba0 osd_map(430..430 src has 1..430) 2024-09-16T05:18:32.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.174+0000 7f817f7fe640 3 client.74391.objecter handle_osd_map got epochs [430,430] > 0 2024-09-16T05:18:32.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.174+0000 7f817f7fe640 3 client.74391.objecter handle_osd_map decoding full epoch 430 2024-09-16T05:18:32.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.174+0000 7f818d45a640 1 -- 172.21.15.114:0/1233433913 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81880a36e0 con 0x7f81740052f0 2024-09-16T05:18:32.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.174+0000 7f817f7fe640 20 client.74391.objecter dump_active .. 0 homeless 2024-09-16T05:18:32.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.174+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1233433913 >> v2:172.21.15.149:6812/1185531135 conn(0x7f81680380d0 0x7f816803a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:32.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.174+0000 7f8186ffd640 10 monclient: get_auth_request con 0x7f81680380d0 auth_method 0 2024-09-16T05:18:32.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.174+0000 7f817f7fe640 10 client.74391.objecter ms_handle_connect 0x7f81680380d0 2024-09-16T05:18:32.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.174+0000 7f8186ffd640 1 --2- 172.21.15.114:0/1233433913 >> v2:172.21.15.149:6812/1185531135 conn(0x7f81680380d0 0x7f816803a590 secure :-1 s=READY pgs=3537 cs=0 l=1 rev1=1 crypto rx=0x7f817801da20 tx=0x7f81780b3040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:32.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.178+0000 7f817f7fe640 1 -- 172.21.15.114:0/1233433913 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f8180004120 con 0x7f81740052f0 2024-09-16T05:18:32.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.178+0000 7f817f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:32.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.178+0000 7f817f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.398+0000 7f818d45a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:18:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.398+0000 7f818d45a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.398+0000 7f818d45a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.398+0000 7f818d45a640 1 -- 172.21.15.114:0/1233433913 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f81880988e0 con 0x7f81740052f0 2024-09-16T05:18:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.398+0000 7f817f7fe640 1 -- 172.21.15.114:0/1233433913 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f8180021540 con 0x7f81740052f0 2024-09-16T05:18:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.398+0000 7f817f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:32.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.398+0000 7f817f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:32.398 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":48,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:18:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 10 librados: watch_flush enter 2024-09-16T05:18:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 10 librados: watch_flush exit 2024-09-16T05:18:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 20 client.74391.objecter shutdown clearing up homeless session... 2024-09-16T05:18:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 10 client.74391.objecter successfully canceled tick 2024-09-16T05:18:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 1 -- 172.21.15.114:0/1233433913 >> v2:172.21.15.149:6812/1185531135 conn(0x7f81680380d0 msgr2=0x7f816803a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 1 --2- 172.21.15.114:0/1233433913 >> v2:172.21.15.149:6812/1185531135 conn(0x7f81680380d0 0x7f816803a590 secure :-1 s=READY pgs=3537 cs=0 l=1 rev1=1 crypto rx=0x7f817801da20 tx=0x7f81780b3040 comp rx=0 tx=0).stop 2024-09-16T05:18:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 10 monclient: shutdown 2024-09-16T05:18:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 1 -- 172.21.15.114:0/1233433913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81740052f0 msgr2=0x7f818812d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:32.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 1 --2- 172.21.15.114:0/1233433913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81740052f0 0x7f818812d2b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f818809d8b0 tx=0x7f8180009e90 comp rx=0 tx=0).stop 2024-09-16T05:18:32.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 1 -- 172.21.15.114:0/1233433913 shutdown_connections 2024-09-16T05:18:32.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 1 --2- 172.21.15.114:0/1233433913 >> v2:172.21.15.149:6812/1185531135 conn(0x7f81680380d0 0x7f816803a590 unknown :-1 s=CLOSED pgs=3537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:32.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 1 --2- 172.21.15.114:0/1233433913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81740052f0 0x7f818812d2b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:32.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 1 --2- 172.21.15.114:0/1233433913 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8174004810 0x7f818812cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:32.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 1 --2- 172.21.15.114:0/1233433913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81740039c0 0x7f818812c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:32.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 1 -- 172.21.15.114:0/1233433913 >> 172.21.15.114:0/1233433913 conn(0x7f81880a1fb0 msgr2=0x7f818806f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:32.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 1 -- 172.21.15.114:0/1233433913 shutdown_connections 2024-09-16T05:18:32.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 1 -- 172.21.15.114:0/1233433913 wait complete. 2024-09-16T05:18:32.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:32.402+0000 7f818d45a640 1 librados: shutdown 2024-09-16T05:18:35.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:18:35.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.546+0000 7f67c1564640 10 monclient: get_monmap_and_config 2024-09-16T05:18:35.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.546+0000 7f67c1564640 10 monclient: build_initial_monmap 2024-09-16T05:18:35.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.546+0000 7f67c1564640 10 monclient: monmap: 2024-09-16T05:18:35.546 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:35.546 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:35.546 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:35.551073+0000 2024-09-16T05:18:35.546 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:35.551073+0000 2024-09-16T05:18:35.546 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:35.546 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:35.546 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:35.546 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:35.547 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:35.547 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:35.547 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:35.547 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:35.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 1 Processor -- start 2024-09-16T05:18:35.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 1 -- start start 2024-09-16T05:18:35.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 10 monclient: init 2024-09-16T05:18:35.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:35.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-16T05:18:35.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67bc137730 0x7f67bc137b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:35.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 10 monclient(hunting): picked mon.noname-b con 0x7f67bc137730 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:35.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f67bc0a5d00 0x7f67bc0a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:35.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 10 monclient(hunting): picked mon.noname-c con 0x7f67bc0a5d00 addr v2:172.21.15.149:3300/0 2024-09-16T05:18:35.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f67bc0a47b0 0x7f67bc0a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:35.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 10 monclient(hunting): picked mon.noname-a con 0x7f67bc0a47b0 addr v2:172.21.15.114:3300/0 2024-09-16T05:18:35.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:35.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f67bc09c450 con 0x7f67bc0a47b0 2024-09-16T05:18:35.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:35.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f67bc0a55e0 con 0x7f67bc0a5d00 2024-09-16T05:18:35.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:35.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f67bc0a3710 con 0x7f67bc137730 2024-09-16T05:18:35.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:35.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 10 monclient(hunting): authenticate will time out at 1600.180969s 2024-09-16T05:18:35.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f67bc0a47b0 0x7f67bc0a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:35.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67baffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67bc137730 0x7f67bc137b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:35.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67baffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67bc137730 0x7f67bc137b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:50702/0 (socket says 172.21.15.114:50702) 2024-09-16T05:18:35.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67baffd640 1 -- 172.21.15.114:0/3184375788 learned_addr learned my addr 172.21.15.114:0/3184375788 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:35.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67baffd640 10 monclient(hunting): get_auth_request con 0x7f67bc137730 auth_method 0 2024-09-16T05:18:35.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:35.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:35.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:35.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 10 monclient(hunting): get_auth_request con 0x7f67bc0a47b0 auth_method 0 2024-09-16T05:18:35.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:35.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:35.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:35.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:35.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:35.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:35.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:35.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:35.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:35.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67ba7fc640 1 --2- 172.21.15.114:0/3184375788 >> v2:172.21.15.149:3300/0 conn(0x7f67bc0a5d00 0x7f67bc0a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:35.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67ba7fc640 10 monclient(hunting): get_auth_request con 0x7f67bc0a5d00 auth_method 0 2024-09-16T05:18:35.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:35.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:35.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:35.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 10 monclient(hunting): handle_auth_done global_id 74394 payload 274 2024-09-16T05:18:35.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 1 -- 172.21.15.114:0/3184375788 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67bc137730 msgr2=0x7f67bc137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:35.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 1 --2- 172.21.15.114:0/3184375788 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67bc137730 0x7f67bc137b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 1 -- 172.21.15.114:0/3184375788 >> v2:172.21.15.149:3300/0 conn(0x7f67bc0a5d00 msgr2=0x7f67bc0a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:35.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 1 --2- 172.21.15.114:0/3184375788 >> v2:172.21.15.149:3300/0 conn(0x7f67bc0a5d00 0x7f67bc0a4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:18:35.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 1 monclient: found mon.noname-a 2024-09-16T05:18:35.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:35.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:35.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 1 -- 172.21.15.114:0/3184375788 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc09be00 con 0x7f67bc0a47b0 2024-09-16T05:18:35.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67baffd640 1 --2- 172.21.15.114:0/3184375788 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67bc137730 0x7f67bc137b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:18:35.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67bb7fe640 1 --2- 172.21.15.114:0/3184375788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67bc0a47b0 0x7f67bc0a4c20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f67bc072f50 tx=0x7f67b401c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb37c65688a15bea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:35.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67b9ffb640 1 -- 172.21.15.114:0/3184375788 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b40b2070 con 0x7f67bc0a47b0 2024-09-16T05:18:35.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:35.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:35.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67b9ffb640 10 monclient: dump: 2024-09-16T05:18:35.559 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:35.560 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:35.560 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.560 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.560 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:35.560 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:35.560 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:35.560 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:35.560 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:35.560 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:35.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67b9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:18:35.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67b9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:18:35.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:35.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67b9ffb640 1 -- 172.21.15.114:0/3184375788 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67b40a5810 con 0x7f67bc0a47b0 2024-09-16T05:18:35.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67b9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:35.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67b9ffb640 1 -- 172.21.15.114:0/3184375788 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b40a5be0 con 0x7f67bc0a47b0 2024-09-16T05:18:35.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 5 monclient: authenticate success, global_id 74394 2024-09-16T05:18:35.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:35.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67c1564640 4 set_mon_vals no callback set 2024-09-16T05:18:35.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:35.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:35.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.550+0000 7f67b9ffb640 10 monclient: dump: 2024-09-16T05:18:35.561 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:35.561 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:35.561 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.561 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.561 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:35.561 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:35.562 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:35.562 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:35.562 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:35.562 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:35.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient: shutdown 2024-09-16T05:18:35.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:35.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 -- 172.21.15.114:0/3184375788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67bc0a47b0 msgr2=0x7f67bc0a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:35.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 --2- 172.21.15.114:0/3184375788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67bc0a47b0 0x7f67bc0a4c20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f67bc072f50 tx=0x7f67b401c880 comp rx=0 tx=0).stop 2024-09-16T05:18:35.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 -- 172.21.15.114:0/3184375788 shutdown_connections 2024-09-16T05:18:35.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 --2- 172.21.15.114:0/3184375788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67bc0a47b0 0x7f67bc0a4c20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 --2- 172.21.15.114:0/3184375788 >> v2:172.21.15.149:3300/0 conn(0x7f67bc0a5d00 0x7f67bc0a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 --2- 172.21.15.114:0/3184375788 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67bc137730 0x7f67bc137b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 -- 172.21.15.114:0/3184375788 >> 172.21.15.114:0/3184375788 conn(0x7f67bc0a9c70 msgr2=0x7f67bc0aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:35.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 -- 172.21.15.114:0/3184375788 shutdown_connections 2024-09-16T05:18:35.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 -- 172.21.15.114:0/3184375788 wait complete. 2024-09-16T05:18:35.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient: build_initial_monmap 2024-09-16T05:18:35.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient: monmap: 2024-09-16T05:18:35.563 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:35.563 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:35.563 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:35.563 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:35.563 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:35.563 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:35.563 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:35.563 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:35.563 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:35.563 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:35.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 librados: starting msgr at 2024-09-16T05:18:35.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 librados: starting objecter 2024-09-16T05:18:35.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 Processor -- start 2024-09-16T05:18:35.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 -- start start 2024-09-16T05:18:35.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 librados: setting wanted keys 2024-09-16T05:18:35.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 librados: calling monclient init 2024-09-16T05:18:35.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient: init 2024-09-16T05:18:35.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:35.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:18:35.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f67bc0a47b0 0x7f67bc1cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:35.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient(hunting): picked mon.noname-b con 0x7f67bc0a47b0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:35.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67bc0a5d00 0x7f67bc1cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:35.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient(hunting): picked mon.noname-c con 0x7f67bc0a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:35.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67bc137730 0x7f67bc1cf900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:35.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient(hunting): picked mon.noname-a con 0x7f67bc137730 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:35.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:35.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc137320 con 0x7f67bc137730 2024-09-16T05:18:35.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:35.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc1371a0 con 0x7f67bc0a47b0 2024-09-16T05:18:35.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:35.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f67bc140cf0 con 0x7f67bc0a5d00 2024-09-16T05:18:35.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:35.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient(hunting): authenticate will time out at 1600.186266s 2024-09-16T05:18:35.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67bc137730 0x7f67bc1cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:35.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67ba7fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67bc0a5d00 0x7f67bc1cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:35.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67bc137730 0x7f67bc1cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57208/0 (socket says 172.21.15.114:57208) 2024-09-16T05:18:35.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 1 -- 172.21.15.114:0/2083368979 learned_addr learned my addr 172.21.15.114:0/2083368979 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:35.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 10 monclient(hunting): get_auth_request con 0x7f67bc137730 auth_method 0 2024-09-16T05:18:35.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:35.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:35.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:35.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67ba7fc640 10 monclient(hunting): get_auth_request con 0x7f67bc0a5d00 auth_method 0 2024-09-16T05:18:35.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:35.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:35.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:35.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:35.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:35.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:35.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67ba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:35.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67ba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:35.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67ba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:35.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67baffd640 1 --2- 172.21.15.114:0/2083368979 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f67bc0a47b0 0x7f67bc1cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:35.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67baffd640 10 monclient(hunting): get_auth_request con 0x7f67bc0a47b0 auth_method 0 2024-09-16T05:18:35.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:35.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:35.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:35.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 10 monclient(hunting): handle_auth_done global_id 74397 payload 824 2024-09-16T05:18:35.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 1 -- 172.21.15.114:0/2083368979 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67bc0a5d00 msgr2=0x7f67bc1cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:35.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 1 --2- 172.21.15.114:0/2083368979 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67bc0a5d00 0x7f67bc1cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 1 -- 172.21.15.114:0/2083368979 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f67bc0a47b0 msgr2=0x7f67bc1cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:35.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 1 --2- 172.21.15.114:0/2083368979 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f67bc0a47b0 0x7f67bc1cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:18:35.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 1 monclient: found mon.noname-a 2024-09-16T05:18:35.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:35.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:35.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 1 -- 172.21.15.114:0/2083368979 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc13d1a0 con 0x7f67bc137730 2024-09-16T05:18:35.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67ba7fc640 1 --2- 172.21.15.114:0/2083368979 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67bc0a5d00 0x7f67bc1cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:18:35.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67bb7fe640 1 --2- 172.21.15.114:0/2083368979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67bc137730 0x7f67bc1cf900 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f67b401af30 tx=0x7f67b4019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:35.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b37fe640 10 client.?.objecter ms_handle_connect 0x7f67bc137730 2024-09-16T05:18:35.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:35.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b37fe640 1 -- 172.21.15.114:0/2083368979 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b40b2050 con 0x7f67bc137730 2024-09-16T05:18:35.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:35.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:35.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b37fe640 10 monclient: dump: 2024-09-16T05:18:35.568 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:35.568 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:35.568 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.568 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.569 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:35.569 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:35.569 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:35.569 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:35.569 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:35.569 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:35.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b37fe640 10 monclient: _finish_auth 0 2024-09-16T05:18:35.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:18:35.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:35.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 5 monclient: authenticate success, global_id 74397 2024-09-16T05:18:35.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b37fe640 1 -- 172.21.15.114:0/2083368979 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67b40be070 con 0x7f67bc137730 2024-09-16T05:18:35.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:35.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b37fe640 1 -- 172.21.15.114:0/2083368979 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b40ad870 con 0x7f67bc137730 2024-09-16T05:18:35.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:35.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:35.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b37fe640 10 monclient: dump: 2024-09-16T05:18:35.570 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:35.570 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:35.570 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.570 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.570 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:35.570 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:35.570 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:35.570 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:35.570 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:35.570 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:35.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67b97fa640 4 set_mon_vals no callback set 2024-09-16T05:18:35.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient: _renew_subs 2024-09-16T05:18:35.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:35.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 -- 172.21.15.114:0/2083368979 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67bc13d480 con 0x7f67bc137730 2024-09-16T05:18:35.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 client.74397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:35.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient: _renew_subs 2024-09-16T05:18:35.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:35.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 -- 172.21.15.114:0/2083368979 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67bc0aa130 con 0x7f67bc137730 2024-09-16T05:18:35.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.554+0000 7f67c1564640 1 librados: init done 2024-09-16T05:18:35.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.558+0000 7f67b37fe640 1 -- 172.21.15.114:0/2083368979 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f67b40ada10 con 0x7f67bc137730 2024-09-16T05:18:35.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.558+0000 7f67c1564640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:35.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.558+0000 7f67c1564640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:35.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.558+0000 7f67c1564640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:35.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.558+0000 7f67c1564640 1 -- 172.21.15.114:0/2083368979 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67bc0a4270 con 0x7f67bc137730 2024-09-16T05:18:35.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.558+0000 7f67b37fe640 1 --2- 172.21.15.114:0/2083368979 >> v2:172.21.15.149:6812/1185531135 conn(0x7f679c037810 0x7f679c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:35.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.558+0000 7f67b37fe640 1 -- 172.21.15.114:0/2083368979 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(430..430 src has 1..430) ==== 5406+0+0 (secure 0 0 0) 0x7f67b40e3db0 con 0x7f67bc137730 2024-09-16T05:18:35.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.558+0000 7f67b37fe640 10 client.74397.objecter ms_dispatch 0x7f67bc000ba0 osd_map(430..430 src has 1..430) 2024-09-16T05:18:35.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.558+0000 7f67b37fe640 3 client.74397.objecter handle_osd_map got epochs [430,430] > 0 2024-09-16T05:18:35.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.558+0000 7f67b37fe640 3 client.74397.objecter handle_osd_map decoding full epoch 430 2024-09-16T05:18:35.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.558+0000 7f67b37fe640 20 client.74397.objecter dump_active .. 0 homeless 2024-09-16T05:18:35.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.558+0000 7f67baffd640 1 --2- 172.21.15.114:0/2083368979 >> v2:172.21.15.149:6812/1185531135 conn(0x7f679c037810 0x7f679c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:35.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.558+0000 7f67baffd640 10 monclient: get_auth_request con 0x7f679c037810 auth_method 0 2024-09-16T05:18:35.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.558+0000 7f67baffd640 1 --2- 172.21.15.114:0/2083368979 >> v2:172.21.15.149:6812/1185531135 conn(0x7f679c037810 0x7f679c039cd0 secure :-1 s=READY pgs=3538 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009630 tx=0x7f67ac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:35.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.558+0000 7f67b37fe640 10 client.74397.objecter ms_handle_connect 0x7f679c037810 2024-09-16T05:18:35.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.562+0000 7f67b37fe640 1 -- 172.21.15.114:0/2083368979 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f67b4016500 con 0x7f67bc137730 2024-09-16T05:18:35.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.562+0000 7f67b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:35.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.562+0000 7f67b37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:35.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.782+0000 7f67c1564640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:18:35.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.782+0000 7f67c1564640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:35.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.782+0000 7f67c1564640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:35.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.782+0000 7f67c1564640 1 -- 172.21.15.114:0/2083368979 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f67bc13ba00 con 0x7f67bc137730 2024-09-16T05:18:35.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.782+0000 7f67b37fe640 1 -- 172.21.15.114:0/2083368979 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f67b40c3980 con 0x7f67bc137730 2024-09-16T05:18:35.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.782+0000 7f67b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:35.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.782+0000 7f67b37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:35.785 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:18:35.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 10 librados: watch_flush enter 2024-09-16T05:18:35.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 10 librados: watch_flush exit 2024-09-16T05:18:35.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 20 client.74397.objecter shutdown clearing up homeless session... 2024-09-16T05:18:35.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 10 client.74397.objecter successfully canceled tick 2024-09-16T05:18:35.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 1 -- 172.21.15.114:0/2083368979 >> v2:172.21.15.149:6812/1185531135 conn(0x7f679c037810 msgr2=0x7f679c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:35.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 1 --2- 172.21.15.114:0/2083368979 >> v2:172.21.15.149:6812/1185531135 conn(0x7f679c037810 0x7f679c039cd0 secure :-1 s=READY pgs=3538 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009630 tx=0x7f67ac008040 comp rx=0 tx=0).stop 2024-09-16T05:18:35.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 10 monclient: shutdown 2024-09-16T05:18:35.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:35.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 1 -- 172.21.15.114:0/2083368979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67bc137730 msgr2=0x7f67bc1cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:35.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 1 --2- 172.21.15.114:0/2083368979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67bc137730 0x7f67bc1cf900 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f67b401af30 tx=0x7f67b4019620 comp rx=0 tx=0).stop 2024-09-16T05:18:35.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 1 -- 172.21.15.114:0/2083368979 shutdown_connections 2024-09-16T05:18:35.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 1 --2- 172.21.15.114:0/2083368979 >> v2:172.21.15.149:6812/1185531135 conn(0x7f679c037810 0x7f679c039cd0 unknown :-1 s=CLOSED pgs=3538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 1 --2- 172.21.15.114:0/2083368979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67bc137730 0x7f67bc1cf900 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 1 --2- 172.21.15.114:0/2083368979 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67bc0a5d00 0x7f67bc1cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 1 --2- 172.21.15.114:0/2083368979 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f67bc0a47b0 0x7f67bc1cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 1 -- 172.21.15.114:0/2083368979 >> 172.21.15.114:0/2083368979 conn(0x7f67bc0a9c70 msgr2=0x7f67bc136a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:35.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 1 -- 172.21.15.114:0/2083368979 shutdown_connections 2024-09-16T05:18:35.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 1 -- 172.21.15.114:0/2083368979 wait complete. 2024-09-16T05:18:35.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.786+0000 7f67c1564640 1 librados: shutdown 2024-09-16T05:18:35.799 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:18:35.799 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:18:35.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:18:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.906+0000 7efd37e7e640 10 monclient: get_monmap_and_config 2024-09-16T05:18:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.906+0000 7efd37e7e640 10 monclient: build_initial_monmap 2024-09-16T05:18:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.906+0000 7efd37e7e640 10 monclient: monmap: 2024-09-16T05:18:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:35.912177+0000 2024-09-16T05:18:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:35.912177+0000 2024-09-16T05:18:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:35.908 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:35.908 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:35.908 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:35.908 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:35.908 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:35.908 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:35.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 1 Processor -- start 2024-09-16T05:18:35.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 1 -- start start 2024-09-16T05:18:35.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 10 monclient: init 2024-09-16T05:18:35.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-16T05:18:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efd30137730 0x7efd30137b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 10 monclient(hunting): picked mon.noname-a con 0x7efd30137730 addr v2:172.21.15.114:3300/0 2024-09-16T05:18:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd300a5d00 0x7efd300a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 10 monclient(hunting): picked mon.noname-b con 0x7efd300a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd300a47b0 addr v1:172.21.15.149:6789/0 2024-09-16T05:18:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd300a55e0 con 0x7efd300a47b0 2024-09-16T05:18:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7efd3009c450 con 0x7efd30137730 2024-09-16T05:18:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7efd300a3710 con 0x7efd300a5d00 2024-09-16T05:18:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd37e7e640 10 monclient(hunting): authenticate will time out at 1600.541345s 2024-09-16T05:18:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd300a5d00 0x7efd300a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35c1a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efd30137730 0x7efd30137b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd300a5d00 0x7efd300a4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:50716/0 (socket says 172.21.15.114:50716) 2024-09-16T05:18:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 1 -- 172.21.15.114:0/3784035884 learned_addr learned my addr 172.21.15.114:0/3784035884 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 10 monclient(hunting): get_auth_request con 0x7efd300a5d00 auth_method 0 2024-09-16T05:18:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35c1a640 10 monclient(hunting): get_auth_request con 0x7efd30137730 auth_method 0 2024-09-16T05:18:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35c1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35c1a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35c1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35c1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35c1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35c1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 10 monclient(hunting): handle_auth_done global_id 94106 payload 274 2024-09-16T05:18:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 1 -- 172.21.15.114:0/3784035884 >> v2:172.21.15.114:3300/0 conn(0x7efd30137730 msgr2=0x7efd30137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 1 --2- 172.21.15.114:0/3784035884 >> v2:172.21.15.114:3300/0 conn(0x7efd30137730 0x7efd30137b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 1 -- 172.21.15.114:0/3784035884 >> v1:172.21.15.149:6789/0 conn(0x7efd300a47b0 legacy=0x7efd3013e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 10 monclient: _finish_hunting 0 2024-09-16T05:18:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 1 monclient: found mon.noname-b 2024-09-16T05:18:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:18:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 1 -- 172.21.15.114:0/3784035884 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3009be00 con 0x7efd300a5d00 2024-09-16T05:18:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.910+0000 7efd35419640 1 --2- 172.21.15.114:0/3784035884 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd300a5d00 0x7efd300a4270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efd20009870 tx=0x7efd2001c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a547a493eff00582 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd34c18640 1 -- 172.21.15.114:0/3784035884 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd200b0070 con 0x7efd300a5d00 2024-09-16T05:18:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd34c18640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd34c18640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:18:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd34c18640 10 monclient: dump: 2024-09-16T05:18:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:35.923 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd34c18640 10 monclient: _finish_auth 0 2024-09-16T05:18:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd34c18640 10 monclient: _check_auth_tickets 2024-09-16T05:18:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd34c18640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 5 monclient: authenticate success, global_id 94106 2024-09-16T05:18:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd34c18640 1 -- 172.21.15.114:0/3784035884 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd200ab600 con 0x7efd300a5d00 2024-09-16T05:18:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd34c18640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:35.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 4 set_mon_vals no callback set 2024-09-16T05:18:35.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd34c18640 1 -- 172.21.15.114:0/3784035884 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd200ab9d0 con 0x7efd300a5d00 2024-09-16T05:18:35.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd34c18640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:35.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd34c18640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:18:35.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd34c18640 10 monclient: dump: 2024-09-16T05:18:35.924 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:35.924 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:35.924 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.924 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.924 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:35.924 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:35.924 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:35.924 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:35.924 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:35.925 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient: shutdown 2024-09-16T05:18:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 -- 172.21.15.114:0/3784035884 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd300a5d00 msgr2=0x7efd300a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 --2- 172.21.15.114:0/3784035884 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd300a5d00 0x7efd300a4270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efd20009870 tx=0x7efd2001c440 comp rx=0 tx=0).stop 2024-09-16T05:18:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 -- 172.21.15.114:0/3784035884 shutdown_connections 2024-09-16T05:18:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 --2- 172.21.15.114:0/3784035884 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd300a5d00 0x7efd300a4270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 --2- 172.21.15.114:0/3784035884 >> v2:172.21.15.114:3300/0 conn(0x7efd30137730 0x7efd30137b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 -- 172.21.15.114:0/3784035884 >> 172.21.15.114:0/3784035884 conn(0x7efd300a9c70 msgr2=0x7efd300aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 -- 172.21.15.114:0/3784035884 shutdown_connections 2024-09-16T05:18:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 -- 172.21.15.114:0/3784035884 wait complete. 2024-09-16T05:18:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient: build_initial_monmap 2024-09-16T05:18:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient: monmap: 2024-09-16T05:18:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:35.926 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 librados: starting msgr at 2024-09-16T05:18:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 librados: starting objecter 2024-09-16T05:18:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 Processor -- start 2024-09-16T05:18:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 -- start start 2024-09-16T05:18:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 librados: setting wanted keys 2024-09-16T05:18:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 librados: calling monclient init 2024-09-16T05:18:35.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient: init 2024-09-16T05:18:35.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:35.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:18:35.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd300a47b0 0x7efd3013e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:35.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient(hunting): picked mon.noname-a con 0x7efd300a47b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:35.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd300a5d00 0x7efd3013eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:35.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient(hunting): picked mon.noname-c con 0x7efd300a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:35.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd30137730 0x7efd30139190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:35.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient(hunting): picked mon.noname-b con 0x7efd30137730 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:35.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:35.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd30072a20 con 0x7efd300a47b0 2024-09-16T05:18:35.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:35.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7efd300728a0 con 0x7efd30137730 2024-09-16T05:18:35.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:35.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7efd301371a0 con 0x7efd300a5d00 2024-09-16T05:18:35.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:35.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient(hunting): authenticate will time out at 1600.546007s 2024-09-16T05:18:35.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd300a47b0 0x7efd3013e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:35.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd300a47b0 0x7efd3013e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57228/0 (socket says 172.21.15.114:57228) 2024-09-16T05:18:35.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd300a5d00 0x7efd3013eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:35.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 1 -- 172.21.15.114:0/654909696 learned_addr learned my addr 172.21.15.114:0/654909696 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:35.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 10 monclient(hunting): get_auth_request con 0x7efd300a47b0 auth_method 0 2024-09-16T05:18:35.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:35.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:35.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:35.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 10 monclient(hunting): get_auth_request con 0x7efd300a5d00 auth_method 0 2024-09-16T05:18:35.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:35.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:35.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:35.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:35.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:35.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:35.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:35.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:35.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:35.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 10 monclient(hunting): handle_auth_done global_id 74403 payload 824 2024-09-16T05:18:35.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 1 -- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd300a5d00 msgr2=0x7efd3013eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:35.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 1 --2- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd300a5d00 0x7efd3013eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 1 -- 172.21.15.114:0/654909696 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd30137730 msgr2=0x7efd30139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:35.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 1 --2- 172.21.15.114:0/654909696 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd30137730 0x7efd30139190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 10 monclient: _finish_hunting 0 2024-09-16T05:18:35.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 1 monclient: found mon.noname-a 2024-09-16T05:18:35.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:35.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:35.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 1 -- 172.21.15.114:0/654909696 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3013ce40 con 0x7efd300a47b0 2024-09-16T05:18:35.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35c1a640 1 --2- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd300a47b0 0x7efd3013e630 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00e960 tx=0x7efd2c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:35.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 10 client.?.objecter ms_handle_connect 0x7efd300a47b0 2024-09-16T05:18:35.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:35.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 1 -- 172.21.15.114:0/654909696 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2c017070 con 0x7efd300a47b0 2024-09-16T05:18:35.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:35.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:35.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 10 monclient: dump: 2024-09-16T05:18:35.931 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:35.931 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:35.931 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.931 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.931 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:35.931 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:35.931 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:35.931 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:35.932 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:35.932 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:35.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 10 monclient: _finish_auth 0 2024-09-16T05:18:35.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 10 monclient: _check_auth_tickets 2024-09-16T05:18:35.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:35.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 1 -- 172.21.15.114:0/654909696 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd2c00ca00 con 0x7efd300a47b0 2024-09-16T05:18:35.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 5 monclient: authenticate success, global_id 74403 2024-09-16T05:18:35.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient: _renew_subs 2024-09-16T05:18:35.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:35.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 -- 172.21.15.114:0/654909696 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd3013d120 con 0x7efd300a47b0 2024-09-16T05:18:35.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:35.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 1 -- 172.21.15.114:0/654909696 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2c00cdb0 con 0x7efd300a47b0 2024-09-16T05:18:35.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:35.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:35.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 10 monclient: dump: 2024-09-16T05:18:35.933 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:35.933 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:35.933 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.933 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.933 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:35.933 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:35.933 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:35.933 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:35.933 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:35.933 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:35.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd27fff640 4 set_mon_vals no callback set 2024-09-16T05:18:35.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 client.74403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:35.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient: _renew_subs 2024-09-16T05:18:35.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:35.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 -- 172.21.15.114:0/654909696 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd3013f8b0 con 0x7efd300a47b0 2024-09-16T05:18:35.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 librados: init done 2024-09-16T05:18:35.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:35.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 --2- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd301cf7a0 0x7efd301396d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:35.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:35.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd37e7e640 1 -- 172.21.15.114:0/654909696 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd300738a0 con 0x7efd301cf7a0 2024-09-16T05:18:35.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 1 --2- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd301cf7a0 0x7efd301396d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:35.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 10 monclient: get_auth_request con 0x7efd301cf7a0 auth_method 0 2024-09-16T05:18:35.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:35.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:35.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:35.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 1 -- 172.21.15.114:0/654909696 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7efd2c01a070 con 0x7efd300a47b0 2024-09-16T05:18:35.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:35.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:35.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:35.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 10 monclient(hunting): handle_auth_done global_id 74406 payload 824 2024-09-16T05:18:35.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd35419640 1 -- 172.21.15.114:0/654909696 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd30068ef0 con 0x7efd301cf7a0 2024-09-16T05:18:35.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 1 --2- 172.21.15.114:0/654909696 >> v2:172.21.15.149:6812/1185531135 conn(0x7efd14037810 0x7efd14039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:35.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 1 -- 172.21.15.114:0/654909696 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(430..430 src has 1..430) ==== 5406+0+0 (secure 0 0 0) 0x7efd2c048e20 con 0x7efd300a47b0 2024-09-16T05:18:35.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 10 client.74403.objecter ms_dispatch 0x7efd30000ba0 osd_map(430..430 src has 1..430) 2024-09-16T05:18:35.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 3 client.74403.objecter handle_osd_map got epochs [430,430] > 0 2024-09-16T05:18:35.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.914+0000 7efd267fc640 3 client.74403.objecter handle_osd_map decoding full epoch 430 2024-09-16T05:18:35.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd35419640 1 --2- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd301cf7a0 0x7efd301396d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efd300a3e10 tx=0x7efd2001ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:35.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd267fc640 20 client.74403.objecter dump_active .. 0 homeless 2024-09-16T05:18:35.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd267fc640 10 client.74403.objecter ms_handle_connect 0x7efd301cf7a0 2024-09-16T05:18:35.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd267fc640 10 client.74403.objecter resend_mon_ops 2024-09-16T05:18:35.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd267fc640 1 -- 172.21.15.114:0/654909696 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd200bb070 con 0x7efd301cf7a0 2024-09-16T05:18:35.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:35.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:35.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd267fc640 10 monclient: dump: 2024-09-16T05:18:35.936 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:35.936 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:35.936 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.936 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.937 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:35.937 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:35.937 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:35.937 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:35.937 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:35.937 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:35.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd3641b640 1 --2- 172.21.15.114:0/654909696 >> v2:172.21.15.149:6812/1185531135 conn(0x7efd14037810 0x7efd14039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:35.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd3641b640 10 monclient: get_auth_request con 0x7efd14037810 auth_method 0 2024-09-16T05:18:35.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd267fc640 1 -- 172.21.15.114:0/654909696 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efd200b1040 con 0x7efd301cf7a0 2024-09-16T05:18:35.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd267fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:35.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd267fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:35.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd267fc640 1 -- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd301cf7a0 msgr2=0x7efd301396d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:35.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd267fc640 1 --2- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd301cf7a0 0x7efd301396d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efd300a3e10 tx=0x7efd2001ac80 comp rx=0 tx=0).stop 2024-09-16T05:18:35.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd267fc640 1 -- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd301cf7a0 msgr2=0x7efd301396d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:35.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd267fc640 1 --2- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd301cf7a0 0x7efd301396d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd3641b640 1 --2- 172.21.15.114:0/654909696 >> v2:172.21.15.149:6812/1185531135 conn(0x7efd14037810 0x7efd14039cd0 secure :-1 s=READY pgs=3539 cs=0 l=1 rev1=1 crypto rx=0x7efd28004370 tx=0x7efd28007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:35.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.918+0000 7efd267fc640 10 client.74403.objecter ms_handle_connect 0x7efd14037810 2024-09-16T05:18:35.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:18:35.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 1 --2- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd30139c10 0x7efd3009ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:35.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:35.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 1 -- 172.21.15.114:0/654909696 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd3013a020 con 0x7efd30139c10 2024-09-16T05:18:35.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd35419640 1 --2- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd30139c10 0x7efd3009ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:35.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd35419640 10 monclient: get_auth_request con 0x7efd30139c10 auth_method 0 2024-09-16T05:18:35.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd35419640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:35.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd35419640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:35.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd35419640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:35.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd35419640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:35.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd35419640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:35.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd35419640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:35.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd35419640 10 monclient(hunting): handle_auth_done global_id 74409 payload 824 2024-09-16T05:18:35.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd35419640 1 -- 172.21.15.114:0/654909696 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efd3013f390 con 0x7efd30139c10 2024-09-16T05:18:35.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd35419640 1 --2- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd30139c10 0x7efd3009ee80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efd2001aaf0 tx=0x7efd200191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:35.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd267fc640 10 client.74403.objecter ms_handle_connect 0x7efd30139c10 2024-09-16T05:18:35.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd267fc640 10 client.74403.objecter resend_mon_ops 2024-09-16T05:18:35.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd267fc640 1 -- 172.21.15.114:0/654909696 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd200bb030 con 0x7efd30139c10 2024-09-16T05:18:35.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:35.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:35.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd267fc640 10 monclient: dump: 2024-09-16T05:18:35.940 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:35.940 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:35.940 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.940 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:35.940 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:35.940 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:35.940 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:35.940 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:35.941 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:35.941 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:35.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd267fc640 1 -- 172.21.15.114:0/654909696 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7efd20003620 con 0x7efd30139c10 2024-09-16T05:18:35.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd267fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:18:35.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd267fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:35.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd267fc640 1 -- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd30139c10 msgr2=0x7efd3009ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:35.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd267fc640 1 --2- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd30139c10 0x7efd3009ee80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efd2001aaf0 tx=0x7efd200191e0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd267fc640 1 -- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd30139c10 msgr2=0x7efd3009ee80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:35.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd267fc640 1 --2- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd30139c10 0x7efd3009ee80 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efd2001aaf0 tx=0x7efd200191e0 comp rx=0 tx=0).stop 2024-09-16T05:18:35.942 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:18:35.942 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:18:35.942 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:18:35.942 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:18:35.942 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 50, 2024-09-16T05:18:35.942 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 274977, 2024-09-16T05:18:35.942 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:18:35.942 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:18:35.942 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:18:35.942 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:18:35.942 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:35.943 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:18:35.943 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:35.943 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:18:35.943 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:18:35.943 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:35.943 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:35.943 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:35.943 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:35.943 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:35.943 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:35.943 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:35.943 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:35.943 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:35.944 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:35.944 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:35.944 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:35.944 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:18:35.944 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:18:35.944 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:35.944 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:35.944 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:35.944 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:35.944 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:35.944 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:35.944 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:35.945 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:35.945 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:35.945 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:35.945 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:35.945 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:35.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:35.945 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:18:35.945 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:18:35.945 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:18:35.945 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:18:35.945 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:18:35.945 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:18:35.945 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:35.945 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:35.946 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:18:35.946 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:18:35.946 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:18:35.946 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:18:35.946 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:18:35.946 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:18:35.946 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:18:35.946 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:35.946 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:18:35.946 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:35.946 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:35.946 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:35.946 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:35.947 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:35.947 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:35.947 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:35.947 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:35.947 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:35.947 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:35.947 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:35.947 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:35.947 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:18:35.947 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:35.947 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:18:35.947 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:35.947 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:18:35.948 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:18:35.948 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:35.948 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:35.948 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:35.948 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:35.948 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:18:35.948 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:35.948 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:35.948 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:35.948 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:35.948 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:18:35.948 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:35.948 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:35.948 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:35.949 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:35.949 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:18:35.949 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:18:35.949 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:35.949 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:35.949 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:35.949 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:35.949 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:35.949 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:18:35.949 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:18:35.949 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:35.949 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:35.949 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:35.949 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:35.950 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:18:35.950 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:35.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:35.950 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:35.950 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:35.950 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:18:35.950 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:35.950 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:35.950 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:35.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:35.950 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:18:35.950 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:18:35.950 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:35.950 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:35.951 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:35.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:35.951 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:35.951 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:18:35.951 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:18:35.951 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:35.951 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:35.951 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:35.951 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:35.951 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:18:35.951 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:35.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:35.951 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:35.951 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:35.952 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:18:35.952 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:35.952 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:35.952 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:35.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:35.952 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:18:35.952 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:18:35.952 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:35.952 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:35.952 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:35.952 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:35.952 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:35.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:35.952 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:18:35.953 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:18:35.953 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:35.953 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:35.953 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:35.953 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:18:35.953 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:35.953 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:35.953 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:18:35.953 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:35.953 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:35.953 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:35.953 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:18:35.953 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:35.953 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:35.954 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:18:35.954 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:35.954 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:35.954 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:35.954 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:18:35.954 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:35.954 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:35.954 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:35.954 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:18:35.954 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:18:35.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 10 librados: watch_flush enter 2024-09-16T05:18:35.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 10 librados: watch_flush exit 2024-09-16T05:18:35.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 20 client.74403.objecter shutdown clearing up homeless session... 2024-09-16T05:18:35.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 10 client.74403.objecter successfully canceled tick 2024-09-16T05:18:35.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 1 -- 172.21.15.114:0/654909696 >> v2:172.21.15.149:6812/1185531135 conn(0x7efd14037810 msgr2=0x7efd14039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:35.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 1 --2- 172.21.15.114:0/654909696 >> v2:172.21.15.149:6812/1185531135 conn(0x7efd14037810 0x7efd14039cd0 secure :-1 s=READY pgs=3539 cs=0 l=1 rev1=1 crypto rx=0x7efd28004370 tx=0x7efd28007d40 comp rx=0 tx=0).stop 2024-09-16T05:18:35.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 10 monclient: shutdown 2024-09-16T05:18:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 1 -- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd300a47b0 msgr2=0x7efd3013e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 1 --2- 172.21.15.114:0/654909696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd300a47b0 0x7efd3013e630 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00e960 tx=0x7efd2c00ee20 comp rx=0 tx=0).stop 2024-09-16T05:18:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd3641b640 1 -- 172.21.15.114:0/654909696 reap_dead start 2024-09-16T05:18:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd3641b640 1 -- 172.21.15.114:0/654909696 reap_dead start 2024-09-16T05:18:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 1 -- 172.21.15.114:0/654909696 shutdown_connections 2024-09-16T05:18:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 1 -- 172.21.15.114:0/654909696 >> 172.21.15.114:0/654909696 conn(0x7efd300a9c70 msgr2=0x7efd30136220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 1 -- 172.21.15.114:0/654909696 shutdown_connections 2024-09-16T05:18:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 1 -- 172.21.15.114:0/654909696 wait complete. 2024-09-16T05:18:35.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:35.922+0000 7efd37e7e640 1 librados: shutdown 2024-09-16T05:18:35.957 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:18:35.958 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:18:35.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:18:36.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 10 monclient: get_monmap_and_config 2024-09-16T05:18:36.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 10 monclient: build_initial_monmap 2024-09-16T05:18:36.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 10 monclient: monmap: 2024-09-16T05:18:36.072 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:36.072 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.072 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:36.076492+0000 2024-09-16T05:18:36.072 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:36.076492+0000 2024-09-16T05:18:36.072 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:36.072 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.072 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:36.072 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:36.072 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:36.072 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:36.072 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:36.073 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 1 Processor -- start 2024-09-16T05:18:36.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 1 -- start start 2024-09-16T05:18:36.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 10 monclient: init 2024-09-16T05:18:36.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:36.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T05:18:36.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb8dc06bcc0 addr v1:172.21.15.114:6789/0 2024-09-16T05:18:36.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb8dc06c7b0 0x7fb8dc09f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 10 monclient(hunting): picked mon.noname-a con 0x7fb8dc06c7b0 addr v2:172.21.15.114:3300/0 2024-09-16T05:18:36.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb8dc13c7e0 addr v1:172.21.15.149:6789/0 2024-09-16T05:18:36.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8dc06d120 con 0x7fb8dc06bcc0 2024-09-16T05:18:36.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8dc06d740 con 0x7fb8dc13c7e0 2024-09-16T05:18:36.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb8dc09f5b0 con 0x7fb8dc06c7b0 2024-09-16T05:18:36.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:36.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e2aa5640 10 monclient(hunting): authenticate will time out at 1600.705607s 2024-09-16T05:18:36.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb8dc06c7b0 0x7fb8dc09f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e0841640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb8dc06bcc0 0x7fb8dc06c0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50176/0 (socket says 172.21.15.114:50176) 2024-09-16T05:18:36.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8e0841640 1 -- 172.21.15.114:0/965517347 learned_addr learned my addr 172.21.15.114:0/965517347 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:18:36.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 10 monclient(hunting): get_auth_request con 0x7fb8dc06c7b0 auth_method 0 2024-09-16T05:18:36.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:36.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:36.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:36.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 10 monclient(hunting): handle_auth_done global_id 74412 payload 274 2024-09-16T05:18:36.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 1 -- 172.21.15.114:0/965517347 >> v1:172.21.15.149:6789/0 conn(0x7fb8dc13c7e0 legacy=0x7fb8dc13ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:36.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 1 -- 172.21.15.114:0/965517347 >> v1:172.21.15.114:6789/0 conn(0x7fb8dc06bcc0 legacy=0x7fb8dc06c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 10 monclient: _finish_hunting 0 2024-09-16T05:18:36.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 1 monclient: found mon.noname-a 2024-09-16T05:18:36.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:36.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:36.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8dbfff640 1 -- 172.21.15.114:0/965517347 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8dc0a36a0 con 0x7fb8dc06c7b0 2024-09-16T05:18:36.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8db7fe640 1 -- 172.21.15.114:0/965517347 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2682206911 0 0) 0x7fb8dc06d120 con 0x7fb8dc06bcc0 2024-09-16T05:18:36.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.074+0000 7fb8db7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:18:36.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8dbfff640 1 --2- 172.21.15.114:0/965517347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc06c7b0 0x7fb8dc09f070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40097a0 tx=0x7fb8c401c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9988b11dd427f5b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:36.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8db7fe640 1 -- 172.21.15.114:0/965517347 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c40b0070 con 0x7fb8dc06c7b0 2024-09-16T05:18:36.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8db7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:36.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8db7fe640 10 monclient: dump: 2024-09-16T05:18:36.087 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.087 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.087 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.087 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.087 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.087 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.087 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.087 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.087 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.087 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8db7fe640 10 monclient: _finish_auth 0 2024-09-16T05:18:36.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8db7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:18:36.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:36.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 5 monclient: authenticate success, global_id 74412 2024-09-16T05:18:36.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:36.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8db7fe640 1 -- 172.21.15.114:0/965517347 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8c40ab600 con 0x7fb8dc06c7b0 2024-09-16T05:18:36.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8db7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:36.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:36.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 4 set_mon_vals no callback set 2024-09-16T05:18:36.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8db7fe640 1 -- 172.21.15.114:0/965517347 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c40ab9d0 con 0x7fb8dc06c7b0 2024-09-16T05:18:36.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:36.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8db7fe640 10 monclient: dump: 2024-09-16T05:18:36.088 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.089 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.089 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.089 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.089 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.089 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.089 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.089 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.089 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.089 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 10 monclient: shutdown 2024-09-16T05:18:36.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:36.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 -- 172.21.15.114:0/965517347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc06c7b0 msgr2=0x7fb8dc09f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 --2- 172.21.15.114:0/965517347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc06c7b0 0x7fb8dc09f070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40097a0 tx=0x7fb8c401c300 comp rx=0 tx=0).stop 2024-09-16T05:18:36.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 -- 172.21.15.114:0/965517347 shutdown_connections 2024-09-16T05:18:36.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 --2- 172.21.15.114:0/965517347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc06c7b0 0x7fb8dc09f070 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 -- 172.21.15.114:0/965517347 >> 172.21.15.114:0/965517347 conn(0x7fb8dc0a4140 msgr2=0x7fb8dc0a4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:36.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 -- 172.21.15.114:0/965517347 shutdown_connections 2024-09-16T05:18:36.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 -- 172.21.15.114:0/965517347 wait complete. 2024-09-16T05:18:36.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 10 monclient: build_initial_monmap 2024-09-16T05:18:36.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 10 monclient: monmap: 2024-09-16T05:18:36.090 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:36.090 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:36.090 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:36.090 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:36.090 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:36.091 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.091 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:36.091 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:36.091 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:36.091 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 librados: starting msgr at 2024-09-16T05:18:36.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 librados: starting objecter 2024-09-16T05:18:36.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 Processor -- start 2024-09-16T05:18:36.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 -- start start 2024-09-16T05:18:36.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 librados: setting wanted keys 2024-09-16T05:18:36.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 librados: calling monclient init 2024-09-16T05:18:36.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 10 monclient: init 2024-09-16T05:18:36.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:36.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:18:36.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc06bcc0 0x7fb8dc13ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 10 monclient(hunting): picked mon.noname-c con 0x7fb8dc06bcc0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:36.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb8dc06c7b0 0x7fb8dc099750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 10 monclient(hunting): picked mon.noname-b con 0x7fb8dc06c7b0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:36.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc13c7e0 0x7fb8dc0721d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 10 monclient(hunting): picked mon.noname-a con 0x7fb8dc13c7e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:36.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc142ea0 con 0x7fb8dc13c7e0 2024-09-16T05:18:36.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb8dc142d20 con 0x7fb8dc06c7b0 2024-09-16T05:18:36.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb8dc143020 con 0x7fb8dc06bcc0 2024-09-16T05:18:36.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:36.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e2aa5640 10 monclient(hunting): authenticate will time out at 1600.711334s 2024-09-16T05:18:36.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc06bcc0 0x7fb8dc13ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc06bcc0 0x7fb8dc13ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:50730/0 (socket says 172.21.15.114:50730) 2024-09-16T05:18:36.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 1 -- 172.21.15.114:0/615902805 learned_addr learned my addr 172.21.15.114:0/615902805 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:36.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 10 monclient(hunting): get_auth_request con 0x7fb8dc06bcc0 auth_method 0 2024-09-16T05:18:36.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e1042640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc13c7e0 0x7fb8dc0721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e1042640 10 monclient(hunting): get_auth_request con 0x7fb8dc13c7e0 auth_method 0 2024-09-16T05:18:36.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e1042640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e1042640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e1042640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e1042640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e1042640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e1042640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8dbfff640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb8dc06c7b0 0x7fb8dc099750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 10 monclient(hunting): handle_auth_done global_id 94112 payload 824 2024-09-16T05:18:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 1 -- 172.21.15.114:0/615902805 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb8dc06c7b0 msgr2=0x7fb8dc099750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb8dc06c7b0 0x7fb8dc099750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 1 -- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc13c7e0 msgr2=0x7fb8dc0721d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8dc13c7e0 0x7fb8dc0721d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 10 monclient: _finish_hunting 0 2024-09-16T05:18:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 1 monclient: found mon.noname-c 2024-09-16T05:18:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:18:36.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 1 -- 172.21.15.114:0/615902805 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8dc073e10 con 0x7fb8dc06bcc0 2024-09-16T05:18:36.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8dbfff640 10 monclient: get_auth_request con 0x7fb8dc06c7b0 auth_method 0 2024-09-16T05:18:36.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8dbfff640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb8dc06c7b0 0x7fb8dc099750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:18:36.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8e0841640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc06bcc0 0x7fb8dc13ba00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400c890 tx=0x7fb8d400cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:36.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8d97fa640 10 client.?.objecter ms_handle_connect 0x7fb8dc06bcc0 2024-09-16T05:18:36.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.078+0000 7fb8d97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:36.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 1 -- 172.21.15.114:0/615902805 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4018070 con 0x7fb8dc06bcc0 2024-09-16T05:18:36.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:18:36.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 monclient: dump: 2024-09-16T05:18:36.095 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.095 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.095 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.095 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.096 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.096 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.096 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.096 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.096 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.096 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 monclient: _finish_auth 0 2024-09-16T05:18:36.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 monclient: _check_auth_tickets 2024-09-16T05:18:36.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:36.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 1 -- 172.21.15.114:0/615902805 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d400a640 con 0x7fb8dc06bcc0 2024-09-16T05:18:36.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:36.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 1 -- 172.21.15.114:0/615902805 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d400aa30 con 0x7fb8dc06bcc0 2024-09-16T05:18:36.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:18:36.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 monclient: dump: 2024-09-16T05:18:36.097 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.097 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.097 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.097 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.097 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.097 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.097 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.097 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.097 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.097 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e2aa5640 5 monclient: authenticate success, global_id 94112 2024-09-16T05:18:36.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e2aa5640 10 monclient: _renew_subs 2024-09-16T05:18:36.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8daffd640 4 set_mon_vals no callback set 2024-09-16T05:18:36.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e2aa5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:18:36.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e2aa5640 1 -- 172.21.15.114:0/615902805 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8dc13ff10 con 0x7fb8dc06bcc0 2024-09-16T05:18:36.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e2aa5640 10 client.94112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:36.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e2aa5640 10 monclient: _renew_subs 2024-09-16T05:18:36.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e2aa5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:18:36.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e2aa5640 1 -- 172.21.15.114:0/615902805 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8dc092720 con 0x7fb8dc06bcc0 2024-09-16T05:18:36.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e2aa5640 1 librados: init done 2024-09-16T05:18:36.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e2aa5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:36.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e2aa5640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc099c90 0x7fb8dc072710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e2aa5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e2aa5640 1 -- 172.21.15.114:0/615902805 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb8dc09e830 con 0x7fb8dc099c90 2024-09-16T05:18:36.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8dbfff640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc099c90 0x7fb8dc072710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8dbfff640 10 monclient: get_auth_request con 0x7fb8dc099c90 auth_method 0 2024-09-16T05:18:36.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 1 -- 172.21.15.114:0/615902805 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb8d400abd0 con 0x7fb8dc06bcc0 2024-09-16T05:18:36.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:36.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:36.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:36.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8dbfff640 10 monclient(hunting): handle_auth_done global_id 94115 payload 824 2024-09-16T05:18:36.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8dbfff640 1 -- 172.21.15.114:0/615902805 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb8dc0930e0 con 0x7fb8dc099c90 2024-09-16T05:18:36.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 1 --2- 172.21.15.114:0/615902805 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb8bc037840 0x7fb8bc039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 1 -- 172.21.15.114:0/615902805 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(431..431 src has 1..431) ==== 5406+0+0 (secure 0 0 0) 0x7fb8d40483e0 con 0x7fb8dc06bcc0 2024-09-16T05:18:36.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 client.94112.objecter ms_dispatch 0x7fb8dc000ba0 osd_map(431..431 src has 1..431) 2024-09-16T05:18:36.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 3 client.94112.objecter handle_osd_map got epochs [431,431] > 0 2024-09-16T05:18:36.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 3 client.94112.objecter handle_osd_map decoding full epoch 431 2024-09-16T05:18:36.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 20 client.94112.objecter dump_active .. 0 homeless 2024-09-16T05:18:36.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 client.94112.objecter ms_handle_connect 0x7fb8dc099c90 2024-09-16T05:18:36.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 client.94112.objecter resend_mon_ops 2024-09-16T05:18:36.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8dbfff640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc099c90 0x7fb8dc072710 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc06f540 tx=0x7fb8c401ab40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:36.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 1 -- 172.21.15.114:0/615902805 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c40bb070 con 0x7fb8dc099c90 2024-09-16T05:18:36.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:18:36.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 monclient: dump: 2024-09-16T05:18:36.100 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.100 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.100 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.100 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.100 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.100 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.100 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.101 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.101 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.101 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e1042640 1 --2- 172.21.15.114:0/615902805 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb8bc037840 0x7fb8bc039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e1042640 10 monclient: get_auth_request con 0x7fb8bc037840 auth_method 0 2024-09-16T05:18:36.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 1 -- 172.21.15.114:0/615902805 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb8c40abd00 con 0x7fb8dc099c90 2024-09-16T05:18:36.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:36.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:36.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 1 -- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc099c90 msgr2=0x7fb8dc072710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc099c90 0x7fb8dc072710 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc06f540 tx=0x7fb8c401ab40 comp rx=0 tx=0).stop 2024-09-16T05:18:36.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 1 -- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc099c90 msgr2=0x7fb8dc072710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:36.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc099c90 0x7fb8dc072710 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc06f540 tx=0x7fb8c401ab40 comp rx=0 tx=0).stop 2024-09-16T05:18:36.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8d97fa640 10 client.94112.objecter ms_handle_connect 0x7fb8bc037840 2024-09-16T05:18:36.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.082+0000 7fb8e1042640 1 --2- 172.21.15.114:0/615902805 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb8bc037840 0x7fb8bc039d00 secure :-1 s=READY pgs=3540 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0a2e20 tx=0x7fb8cc007dc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:36.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.086+0000 7fb8e2aa5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:18:36.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.086+0000 7fb8e2aa5640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc072c50 0x7fb8dc09b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.086+0000 7fb8e2aa5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.086+0000 7fb8e2aa5640 1 -- 172.21.15.114:0/615902805 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb8dc073060 con 0x7fb8dc072c50 2024-09-16T05:18:36.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8dbfff640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc072c50 0x7fb8dc09b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8dbfff640 10 monclient: get_auth_request con 0x7fb8dc072c50 auth_method 0 2024-09-16T05:18:36.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:36.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:36.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:36.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8dbfff640 10 monclient(hunting): handle_auth_done global_id 94118 payload 824 2024-09-16T05:18:36.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8dbfff640 1 -- 172.21.15.114:0/615902805 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb8dc09f070 con 0x7fb8dc072c50 2024-09-16T05:18:36.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8dbfff640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc072c50 0x7fb8dc09b510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4019f30 tx=0x7fb8c40190a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:36.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8d97fa640 10 client.94112.objecter ms_handle_connect 0x7fb8dc072c50 2024-09-16T05:18:36.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8d97fa640 10 client.94112.objecter resend_mon_ops 2024-09-16T05:18:36.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8d97fa640 1 -- 172.21.15.114:0/615902805 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c40b0040 con 0x7fb8dc072c50 2024-09-16T05:18:36.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:18:36.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8d97fa640 10 monclient: dump: 2024-09-16T05:18:36.105 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.105 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.105 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.105 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.105 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.105 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.105 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.105 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.105 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.105 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8d97fa640 1 -- 172.21.15.114:0/615902805 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7fb8c40b6470 con 0x7fb8dc072c50 2024-09-16T05:18:36.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8d97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:18:36.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8d97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:36.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8d97fa640 1 -- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc072c50 msgr2=0x7fb8dc09b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8d97fa640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc072c50 0x7fb8dc09b510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4019f30 tx=0x7fb8c40190a0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8d97fa640 1 -- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc072c50 msgr2=0x7fb8dc09b510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:36.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8d97fa640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc072c50 0x7fb8dc09b510 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4019f30 tx=0x7fb8c40190a0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.106 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:18:36.106 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:18:36.106 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:18:36.106 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:18:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 50, 2024-09-16T05:18:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 17639, 2024-09-16T05:18:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:18:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:18:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:18:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:18:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:18:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:18:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:18:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:36.108 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:36.108 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:36.108 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:36.108 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:36.108 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:36.108 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:36.108 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:36.108 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:36.108 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:36.108 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:36.108 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:18:36.108 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:18:36.108 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:36.108 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:36.109 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:36.109 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:36.109 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:36.109 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:36.109 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:36.109 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:36.109 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:36.109 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:36.109 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:36.109 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:36.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.109 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:18:36.109 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:18:36.109 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:18:36.110 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:18:36.110 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:18:36.110 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:18:36.110 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:36.110 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:36.110 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:18:36.110 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:18:36.110 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:18:36.110 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:18:36.110 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:18:36.110 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:18:36.110 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:18:36.110 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:36.110 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:18:36.110 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:36.111 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:36.111 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:36.111 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:36.111 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:36.111 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:36.111 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:36.111 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:36.111 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:36.111 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:36.111 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:36.111 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:36.111 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:18:36.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.111 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:18:36.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.112 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:18:36.112 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:18:36.112 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:36.112 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:36.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.112 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:36.112 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:18:36.112 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:36.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.112 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:36.112 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:18:36.112 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:36.113 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:36.113 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:36.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.113 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:18:36.113 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:18:36.113 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:36.113 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:36.113 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:36.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.113 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:18:36.113 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:18:36.113 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:36.113 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:36.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.114 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:36.114 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:18:36.114 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:36.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.114 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:36.114 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:18:36.114 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:36.114 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:36.114 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:36.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.114 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:18:36.114 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:18:36.115 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:36.115 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:36.115 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:36.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.115 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:18:36.115 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:18:36.115 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:36.115 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:36.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.115 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:36.115 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:18:36.115 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:36.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.116 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:36.116 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:18:36.116 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:36.116 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:36.116 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:36.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.116 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:18:36.116 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:18:36.116 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:36.116 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:36.116 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:36.116 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:36.116 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:36.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.117 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:18:36.117 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:18:36.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.117 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:36.117 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:36.117 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:18:36.117 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:36.117 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:36.117 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:18:36.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.117 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:36.117 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:36.117 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:18:36.117 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:36.118 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:36.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.118 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:18:36.118 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:18:36.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e2aa5640 10 librados: watch_flush enter 2024-09-16T05:18:36.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e2aa5640 10 librados: watch_flush exit 2024-09-16T05:18:36.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e2aa5640 20 client.94112.objecter shutdown clearing up homeless session... 2024-09-16T05:18:36.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e2aa5640 10 client.94112.objecter successfully canceled tick 2024-09-16T05:18:36.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e2aa5640 1 -- 172.21.15.114:0/615902805 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb8bc037840 msgr2=0x7fb8bc039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e2aa5640 1 --2- 172.21.15.114:0/615902805 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb8bc037840 0x7fb8bc039d00 secure :-1 s=READY pgs=3540 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0a2e20 tx=0x7fb8cc007dc0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e2aa5640 10 monclient: shutdown 2024-09-16T05:18:36.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e2aa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:36.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e2aa5640 1 -- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc06bcc0 msgr2=0x7fb8dc13ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e2aa5640 1 --2- 172.21.15.114:0/615902805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8dc06bcc0 0x7fb8dc13ba00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400c890 tx=0x7fb8d400cd50 comp rx=0 tx=0).stop 2024-09-16T05:18:36.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e1042640 1 -- 172.21.15.114:0/615902805 reap_dead start 2024-09-16T05:18:36.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e1042640 1 -- 172.21.15.114:0/615902805 reap_dead start 2024-09-16T05:18:36.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e2aa5640 1 -- 172.21.15.114:0/615902805 shutdown_connections 2024-09-16T05:18:36.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e2aa5640 1 -- 172.21.15.114:0/615902805 >> 172.21.15.114:0/615902805 conn(0x7fb8dc0a4140 msgr2=0x7fb8dc13ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:36.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e2aa5640 1 -- 172.21.15.114:0/615902805 shutdown_connections 2024-09-16T05:18:36.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e2aa5640 1 -- 172.21.15.114:0/615902805 wait complete. 2024-09-16T05:18:36.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.090+0000 7fb8e2aa5640 1 librados: shutdown 2024-09-16T05:18:36.121 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:18:36.121 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:18:36.121 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:18:36.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.234+0000 7f45b2f17640 10 monclient: get_monmap_and_config 2024-09-16T05:18:36.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.234+0000 7f45b2f17640 10 monclient: build_initial_monmap 2024-09-16T05:18:36.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.234+0000 7f45b2f17640 10 monclient: monmap: 2024-09-16T05:18:36.235 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:36.235 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.235 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:36.239799+0000 2024-09-16T05:18:36.235 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:36.239799+0000 2024-09-16T05:18:36.235 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:36.235 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.235 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:36.235 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:36.235 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:36.235 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:36.235 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:36.235 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 1 Processor -- start 2024-09-16T05:18:36.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 1 -- start start 2024-09-16T05:18:36.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 10 monclient: init 2024-09-16T05:18:36.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:36.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T05:18:36.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f45ac09b6c0 0x7f45ac0a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 10 monclient(hunting): picked mon.noname-c con 0x7f45ac09b6c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:18:36.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f45ac0a1400 0x7f45ac0a1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 10 monclient(hunting): picked mon.noname-a con 0x7f45ac0a1400 addr v2:172.21.15.114:3300/0 2024-09-16T05:18:36.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45ac139640 0x7f45ac13ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 10 monclient(hunting): picked mon.noname-b con 0x7f45ac139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:36.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f45ac09e390 con 0x7f45ac0a1400 2024-09-16T05:18:36.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f45ac0a1d50 con 0x7f45ac09b6c0 2024-09-16T05:18:36.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f45ac09ece0 con 0x7f45ac139640 2024-09-16T05:18:36.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:36.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b2f17640 10 monclient(hunting): authenticate will time out at 1600.869212s 2024-09-16T05:18:36.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b14b4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45ac139640 0x7f45ac13ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f45ac0a1400 0x7f45ac0a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b14b4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45ac139640 0x7f45ac13ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:50770/0 (socket says 172.21.15.114:50770) 2024-09-16T05:18:36.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b14b4640 1 -- 172.21.15.114:0/3098029879 learned_addr learned my addr 172.21.15.114:0/3098029879 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:36.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b14b4640 10 monclient(hunting): get_auth_request con 0x7f45ac139640 auth_method 0 2024-09-16T05:18:36.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b14b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b14b4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b14b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 10 monclient(hunting): get_auth_request con 0x7f45ac0a1400 auth_method 0 2024-09-16T05:18:36.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b14b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:36.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b14b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:36.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b14b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:36.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:36.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:36.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:36.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b0cb3640 1 --2- 172.21.15.114:0/3098029879 >> v2:172.21.15.149:3300/0 conn(0x7f45ac09b6c0 0x7f45ac0a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b0cb3640 10 monclient(hunting): get_auth_request con 0x7f45ac09b6c0 auth_method 0 2024-09-16T05:18:36.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b0cb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b0cb3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b0cb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 10 monclient(hunting): handle_auth_done global_id 74421 payload 274 2024-09-16T05:18:36.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 1 -- 172.21.15.114:0/3098029879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45ac139640 msgr2=0x7f45ac13ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:36.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 1 --2- 172.21.15.114:0/3098029879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45ac139640 0x7f45ac13ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 1 -- 172.21.15.114:0/3098029879 >> v2:172.21.15.149:3300/0 conn(0x7f45ac09b6c0 msgr2=0x7f45ac0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:36.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 1 --2- 172.21.15.114:0/3098029879 >> v2:172.21.15.149:3300/0 conn(0x7f45ac09b6c0 0x7f45ac0a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 10 monclient: _finish_hunting 0 2024-09-16T05:18:36.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 1 monclient: found mon.noname-a 2024-09-16T05:18:36.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:36.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:36.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 1 -- 172.21.15.114:0/3098029879 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac09d4e0 con 0x7f45ac0a1400 2024-09-16T05:18:36.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45b14b4640 1 --2- 172.21.15.114:0/3098029879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45ac139640 0x7f45ac13ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:18:36.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.238+0000 7f45abfff640 1 --2- 172.21.15.114:0/3098029879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45ac0a1400 0x7f45ac0a1810 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f45ac0a2f70 tx=0x7f459401c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d29b0b8437bde66 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:36.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45ab7fe640 1 -- 172.21.15.114:0/3098029879 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45940ac070 con 0x7f45ac0a1400 2024-09-16T05:18:36.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:36.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45ab7fe640 10 monclient: dump: 2024-09-16T05:18:36.249 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.249 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.249 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.249 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.249 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.250 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.250 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.250 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.250 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.250 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45ab7fe640 10 monclient: _finish_auth 0 2024-09-16T05:18:36.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45ab7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:18:36.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:36.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45ab7fe640 1 -- 172.21.15.114:0/3098029879 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45940a6660 con 0x7f45ac0a1400 2024-09-16T05:18:36.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45ab7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:36.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45ab7fe640 1 -- 172.21.15.114:0/3098029879 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45940a6a30 con 0x7f45ac0a1400 2024-09-16T05:18:36.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 5 monclient: authenticate success, global_id 74421 2024-09-16T05:18:36.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:36.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 4 set_mon_vals no callback set 2024-09-16T05:18:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45ab7fe640 10 monclient: dump: 2024-09-16T05:18:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.251 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient: shutdown 2024-09-16T05:18:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 -- 172.21.15.114:0/3098029879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45ac0a1400 msgr2=0x7f45ac0a1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 --2- 172.21.15.114:0/3098029879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45ac0a1400 0x7f45ac0a1810 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f45ac0a2f70 tx=0x7f459401c7d0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 -- 172.21.15.114:0/3098029879 shutdown_connections 2024-09-16T05:18:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 --2- 172.21.15.114:0/3098029879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45ac139640 0x7f45ac13ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 --2- 172.21.15.114:0/3098029879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45ac0a1400 0x7f45ac0a1810 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 --2- 172.21.15.114:0/3098029879 >> v2:172.21.15.149:3300/0 conn(0x7f45ac09b6c0 0x7f45ac0a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 -- 172.21.15.114:0/3098029879 >> 172.21.15.114:0/3098029879 conn(0x7f45ac0a1fb0 msgr2=0x7f45ac0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 -- 172.21.15.114:0/3098029879 shutdown_connections 2024-09-16T05:18:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 -- 172.21.15.114:0/3098029879 wait complete. 2024-09-16T05:18:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient: build_initial_monmap 2024-09-16T05:18:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient: monmap: 2024-09-16T05:18:36.252 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:36.253 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 librados: starting msgr at 2024-09-16T05:18:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 librados: starting objecter 2024-09-16T05:18:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 Processor -- start 2024-09-16T05:18:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 -- start start 2024-09-16T05:18:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 librados: setting wanted keys 2024-09-16T05:18:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 librados: calling monclient init 2024-09-16T05:18:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient: init 2024-09-16T05:18:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:18:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45ac09b6c0 0x7f45ac0994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient(hunting): picked mon.noname-c con 0x7f45ac09b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45ac0a1400 0x7f45ac0999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient(hunting): picked mon.noname-a con 0x7f45ac0a1400 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac139640 0x7f45ac099f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient(hunting): picked mon.noname-b con 0x7f45ac139640 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac13e790 con 0x7f45ac0a1400 2024-09-16T05:18:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac09cad0 con 0x7f45ac139640 2024-09-16T05:18:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f45ac13e910 con 0x7f45ac09b6c0 2024-09-16T05:18:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient(hunting): authenticate will time out at 1600.874075s 2024-09-16T05:18:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b0cb3640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45ac09b6c0 0x7f45ac0994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45ac0a1400 0x7f45ac0999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b0cb3640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45ac09b6c0 0x7f45ac0994b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:50776/0 (socket says 172.21.15.114:50776) 2024-09-16T05:18:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b0cb3640 1 -- 172.21.15.114:0/699685015 learned_addr learned my addr 172.21.15.114:0/699685015 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b0cb3640 10 monclient(hunting): get_auth_request con 0x7f45ac09b6c0 auth_method 0 2024-09-16T05:18:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b0cb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b0cb3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b0cb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 10 monclient(hunting): get_auth_request con 0x7f45ac0a1400 auth_method 0 2024-09-16T05:18:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b0cb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b0cb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b0cb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 10 monclient(hunting): handle_auth_done global_id 74424 payload 824 2024-09-16T05:18:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 1 -- 172.21.15.114:0/699685015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45ac09b6c0 msgr2=0x7f45ac0994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 1 --2- 172.21.15.114:0/699685015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f45ac09b6c0 0x7f45ac0994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 1 -- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac139640 msgr2=0x7f45ac099f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 1 --2- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac139640 0x7f45ac099f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 10 monclient: _finish_hunting 0 2024-09-16T05:18:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 1 monclient: found mon.noname-a 2024-09-16T05:18:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 1 -- 172.21.15.114:0/699685015 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac094720 con 0x7f45ac0a1400 2024-09-16T05:18:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45abfff640 1 --2- 172.21.15.114:0/699685015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45ac0a1400 0x7f45ac0999f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f45940ab000 tx=0x7f4594019570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 10 client.?.objecter ms_handle_connect 0x7f45ac0a1400 2024-09-16T05:18:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 1 -- 172.21.15.114:0/699685015 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45940c0070 con 0x7f45ac0a1400 2024-09-16T05:18:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 10 monclient: dump: 2024-09-16T05:18:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.258 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 10 monclient: _finish_auth 0 2024-09-16T05:18:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 10 monclient: _check_auth_tickets 2024-09-16T05:18:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 1 -- 172.21.15.114:0/699685015 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45940a6460 con 0x7f45ac0a1400 2024-09-16T05:18:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 5 monclient: authenticate success, global_id 74424 2024-09-16T05:18:36.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient: _renew_subs 2024-09-16T05:18:36.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:36.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 -- 172.21.15.114:0/699685015 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45ac139a20 con 0x7f45ac0a1400 2024-09-16T05:18:36.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:36.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 1 -- 172.21.15.114:0/699685015 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45940ac050 con 0x7f45ac0a1400 2024-09-16T05:18:36.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:36.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 10 monclient: dump: 2024-09-16T05:18:36.259 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.259 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.259 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.259 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.259 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.259 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.260 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45aa7fc640 4 set_mon_vals no callback set 2024-09-16T05:18:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 client.74424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient: _renew_subs 2024-09-16T05:18:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 -- 172.21.15.114:0/699685015 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45ac094440 con 0x7f45ac0a1400 2024-09-16T05:18:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 librados: init done 2024-09-16T05:18:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 1 -- 172.21.15.114:0/699685015 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f45940c3070 con 0x7f45ac0a1400 2024-09-16T05:18:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 --2- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac02d7d0 0x7f45ac09a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45b2f17640 1 -- 172.21.15.114:0/699685015 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac069390 con 0x7f45ac02d7d0 2024-09-16T05:18:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 1 --2- 172.21.15.114:0/699685015 >> v2:172.21.15.149:6812/1185531135 conn(0x7f458c037810 0x7f458c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 1 -- 172.21.15.114:0/699685015 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(431..431 src has 1..431) ==== 5406+0+0 (secure 0 0 0) 0x7f45940e3690 con 0x7f45ac0a1400 2024-09-16T05:18:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 10 client.74424.objecter ms_dispatch 0x7f45ac000ba0 osd_map(431..431 src has 1..431) 2024-09-16T05:18:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 3 client.74424.objecter handle_osd_map got epochs [431,431] > 0 2024-09-16T05:18:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.242+0000 7f45a97fa640 3 client.74424.objecter handle_osd_map decoding full epoch 431 2024-09-16T05:18:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45a97fa640 20 client.74424.objecter dump_active .. 0 homeless 2024-09-16T05:18:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45b0cb3640 1 --2- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac02d7d0 0x7f45ac09a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45b0cb3640 10 monclient: get_auth_request con 0x7f45ac02d7d0 auth_method 0 2024-09-16T05:18:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45b0cb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45b0cb3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45b0cb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45b14b4640 1 --2- 172.21.15.114:0/699685015 >> v2:172.21.15.149:6812/1185531135 conn(0x7f458c037810 0x7f458c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45b0cb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45b0cb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45b0cb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45b14b4640 10 monclient: get_auth_request con 0x7f458c037810 auth_method 0 2024-09-16T05:18:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45b0cb3640 10 monclient(hunting): handle_auth_done global_id 64516 payload 824 2024-09-16T05:18:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45b0cb3640 1 -- 172.21.15.114:0/699685015 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45ac02dbb0 con 0x7f45ac02d7d0 2024-09-16T05:18:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45a97fa640 10 client.74424.objecter ms_handle_connect 0x7f45ac02d7d0 2024-09-16T05:18:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45a97fa640 10 client.74424.objecter resend_mon_ops 2024-09-16T05:18:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45b0cb3640 1 --2- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac02d7d0 0x7f45ac09a470 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f459c010820 tx=0x7f459c010ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45a97fa640 10 client.74424.objecter ms_handle_connect 0x7f458c037810 2024-09-16T05:18:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45a97fa640 1 -- 172.21.15.114:0/699685015 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c018070 con 0x7f45ac02d7d0 2024-09-16T05:18:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45b14b4640 1 --2- 172.21.15.114:0/699685015 >> v2:172.21.15.149:6812/1185531135 conn(0x7f458c037810 0x7f458c039cd0 secure :-1 s=READY pgs=3541 cs=0 l=1 rev1=1 crypto rx=0x7f45ac13bb50 tx=0x7f45a4009450 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:18:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45a97fa640 10 monclient: dump: 2024-09-16T05:18:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.263 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45a97fa640 1 -- 172.21.15.114:0/699685015 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f459c00e9b0 con 0x7f45ac02d7d0 2024-09-16T05:18:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45a97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45a97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45a97fa640 1 -- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac02d7d0 msgr2=0x7f45ac09a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45a97fa640 1 --2- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac02d7d0 0x7f45ac09a470 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f459c010820 tx=0x7f459c010ce0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45a97fa640 1 -- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac02d7d0 msgr2=0x7f45ac09a470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:36.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.246+0000 7f45a97fa640 1 --2- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac02d7d0 0x7f45ac09a470 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f459c010820 tx=0x7f459c010ce0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:18:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 1 --2- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac062ea0 0x7f45ac063280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 1 -- 172.21.15.114:0/699685015 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac139e50 con 0x7f45ac062ea0 2024-09-16T05:18:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b0cb3640 1 --2- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac062ea0 0x7f45ac063280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b0cb3640 10 monclient: get_auth_request con 0x7f45ac062ea0 auth_method 0 2024-09-16T05:18:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b0cb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b0cb3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b0cb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b0cb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b0cb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b0cb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b0cb3640 10 monclient(hunting): handle_auth_done global_id 64519 payload 824 2024-09-16T05:18:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b0cb3640 1 -- 172.21.15.114:0/699685015 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f45ac093110 con 0x7f45ac062ea0 2024-09-16T05:18:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b0cb3640 1 --2- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac062ea0 0x7f45ac063280 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f459c010820 tx=0x7f459c0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45a97fa640 10 client.74424.objecter ms_handle_connect 0x7f45ac062ea0 2024-09-16T05:18:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45a97fa640 10 client.74424.objecter resend_mon_ops 2024-09-16T05:18:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45a97fa640 1 -- 172.21.15.114:0/699685015 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c004230 con 0x7f45ac062ea0 2024-09-16T05:18:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:18:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45a97fa640 10 monclient: dump: 2024-09-16T05:18:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.268 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.268 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.268 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45a97fa640 1 -- 172.21.15.114:0/699685015 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f459c004230 con 0x7f45ac062ea0 2024-09-16T05:18:36.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45a97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:18:36.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45a97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:36.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45a97fa640 1 -- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac062ea0 msgr2=0x7f45ac063280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45a97fa640 1 --2- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac062ea0 0x7f45ac063280 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f459c010820 tx=0x7f459c0046d0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45a97fa640 1 -- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac062ea0 msgr2=0x7f45ac063280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:36.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45a97fa640 1 --2- 172.21.15.114:0/699685015 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f45ac062ea0 0x7f45ac063280 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.269 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:18:36.269 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:18:36.269 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:18:36.269 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:18:36.269 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 50, 2024-09-16T05:18:36.269 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 400390, 2024-09-16T05:18:36.269 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:18:36.269 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:18:36.269 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:18:36.269 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:18:36.269 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:36.270 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 3, 2024-09-16T05:18:36.270 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:36.270 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:18:36.270 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:18:36.270 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:36.270 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:36.270 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:36.270 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:36.270 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:36.270 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:36.270 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:36.270 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:36.270 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:36.270 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:36.271 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:36.271 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:36.271 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:18:36.271 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:18:36.271 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:36.271 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:36.271 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:36.271 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:36.271 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:36.271 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:36.271 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:36.271 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:36.271 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:36.271 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:36.272 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:36.272 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:36.272 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.272 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:18:36.272 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:18:36.272 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:18:36.272 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:18:36.272 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:18:36.272 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:18:36.272 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:36.272 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:36.272 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:18:36.272 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:18:36.273 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:18:36.273 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:18:36.273 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:18:36.273 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:18:36.273 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:18:36.273 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:36.273 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:18:36.273 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:36.273 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:36.273 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:36.273 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:36.273 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:36.273 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:36.273 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:36.274 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:36.274 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:36.274 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:36.274 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:36.274 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:36.274 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:18:36.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.274 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:18:36.274 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.274 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:18:36.274 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:18:36.274 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:36.274 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:36.274 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.275 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:36.275 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:18:36.275 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:36.275 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.275 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.275 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:36.275 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:18:36.275 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:36.275 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:36.275 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:36.275 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.275 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:18:36.275 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:18:36.275 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:36.276 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:36.276 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:36.276 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.276 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.276 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:18:36.276 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:18:36.276 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:36.276 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:36.276 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.276 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:36.276 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:18:36.276 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:36.276 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.276 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.277 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:36.277 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:18:36.277 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:36.277 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:36.277 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:36.277 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.277 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:18:36.277 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:18:36.277 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:36.277 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:36.277 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:36.277 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.277 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.277 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:18:36.278 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:18:36.278 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:36.278 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:36.278 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.278 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:36.278 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:18:36.278 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:36.278 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.278 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.278 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:36.278 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:18:36.278 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:36.278 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:36.278 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:36.279 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.279 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:18:36.279 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:18:36.279 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:36.279 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:36.279 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:36.279 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:36.279 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:36.279 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.279 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:18:36.279 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:18:36.279 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.279 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:36.279 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:36.280 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:18:36.280 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:36.280 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:36.280 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:18:36.280 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.280 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:36.280 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:36.280 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:18:36.280 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:36.280 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:36.280 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:18:36.280 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.280 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:36.281 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:36.281 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:18:36.281 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:36.281 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:36.281 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:18:36.281 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:36.281 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:36.281 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:36.281 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:18:36.281 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:36.281 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:36.281 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:36.281 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:18:36.281 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:18:36.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 10 librados: watch_flush enter 2024-09-16T05:18:36.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 10 librados: watch_flush exit 2024-09-16T05:18:36.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 20 client.74424.objecter shutdown clearing up homeless session... 2024-09-16T05:18:36.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 10 client.74424.objecter successfully canceled tick 2024-09-16T05:18:36.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 1 -- 172.21.15.114:0/699685015 >> v2:172.21.15.149:6812/1185531135 conn(0x7f458c037810 msgr2=0x7f458c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 1 --2- 172.21.15.114:0/699685015 >> v2:172.21.15.149:6812/1185531135 conn(0x7f458c037810 0x7f458c039cd0 secure :-1 s=READY pgs=3541 cs=0 l=1 rev1=1 crypto rx=0x7f45ac13bb50 tx=0x7f45a4009450 comp rx=0 tx=0).stop 2024-09-16T05:18:36.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 10 monclient: shutdown 2024-09-16T05:18:36.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:36.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 1 -- 172.21.15.114:0/699685015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45ac0a1400 msgr2=0x7f45ac0999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 1 --2- 172.21.15.114:0/699685015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45ac0a1400 0x7f45ac0999f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f45940ab000 tx=0x7f4594019570 comp rx=0 tx=0).stop 2024-09-16T05:18:36.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b14b4640 1 -- 172.21.15.114:0/699685015 reap_dead start 2024-09-16T05:18:36.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b14b4640 1 -- 172.21.15.114:0/699685015 reap_dead start 2024-09-16T05:18:36.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 1 -- 172.21.15.114:0/699685015 shutdown_connections 2024-09-16T05:18:36.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 1 -- 172.21.15.114:0/699685015 >> 172.21.15.114:0/699685015 conn(0x7f45ac0a1fb0 msgr2=0x7f45ac138b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:36.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 1 -- 172.21.15.114:0/699685015 shutdown_connections 2024-09-16T05:18:36.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 1 -- 172.21.15.114:0/699685015 wait complete. 2024-09-16T05:18:36.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.254+0000 7f45b2f17640 1 librados: shutdown 2024-09-16T05:18:36.285 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:18:36.285 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:18:36.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:18:36.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.390+0000 7facedc3e640 10 monclient: get_monmap_and_config 2024-09-16T05:18:36.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.390+0000 7facedc3e640 10 monclient: build_initial_monmap 2024-09-16T05:18:36.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.390+0000 7facedc3e640 10 monclient: monmap: 2024-09-16T05:18:36.390 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:36.390 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.390 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:36.395075+0000 2024-09-16T05:18:36.390 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:36.395075+0000 2024-09-16T05:18:36.390 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:36.390 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.390 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:36.390 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:36.391 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:36.391 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:36.391 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:36.391 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.390+0000 7facedc3e640 1 Processor -- start 2024-09-16T05:18:36.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.390+0000 7facedc3e640 1 -- start start 2024-09-16T05:18:36.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.390+0000 7facedc3e640 10 monclient: init 2024-09-16T05:18:36.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7facedc3e640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:36.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7facedc3e640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-16T05:18:36.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7facedc3e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7face8068250 addr v1:172.21.15.149:6789/0 2024-09-16T05:18:36.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7facedc3e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7face8131f30 0x7face813a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7facedc3e640 10 monclient(hunting): picked mon.noname-a con 0x7face8131f30 addr v2:172.21.15.114:3300/0 2024-09-16T05:18:36.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7facedc3e640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7face81323c0 0x7face813c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7facedc3e640 10 monclient(hunting): picked mon.noname-c con 0x7face81323c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:18:36.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7facedc3e640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7face80a2490 con 0x7face8068250 2024-09-16T05:18:36.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7facedc3e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7facedc3e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7face80a30a0 con 0x7face8131f30 2024-09-16T05:18:36.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7facedc3e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7facedc3e640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7face8068630 con 0x7face81323c0 2024-09-16T05:18:36.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7facedc3e640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:36.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7facedc3e640 10 monclient(hunting): authenticate will time out at 1601.024319s 2024-09-16T05:18:36.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7face8131f30 0x7face813a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7face8131f30 0x7face813a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33990/0 (socket says 172.21.15.114:33990) 2024-09-16T05:18:36.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 1 -- 172.21.15.114:0/386796975 learned_addr learned my addr 172.21.15.114:0/386796975 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:36.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 10 monclient(hunting): get_auth_request con 0x7face8131f30 auth_method 0 2024-09-16T05:18:36.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:36.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:36.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:36.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face7fff640 1 --2- 172.21.15.114:0/386796975 >> v2:172.21.15.149:3300/0 conn(0x7face81323c0 0x7face813c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face7fff640 10 monclient(hunting): get_auth_request con 0x7face81323c0 auth_method 0 2024-09-16T05:18:36.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 10 monclient(hunting): handle_auth_done global_id 74427 payload 274 2024-09-16T05:18:36.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 1 -- 172.21.15.114:0/386796975 >> v2:172.21.15.149:3300/0 conn(0x7face81323c0 msgr2=0x7face813c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:36.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 1 --2- 172.21.15.114:0/386796975 >> v2:172.21.15.149:3300/0 conn(0x7face81323c0 0x7face813c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 1 -- 172.21.15.114:0/386796975 >> v1:172.21.15.149:6789/0 conn(0x7face8068250 legacy=0x7face8131820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:18:36.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 1 monclient: found mon.noname-a 2024-09-16T05:18:36.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:36.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:36.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.394+0000 7face6ffd640 1 -- 172.21.15.114:0/386796975 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7face809fde0 con 0x7face8131f30 2024-09-16T05:18:36.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face6ffd640 1 --2- 172.21.15.114:0/386796975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face8131f30 0x7face813a0e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7facd8009e80 tx=0x7facd801c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cbf962b393bc558 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:36.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face67fc640 1 -- 172.21.15.114:0/386796975 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd80a5660 con 0x7face8131f30 2024-09-16T05:18:36.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:36.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face67fc640 10 monclient: dump: 2024-09-16T05:18:36.406 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.406 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.406 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.406 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.406 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.406 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.406 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.406 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.406 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.406 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face67fc640 10 monclient: _finish_auth 0 2024-09-16T05:18:36.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face67fc640 10 monclient: _check_auth_tickets 2024-09-16T05:18:36.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:36.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face67fc640 1 -- 172.21.15.114:0/386796975 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facd80a5800 con 0x7face8131f30 2024-09-16T05:18:36.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:36.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face67fc640 1 -- 172.21.15.114:0/386796975 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd80a5bd0 con 0x7face8131f30 2024-09-16T05:18:36.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:36.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face67fc640 10 monclient: dump: 2024-09-16T05:18:36.407 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.407 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.407 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.407 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.407 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.407 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.408 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.408 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.408 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.408 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 5 monclient: authenticate success, global_id 74427 2024-09-16T05:18:36.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:36.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 4 set_mon_vals no callback set 2024-09-16T05:18:36.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient: shutdown 2024-09-16T05:18:36.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:36.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 -- 172.21.15.114:0/386796975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face8131f30 msgr2=0x7face813a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 --2- 172.21.15.114:0/386796975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face8131f30 0x7face813a0e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7facd8009e80 tx=0x7facd801c890 comp rx=0 tx=0).stop 2024-09-16T05:18:36.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 -- 172.21.15.114:0/386796975 shutdown_connections 2024-09-16T05:18:36.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 --2- 172.21.15.114:0/386796975 >> v2:172.21.15.149:3300/0 conn(0x7face81323c0 0x7face813c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 --2- 172.21.15.114:0/386796975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face8131f30 0x7face813a0e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 -- 172.21.15.114:0/386796975 >> 172.21.15.114:0/386796975 conn(0x7face80a6310 msgr2=0x7face80a2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:36.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 -- 172.21.15.114:0/386796975 shutdown_connections 2024-09-16T05:18:36.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 -- 172.21.15.114:0/386796975 wait complete. 2024-09-16T05:18:36.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient: build_initial_monmap 2024-09-16T05:18:36.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient: monmap: 2024-09-16T05:18:36.409 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:36.409 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:36.409 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:36.409 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:36.409 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:36.410 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.410 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:36.410 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:36.410 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:36.410 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 librados: starting msgr at 2024-09-16T05:18:36.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 librados: starting objecter 2024-09-16T05:18:36.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 Processor -- start 2024-09-16T05:18:36.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 -- start start 2024-09-16T05:18:36.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 librados: setting wanted keys 2024-09-16T05:18:36.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 librados: calling monclient init 2024-09-16T05:18:36.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient: init 2024-09-16T05:18:36.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:36.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:18:36.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7face8068250 0x7face812e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient(hunting): picked mon.noname-b con 0x7face8068250 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:36.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face8131f30 0x7face812ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient(hunting): picked mon.noname-a con 0x7face8131f30 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:36.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7face81323c0 0x7face81cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient(hunting): picked mon.noname-c con 0x7face81323c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:36.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7face813f720 con 0x7face8131f30 2024-09-16T05:18:36.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7face813f5a0 con 0x7face8068250 2024-09-16T05:18:36.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:36.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7face813f8a0 con 0x7face81323c0 2024-09-16T05:18:36.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:36.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient(hunting): authenticate will time out at 1601.030838s 2024-09-16T05:18:36.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7face81323c0 0x7face81cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7face81323c0 0x7face81cb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:46544/0 (socket says 172.21.15.114:46544) 2024-09-16T05:18:36.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 1 -- 172.21.15.114:0/302607520 learned_addr learned my addr 172.21.15.114:0/302607520 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:36.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 10 monclient(hunting): get_auth_request con 0x7face81323c0 auth_method 0 2024-09-16T05:18:36.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face6ffd640 1 --2- 172.21.15.114:0/302607520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face8131f30 0x7face812ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face6ffd640 10 monclient(hunting): get_auth_request con 0x7face8131f30 auth_method 0 2024-09-16T05:18:36.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:36.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:36.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:36.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:36.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:36.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:36.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face77fe640 1 --2- 172.21.15.114:0/302607520 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7face8068250 0x7face812e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face77fe640 10 monclient(hunting): get_auth_request con 0x7face8068250 auth_method 0 2024-09-16T05:18:36.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:36.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:36.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:36.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 10 monclient(hunting): handle_auth_done global_id 94127 payload 824 2024-09-16T05:18:36.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 1 -- 172.21.15.114:0/302607520 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7face8068250 msgr2=0x7face812e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 1 --2- 172.21.15.114:0/302607520 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7face8068250 0x7face812e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 1 -- 172.21.15.114:0/302607520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face8131f30 msgr2=0x7face812ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 1 --2- 172.21.15.114:0/302607520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face8131f30 0x7face812ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:18:36.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 1 monclient: found mon.noname-c 2024-09-16T05:18:36.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:36.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:18:36.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 1 -- 172.21.15.114:0/302607520 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7face813bc80 con 0x7face81323c0 2024-09-16T05:18:36.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face7fff640 1 --2- 172.21.15.114:0/302607520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7face81323c0 0x7face81cb930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7facd000d800 tx=0x7facd000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:36.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facdffff640 10 client.?.objecter ms_handle_connect 0x7face81323c0 2024-09-16T05:18:36.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facdffff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:36.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facdffff640 1 -- 172.21.15.114:0/302607520 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd000b6c0 con 0x7face81323c0 2024-09-16T05:18:36.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facdffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:18:36.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facdffff640 10 monclient: dump: 2024-09-16T05:18:36.415 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.415 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.415 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.415 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.415 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.415 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.415 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.415 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.415 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.415 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facdffff640 10 monclient: _finish_auth 0 2024-09-16T05:18:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facdffff640 10 monclient: _check_auth_tickets 2024-09-16T05:18:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facdffff640 1 -- 172.21.15.114:0/302607520 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facd0019070 con 0x7face81323c0 2024-09-16T05:18:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 5 monclient: authenticate success, global_id 94127 2024-09-16T05:18:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient: _renew_subs 2024-09-16T05:18:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:18:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 -- 172.21.15.114:0/302607520 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7face80a5760 con 0x7face81323c0 2024-09-16T05:18:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facdffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facdffff640 1 -- 172.21.15.114:0/302607520 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd000baa0 con 0x7face81323c0 2024-09-16T05:18:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facdffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:18:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facdffff640 10 monclient: dump: 2024-09-16T05:18:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:36.417 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 client.94127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient: _renew_subs 2024-09-16T05:18:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:18:36.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 -- 172.21.15.114:0/302607520 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7face813b6c0 con 0x7face81323c0 2024-09-16T05:18:36.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7face5ffb640 4 set_mon_vals no callback set 2024-09-16T05:18:36.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.398+0000 7facedc3e640 1 librados: init done 2024-09-16T05:18:36.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.402+0000 7facedc3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:36.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.402+0000 7facdffff640 1 -- 172.21.15.114:0/302607520 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7facd000bc40 con 0x7face81323c0 2024-09-16T05:18:36.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.402+0000 7facedc3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:36.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.402+0000 7facedc3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:18:36.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.402+0000 7facedc3e640 1 -- 172.21.15.114:0/302607520 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7face8131820 con 0x7face81323c0 2024-09-16T05:18:36.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.402+0000 7facdffff640 1 --2- 172.21.15.114:0/302607520 >> v2:172.21.15.149:6812/1185531135 conn(0x7facc8037810 0x7facc8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:36.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.402+0000 7facdffff640 1 -- 172.21.15.114:0/302607520 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(431..431 src has 1..431) ==== 5406+0+0 (secure 0 0 0) 0x7facd00491c0 con 0x7face81323c0 2024-09-16T05:18:36.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.402+0000 7facdffff640 10 client.94127.objecter ms_dispatch 0x7face8000ba0 osd_map(431..431 src has 1..431) 2024-09-16T05:18:36.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.402+0000 7facdffff640 3 client.94127.objecter handle_osd_map got epochs [431,431] > 0 2024-09-16T05:18:36.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.402+0000 7facdffff640 3 client.94127.objecter handle_osd_map decoding full epoch 431 2024-09-16T05:18:36.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.402+0000 7facdffff640 20 client.94127.objecter dump_active .. 0 homeless 2024-09-16T05:18:36.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.402+0000 7face77fe640 1 --2- 172.21.15.114:0/302607520 >> v2:172.21.15.149:6812/1185531135 conn(0x7facc8037810 0x7facc8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:36.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.402+0000 7face77fe640 10 monclient: get_auth_request con 0x7facc8037810 auth_method 0 2024-09-16T05:18:36.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.402+0000 7facdffff640 10 client.94127.objecter ms_handle_connect 0x7facc8037810 2024-09-16T05:18:36.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.402+0000 7face77fe640 1 --2- 172.21.15.114:0/302607520 >> v2:172.21.15.149:6812/1185531135 conn(0x7facc8037810 0x7facc8039cd0 secure :-1 s=READY pgs=3542 cs=0 l=1 rev1=1 crypto rx=0x7face0001650 tx=0x7face0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:36.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.406+0000 7facdffff640 1 -- 172.21.15.114:0/302607520 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7facd001edb0 con 0x7face81323c0 2024-09-16T05:18:36.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.406+0000 7facdffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:36.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.406+0000 7facdffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:36.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.614+0000 7facedc3e640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:18:36.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.614+0000 7facedc3e640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:18:36.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.614+0000 7facedc3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:18:36.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.614+0000 7facedc3e640 1 -- 172.21.15.114:0/302607520 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7face813a230 con 0x7face81323c0 2024-09-16T05:18:36.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facdffff640 1 -- 172.21.15.114:0/302607520 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7facd001e890 con 0x7face81323c0 2024-09-16T05:18:36.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facdffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:18:36.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facdffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:36.616 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.618+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.617 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.618+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=4195813046,config=2586702516,health=2941319762,kv=2757161437,logm=4192330601}) 2024-09-16T05:18:36.617 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.618+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=1,health=12,kv=10,logm=60} crc {auth=649658011,config=1228700967,health=2941319762,kv=2713019634,logm=1019060860}) 2024-09-16T05:18:36.617 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.618+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.617 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.618+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=4195813046,config=2586702516,health=2941319762,kv=2757161437,logm=4192330601}) 2024-09-16T05:18:36.617 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.618+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=649658011,config=2586702516,health=1075061118,kv=1083524254,logm=3341350556}) 2024-09-16T05:18:36.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 10 librados: watch_flush enter 2024-09-16T05:18:36.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 10 librados: watch_flush exit 2024-09-16T05:18:36.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 20 client.94127.objecter shutdown clearing up homeless session... 2024-09-16T05:18:36.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 10 client.94127.objecter successfully canceled tick 2024-09-16T05:18:36.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 1 -- 172.21.15.114:0/302607520 >> v2:172.21.15.149:6812/1185531135 conn(0x7facc8037810 msgr2=0x7facc8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 1 --2- 172.21.15.114:0/302607520 >> v2:172.21.15.149:6812/1185531135 conn(0x7facc8037810 0x7facc8039cd0 secure :-1 s=READY pgs=3542 cs=0 l=1 rev1=1 crypto rx=0x7face0001650 tx=0x7face0009040 comp rx=0 tx=0).stop 2024-09-16T05:18:36.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 10 monclient: shutdown 2024-09-16T05:18:36.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:36.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 1 -- 172.21.15.114:0/302607520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7face81323c0 msgr2=0x7face81cb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:36.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 1 --2- 172.21.15.114:0/302607520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7face81323c0 0x7face81cb930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7facd000d800 tx=0x7facd000dcc0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.620 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.618+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 1 -- 172.21.15.114:0/302607520 shutdown_connections 2024-09-16T05:18:36.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 1 --2- 172.21.15.114:0/302607520 >> v2:172.21.15.149:6812/1185531135 conn(0x7facc8037810 0x7facc8039cd0 unknown :-1 s=CLOSED pgs=3542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 1 --2- 172.21.15.114:0/302607520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face8131f30 0x7face812ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 1 --2- 172.21.15.114:0/302607520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7face81323c0 0x7face81cb930 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.621 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.618+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1847588517}) 2024-09-16T05:18:36.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 1 --2- 172.21.15.114:0/302607520 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7face8068250 0x7face812e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:36.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 1 -- 172.21.15.114:0/302607520 >> 172.21.15.114:0/302607520 conn(0x7face80a6310 msgr2=0x7face80a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:36.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 1 -- 172.21.15.114:0/302607520 shutdown_connections 2024-09-16T05:18:36.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 1 -- 172.21.15.114:0/302607520 wait complete. 2024-09-16T05:18:36.622 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.618+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1493816423}) 2024-09-16T05:18:36.622 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.618+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.622 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.618+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1847588517}) 2024-09-16T05:18:36.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:36.618+0000 7facedc3e640 1 librados: shutdown 2024-09-16T05:18:36.623 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.622+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4225435862}) 2024-09-16T05:18:36.623 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.622+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.623 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.622+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1539031143}) 2024-09-16T05:18:36.623 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.622+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1260017906}) 2024-09-16T05:18:36.623 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.622+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.623 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.622+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1539031143}) 2024-09-16T05:18:36.623 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.622+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2195571678}) 2024-09-16T05:18:36.624 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.626+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.624 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.626+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4193070282}) 2024-09-16T05:18:36.624 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.626+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094124477}) 2024-09-16T05:18:36.625 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.626+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.625 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.626+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4193070282}) 2024-09-16T05:18:36.625 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.626+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3932508638}) 2024-09-16T05:18:36.627 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.626+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.627 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.630+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720076984}) 2024-09-16T05:18:36.627 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.630+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=632319065}) 2024-09-16T05:18:36.627 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.630+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.627 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.630+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720076984}) 2024-09-16T05:18:36.627 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.630+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3047837954}) 2024-09-16T05:18:36.629 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.630+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.629 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.630+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1356166110}) 2024-09-16T05:18:36.629 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.630+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2890112538}) 2024-09-16T05:18:36.629 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.630+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.630 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.630+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1356166110}) 2024-09-16T05:18:36.630 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.630+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2927208646}) 2024-09-16T05:18:36.631 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.634+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.631 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.634+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627517082}) 2024-09-16T05:18:36.632 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.634+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1404017333}) 2024-09-16T05:18:36.632 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.634+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.632 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.634+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627517082}) 2024-09-16T05:18:36.632 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.634+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2640485802}) 2024-09-16T05:18:36.633 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.634+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.634 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.634+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2343203121}) 2024-09-16T05:18:36.634 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.634+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2168651694}) 2024-09-16T05:18:36.634 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.634+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.634 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.634+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2343203121}) 2024-09-16T05:18:36.634 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.634+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2705341236}) 2024-09-16T05:18:36.635 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.638+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.635 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.638+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763484017}) 2024-09-16T05:18:36.636 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.638+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=821540041}) 2024-09-16T05:18:36.636 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.638+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.636 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.638+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763484017}) 2024-09-16T05:18:36.636 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.638+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3373503022}) 2024-09-16T05:18:36.637 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:18:36.638 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.638+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.638 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.638+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1032575474}) 2024-09-16T05:18:36.638 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.638+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=546664070}) 2024-09-16T05:18:36.638 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.638+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.638 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.638+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1032575474}) 2024-09-16T05:18:36.638 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.638+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=344343519}) 2024-09-16T05:18:36.640 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.642+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.640 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.642+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456669778}) 2024-09-16T05:18:36.640 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.642+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=350293564}) 2024-09-16T05:18:36.640 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.642+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.640 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.642+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456669778}) 2024-09-16T05:18:36.640 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.642+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3092649870}) 2024-09-16T05:18:36.642 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.642+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.642 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.642+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1991313354}) 2024-09-16T05:18:36.642 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.642+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=445470272}) 2024-09-16T05:18:36.642 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.642+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.642 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.642+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1991313354}) 2024-09-16T05:18:36.642 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.642+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1520311868}) 2024-09-16T05:18:36.644 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.646+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.644 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.646+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943568635}) 2024-09-16T05:18:36.644 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.646+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=371104807}) 2024-09-16T05:18:36.645 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.646+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.645 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.646+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943568635}) 2024-09-16T05:18:36.645 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.646+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1750142634}) 2024-09-16T05:18:36.646 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.646+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.646 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.646+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139637105}) 2024-09-16T05:18:36.646 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.646+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1310168806}) 2024-09-16T05:18:36.647 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.646+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.647 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.646+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139637105}) 2024-09-16T05:18:36.647 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.646+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2415148768}) 2024-09-16T05:18:36.649 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.650+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.650 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.650+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053312944}) 2024-09-16T05:18:36.650 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.650+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2564769305}) 2024-09-16T05:18:36.650 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.650+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.650 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.650+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053312944}) 2024-09-16T05:18:36.650 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.650+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1184931058}) 2024-09-16T05:18:36.651 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.650+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.651 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.650+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218206017}) 2024-09-16T05:18:36.651 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.650+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2082365504}) 2024-09-16T05:18:36.651 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.650+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.651 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.650+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218206017}) 2024-09-16T05:18:36.652 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.650+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3724012533}) 2024-09-16T05:18:36.653 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.654+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.653 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.654+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740399840}) 2024-09-16T05:18:36.653 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.654+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2378325218}) 2024-09-16T05:18:36.653 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.654+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.653 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.654+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740399840}) 2024-09-16T05:18:36.653 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.654+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2313388515}) 2024-09-16T05:18:36.655 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.654+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.655 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.654+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=442971589}) 2024-09-16T05:18:36.655 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.654+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4116464973}) 2024-09-16T05:18:36.655 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.654+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.655 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.654+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=442971589}) 2024-09-16T05:18:36.655 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.654+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1266259456}) 2024-09-16T05:18:36.657 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.658+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.657 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.658+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3352735257}) 2024-09-16T05:18:36.657 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.658+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=176055996}) 2024-09-16T05:18:36.657 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.658+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.657 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.658+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3352735257}) 2024-09-16T05:18:36.657 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.658+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=984384689}) 2024-09-16T05:18:36.659 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.658+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.659 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.658+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2602403018}) 2024-09-16T05:18:36.659 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.658+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1105042661}) 2024-09-16T05:18:36.659 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.658+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.659 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.658+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2602403018}) 2024-09-16T05:18:36.659 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.658+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1112064952}) 2024-09-16T05:18:36.661 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.662+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.661 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.662+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21049372}) 2024-09-16T05:18:36.661 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.662+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3563278750}) 2024-09-16T05:18:36.661 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.662+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.661 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.662+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21049372}) 2024-09-16T05:18:36.661 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.662+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1703027038}) 2024-09-16T05:18:36.663 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.662+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.663 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.662+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1644468367}) 2024-09-16T05:18:36.663 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.662+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3424820002}) 2024-09-16T05:18:36.663 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.662+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.663 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.662+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1644468367}) 2024-09-16T05:18:36.663 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.662+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334731262}) 2024-09-16T05:18:36.665 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.666+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.665 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.666+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879022903}) 2024-09-16T05:18:36.665 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.666+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=148014273}) 2024-09-16T05:18:36.665 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.666+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.665 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.666+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879022903}) 2024-09-16T05:18:36.665 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.666+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2581031856}) 2024-09-16T05:18:36.667 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.666+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.667 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.666+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464759836}) 2024-09-16T05:18:36.667 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.666+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1844609652}) 2024-09-16T05:18:36.667 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.666+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.667 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.666+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464759836}) 2024-09-16T05:18:36.667 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.666+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1730907119}) 2024-09-16T05:18:36.669 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.670+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.669 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.670+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2236951354}) 2024-09-16T05:18:36.669 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.670+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1363380061}) 2024-09-16T05:18:36.669 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.670+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.669 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.670+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2236951354}) 2024-09-16T05:18:36.669 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.670+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3599495062}) 2024-09-16T05:18:36.671 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.670+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.671 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.670+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799921318}) 2024-09-16T05:18:36.671 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.670+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=650255236}) 2024-09-16T05:18:36.671 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.670+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.671 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.670+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799921318}) 2024-09-16T05:18:36.671 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.670+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2541845181}) 2024-09-16T05:18:36.673 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.674+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.673 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.674+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215519459}) 2024-09-16T05:18:36.673 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.674+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=566424844}) 2024-09-16T05:18:36.673 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.674+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.673 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.674+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215519459}) 2024-09-16T05:18:36.674 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.674+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2038260345}) 2024-09-16T05:18:36.674 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.674+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.675 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.674+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=718587866}) 2024-09-16T05:18:36.675 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.674+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=509635579}) 2024-09-16T05:18:36.675 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.674+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.675 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.674+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=718587866}) 2024-09-16T05:18:36.675 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.674+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2996193946}) 2024-09-16T05:18:36.676 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.678+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.676 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.678+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3621556603}) 2024-09-16T05:18:36.676 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.678+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3637365378}) 2024-09-16T05:18:36.676 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.678+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.677 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.678+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3621556603}) 2024-09-16T05:18:36.677 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.678+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=846576689}) 2024-09-16T05:18:36.678 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.678+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.678 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.678+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1820663674}) 2024-09-16T05:18:36.678 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.678+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3490146622}) 2024-09-16T05:18:36.678 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.678+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.678 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.678+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1820663674}) 2024-09-16T05:18:36.679 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.678+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2127252631}) 2024-09-16T05:18:36.680 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.682+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.680 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.682+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4246766526}) 2024-09-16T05:18:36.680 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.682+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=530705870}) 2024-09-16T05:18:36.681 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.682+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.681 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.682+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4246766526}) 2024-09-16T05:18:36.681 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.682+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=122404754}) 2024-09-16T05:18:36.682 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.682+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.682 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.682+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3754919888}) 2024-09-16T05:18:36.682 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.682+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3212295552}) 2024-09-16T05:18:36.682 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.682+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.682 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.682+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3754919888}) 2024-09-16T05:18:36.682 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.682+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1921396586}) 2024-09-16T05:18:36.684 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.686+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.684 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.686+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3352110890}) 2024-09-16T05:18:36.684 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.686+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2938198140}) 2024-09-16T05:18:36.684 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.686+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.684 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.686+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3352110890}) 2024-09-16T05:18:36.684 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.686+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3991541924}) 2024-09-16T05:18:36.685 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.686+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.686 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.686+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1295841859}) 2024-09-16T05:18:36.686 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.686+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616238668}) 2024-09-16T05:18:36.686 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.686+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.686 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.686+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1295841859}) 2024-09-16T05:18:36.686 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.686+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2478901309}) 2024-09-16T05:18:36.687 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.690+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.687 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.690+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178323639}) 2024-09-16T05:18:36.688 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.690+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1657870152}) 2024-09-16T05:18:36.688 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.690+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.688 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.690+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178323639}) 2024-09-16T05:18:36.688 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.690+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2912994796}) 2024-09-16T05:18:36.689 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.690+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.689 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.690+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2572284203}) 2024-09-16T05:18:36.689 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.690+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3818950100}) 2024-09-16T05:18:36.689 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.690+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.689 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.690+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2572284203}) 2024-09-16T05:18:36.689 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.690+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3640047528}) 2024-09-16T05:18:36.691 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.694+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.691 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.694+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494812367}) 2024-09-16T05:18:36.691 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.694+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2298112197}) 2024-09-16T05:18:36.691 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.694+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.692 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.694+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494812367}) 2024-09-16T05:18:36.692 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.694+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1265960970}) 2024-09-16T05:18:36.693 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.694+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.693 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.694+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2356521025}) 2024-09-16T05:18:36.693 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.694+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2065394907}) 2024-09-16T05:18:36.693 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.694+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.693 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.694+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2356521025}) 2024-09-16T05:18:36.693 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.694+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618652851}) 2024-09-16T05:18:36.695 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.694+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.695 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517130251}) 2024-09-16T05:18:36.695 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692723677}) 2024-09-16T05:18:36.695 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.695 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517130251}) 2024-09-16T05:18:36.695 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3300006933}) 2024-09-16T05:18:36.697 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.697 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242855855}) 2024-09-16T05:18:36.697 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2749861518}) 2024-09-16T05:18:36.697 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.697 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242855855}) 2024-09-16T05:18:36.697 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1442590856}) 2024-09-16T05:18:36.699 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.699 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2035858262}) 2024-09-16T05:18:36.699 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2948291863}) 2024-09-16T05:18:36.699 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.699 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2035858262}) 2024-09-16T05:18:36.699 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.698+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394751442}) 2024-09-16T05:18:36.700 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.702+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.700 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.702+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650678557}) 2024-09-16T05:18:36.700 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.702+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3684374779}) 2024-09-16T05:18:36.700 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.702+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.700 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.702+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650678557}) 2024-09-16T05:18:36.701 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.702+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1798634992}) 2024-09-16T05:18:36.702 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.702+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.702 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.702+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2261446392}) 2024-09-16T05:18:36.702 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.702+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3318618478}) 2024-09-16T05:18:36.702 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.702+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.702 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.702+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2261446392}) 2024-09-16T05:18:36.702 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.702+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=846670119}) 2024-09-16T05:18:36.704 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.706+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.704 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.706+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3902250617}) 2024-09-16T05:18:36.704 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.706+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=870778958}) 2024-09-16T05:18:36.704 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.706+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.704 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.706+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3902250617}) 2024-09-16T05:18:36.704 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.706+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1884233447}) 2024-09-16T05:18:36.706 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.706+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.706 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.706+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2643139215}) 2024-09-16T05:18:36.706 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.706+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2172323730}) 2024-09-16T05:18:36.706 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.706+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.706 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.706+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2643139215}) 2024-09-16T05:18:36.706 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.706+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3443889624}) 2024-09-16T05:18:36.708 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.710+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.708 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.710+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=3130129995,mdsmap=2923618896}) 2024-09-16T05:18:36.708 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.710+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2437643630}) 2024-09-16T05:18:36.708 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.710+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.708 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.710+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=3130129995,mdsmap=2923618896}) 2024-09-16T05:18:36.708 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.710+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3579357711}) 2024-09-16T05:18:36.710 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.710+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.710 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.710+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=24} crc {mdsmap=3140514009,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=533376590,mon_config_key=1589051853,monmap=3371862038,osd_metadata=487805116,osd_pg_creating=2547703029,osd_snap=534452749,osdmap=2327467683}) 2024-09-16T05:18:36.710 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.710+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=25} crc {mdsmap=3140514009,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=1718051513,mon_config_key=2157023358,monmap=3371862038,osd_metadata=4182888218,osd_pg_creating=2547703029,osd_snap=3372344628,osdmap=1479444321}) 2024-09-16T05:18:36.711 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.710+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.711 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.710+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=24} crc {mdsmap=3140514009,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=533376590,mon_config_key=1589051853,monmap=3371862038,osd_metadata=487805116,osd_pg_creating=2547703029,osd_snap=534452749,osdmap=2327467683}) 2024-09-16T05:18:36.711 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.710+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=30,osdmap=24} crc {mdsmap=1228700967,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=533376590,mon_config_key=1589051853,monmap=3045472550,osd_metadata=4163472531,osd_pg_creating=2547703029,osd_snap=3330700606,osdmap=3456507565}) 2024-09-16T05:18:36.713 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.714+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.713 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.714+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4071678405}) 2024-09-16T05:18:36.713 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.714+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3285761205}) 2024-09-16T05:18:36.713 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.714+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.713 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.714+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4071678405}) 2024-09-16T05:18:36.713 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.714+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=271293502}) 2024-09-16T05:18:36.715 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.718+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.715 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.718+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=794399721}) 2024-09-16T05:18:36.715 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.718+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=826004195}) 2024-09-16T05:18:36.715 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.718+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.715 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.718+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=794399721}) 2024-09-16T05:18:36.715 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.718+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=498220806}) 2024-09-16T05:18:36.717 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.718+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.717 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.718+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4187301207}) 2024-09-16T05:18:36.717 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.718+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4039217244}) 2024-09-16T05:18:36.717 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.718+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.717 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.718+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4187301207}) 2024-09-16T05:18:36.717 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.718+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=715842964}) 2024-09-16T05:18:36.719 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.722+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.719 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.722+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1608010355}) 2024-09-16T05:18:36.720 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.722+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=901922795}) 2024-09-16T05:18:36.720 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.722+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.720 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.722+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1608010355}) 2024-09-16T05:18:36.720 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.722+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=275979178}) 2024-09-16T05:18:36.724 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.726+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.724 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.726+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1634867815}) 2024-09-16T05:18:36.724 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.726+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=849628459}) 2024-09-16T05:18:36.724 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.726+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.724 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.726+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1634867815}) 2024-09-16T05:18:36.724 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.726+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=123900556}) 2024-09-16T05:18:36.728 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.730+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.728 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.730+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3311269599}) 2024-09-16T05:18:36.728 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.730+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=726986316}) 2024-09-16T05:18:36.728 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.730+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.729 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.730+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3311269599}) 2024-09-16T05:18:36.729 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.730+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=934063261}) 2024-09-16T05:18:36.729 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.730+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.729 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.730+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=4} crc {osdmap=3699949302}) 2024-09-16T05:18:36.730 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.730+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=3} crc {osdmap=1989011767}) 2024-09-16T05:18:36.730 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.730+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:18:36.730 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.730+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=4} crc {osdmap=3699949302}) 2024-09-16T05:18:36.730 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:18:36.730+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=4} crc {osdmap=3803923087}) 2024-09-16T05:18:37.637 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:18:37.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:18:37.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 10 monclient: get_monmap_and_config 2024-09-16T05:18:37.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 10 monclient: build_initial_monmap 2024-09-16T05:18:37.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 10 monclient: monmap: 2024-09-16T05:18:37.768 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:37.769 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:37.769 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:37.773451+0000 2024-09-16T05:18:37.769 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:37.773451+0000 2024-09-16T05:18:37.769 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:37.769 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:37.769 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:37.769 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:37.769 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:37.769 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:37.769 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:37.770 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:37.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 1 Processor -- start 2024-09-16T05:18:37.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 1 -- start start 2024-09-16T05:18:37.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 10 monclient: init 2024-09-16T05:18:37.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:37.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T05:18:37.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feecc0a0ec0 0x7feecc09dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:37.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 10 monclient(hunting): picked mon.noname-b con 0x7feecc0a0ec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:37.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feecc09e000 addr v1:172.21.15.149:6789/0 2024-09-16T05:18:37.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7feecc139640 0x7feecc13ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:37.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 10 monclient(hunting): picked mon.noname-c con 0x7feecc139640 addr v2:172.21.15.149:3300/0 2024-09-16T05:18:37.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feecc09a760 con 0x7feecc09e000 2024-09-16T05:18:37.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:37.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7feecc0a12a0 con 0x7feecc139640 2024-09-16T05:18:37.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:37.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7feecc0994b0 con 0x7feecc0a0ec0 2024-09-16T05:18:37.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:37.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed2f50640 10 monclient(hunting): authenticate will time out at 1602.402865s 2024-09-16T05:18:37.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feecc0a0ec0 0x7feecc09dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:37.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feecc0a0ec0 0x7feecc09dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:46560/0 (socket says 172.21.15.114:46560) 2024-09-16T05:18:37.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 1 -- 172.21.15.114:0/3240951515 learned_addr learned my addr 172.21.15.114:0/3240951515 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:37.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 10 monclient(hunting): get_auth_request con 0x7feecc0a0ec0 auth_method 0 2024-09-16T05:18:37.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:37.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:37.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:37.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed14ed640 1 --2- 172.21.15.114:0/3240951515 >> v2:172.21.15.149:3300/0 conn(0x7feecc139640 0x7feecc13ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:37.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:37.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:37.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:37.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed14ed640 10 monclient(hunting): get_auth_request con 0x7feecc139640 auth_method 0 2024-09-16T05:18:37.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed14ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:37.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed14ed640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:37.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed14ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:37.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 10 monclient(hunting): handle_auth_done global_id 94130 payload 274 2024-09-16T05:18:37.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 1 -- 172.21.15.114:0/3240951515 >> v2:172.21.15.149:3300/0 conn(0x7feecc139640 msgr2=0x7feecc13ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:37.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 1 --2- 172.21.15.114:0/3240951515 >> v2:172.21.15.149:3300/0 conn(0x7feecc139640 0x7feecc13ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:37.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 1 -- 172.21.15.114:0/3240951515 >> v1:172.21.15.149:6789/0 conn(0x7feecc09e000 legacy=0x7feecc09e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:37.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 10 monclient: _finish_hunting 0 2024-09-16T05:18:37.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 1 monclient: found mon.noname-b 2024-09-16T05:18:37.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:37.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:18:37.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.770+0000 7feed0cec640 1 -- 172.21.15.114:0/3240951515 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feecc0a1870 con 0x7feecc0a0ec0 2024-09-16T05:18:37.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed0cec640 1 --2- 172.21.15.114:0/3240951515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feecc0a0ec0 0x7feecc09dac0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7feebc009870 tx=0x7feebc01c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c54d82dfb275c31 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:37.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feecb7fe640 1 -- 172.21.15.114:0/3240951515 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feebc0b1070 con 0x7feecc0a0ec0 2024-09-16T05:18:37.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feecb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:37.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feecb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:18:37.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feecb7fe640 10 monclient: dump: 2024-09-16T05:18:37.781 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:37.781 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:37.781 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:37.781 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:37.781 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:37.781 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:37.781 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:37.782 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:37.782 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:37.782 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:37.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feecb7fe640 10 monclient: _finish_auth 0 2024-09-16T05:18:37.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feecb7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:18:37.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feecb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:37.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feecb7fe640 1 -- 172.21.15.114:0/3240951515 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feebc0ac620 con 0x7feecc0a0ec0 2024-09-16T05:18:37.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 5 monclient: authenticate success, global_id 94130 2024-09-16T05:18:37.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:37.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feecb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:37.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:37.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 4 set_mon_vals no callback set 2024-09-16T05:18:37.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feecb7fe640 1 -- 172.21.15.114:0/3240951515 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feebc0ac9f0 con 0x7feecc0a0ec0 2024-09-16T05:18:37.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feecb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:37.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feecb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:18:37.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feecb7fe640 10 monclient: dump: 2024-09-16T05:18:37.783 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:37.783 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:37.783 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:37.783 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:37.783 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:37.783 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:37.783 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:37.783 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:37.783 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:37.783 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:37.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 10 monclient: shutdown 2024-09-16T05:18:37.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:37.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 1 -- 172.21.15.114:0/3240951515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feecc0a0ec0 msgr2=0x7feecc09dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:37.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 1 --2- 172.21.15.114:0/3240951515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feecc0a0ec0 0x7feecc09dac0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7feebc009870 tx=0x7feebc01c4b0 comp rx=0 tx=0).stop 2024-09-16T05:18:37.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 1 -- 172.21.15.114:0/3240951515 shutdown_connections 2024-09-16T05:18:37.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 1 --2- 172.21.15.114:0/3240951515 >> v2:172.21.15.149:3300/0 conn(0x7feecc139640 0x7feecc13ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:37.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 1 --2- 172.21.15.114:0/3240951515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feecc0a0ec0 0x7feecc09dac0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:37.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 1 -- 172.21.15.114:0/3240951515 >> 172.21.15.114:0/3240951515 conn(0x7feecc0a1fb0 msgr2=0x7feecc0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:37.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 1 -- 172.21.15.114:0/3240951515 shutdown_connections 2024-09-16T05:18:37.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 1 -- 172.21.15.114:0/3240951515 wait complete. 2024-09-16T05:18:37.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 10 monclient: build_initial_monmap 2024-09-16T05:18:37.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 10 monclient: monmap: 2024-09-16T05:18:37.784 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:37.784 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:37.784 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:37.784 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:37.785 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:37.785 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:37.785 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:37.785 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:37.785 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:37.785 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:37.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 1 librados: starting msgr at 2024-09-16T05:18:37.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 1 librados: starting objecter 2024-09-16T05:18:37.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 1 Processor -- start 2024-09-16T05:18:37.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 1 -- start start 2024-09-16T05:18:37.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 1 librados: setting wanted keys 2024-09-16T05:18:37.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 1 librados: calling monclient init 2024-09-16T05:18:37.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.774+0000 7feed2f50640 10 monclient: init 2024-09-16T05:18:37.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:37.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:18:37.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feecc09e000 0x7feecc138f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:37.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient(hunting): picked mon.noname-c con 0x7feecc09e000 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:37.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feecc0a0ec0 0x7feecc08e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:37.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient(hunting): picked mon.noname-b con 0x7feecc0a0ec0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:37.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feecc139640 0x7feecc08e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:37.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient(hunting): picked mon.noname-a con 0x7feecc139640 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:37.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:37.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feecc13e910 con 0x7feecc139640 2024-09-16T05:18:37.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:37.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7feecc13e790 con 0x7feecc0a0ec0 2024-09-16T05:18:37.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:37.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7feecc13ea90 con 0x7feecc09e000 2024-09-16T05:18:37.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:37.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient(hunting): authenticate will time out at 1602.408266s 2024-09-16T05:18:37.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feecc09e000 0x7feecc138f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:37.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feecc09e000 0x7feecc138f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:46568/0 (socket says 172.21.15.114:46568) 2024-09-16T05:18:37.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 1 -- 172.21.15.114:0/807402816 learned_addr learned my addr 172.21.15.114:0/807402816 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:37.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 10 monclient(hunting): get_auth_request con 0x7feecc09e000 auth_method 0 2024-09-16T05:18:37.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:37.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:37.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:37.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed14ed640 1 --2- 172.21.15.114:0/807402816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feecc139640 0x7feecc08e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:37.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed14ed640 10 monclient(hunting): get_auth_request con 0x7feecc139640 auth_method 0 2024-09-16T05:18:37.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed14ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:37.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed14ed640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:37.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed14ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:37.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:37.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:37.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:37.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed14ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:37.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed14ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:37.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feecbfff640 1 --2- 172.21.15.114:0/807402816 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feecc0a0ec0 0x7feecc08e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:37.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed14ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:37.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feecbfff640 10 monclient(hunting): get_auth_request con 0x7feecc0a0ec0 auth_method 0 2024-09-16T05:18:37.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feecbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:37.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feecbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:37.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:37.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 10 monclient(hunting): handle_auth_done global_id 94133 payload 824 2024-09-16T05:18:37.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 1 -- 172.21.15.114:0/807402816 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feecc0a0ec0 msgr2=0x7feecc08e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:37.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 1 --2- 172.21.15.114:0/807402816 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feecc0a0ec0 0x7feecc08e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:37.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 1 -- 172.21.15.114:0/807402816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feecc139640 msgr2=0x7feecc08e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:37.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 1 --2- 172.21.15.114:0/807402816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feecc139640 0x7feecc08e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:37.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 10 monclient: _finish_hunting 0 2024-09-16T05:18:37.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 1 monclient: found mon.noname-c 2024-09-16T05:18:37.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:37.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:18:37.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 1 -- 172.21.15.114:0/807402816 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feecc091bb0 con 0x7feecc09e000 2024-09-16T05:18:37.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed0cec640 1 --2- 172.21.15.114:0/807402816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feecc09e000 0x7feecc138f60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7feebc01ab60 tx=0x7feebc019250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:37.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 10 client.?.objecter ms_handle_connect 0x7feecc09e000 2024-09-16T05:18:37.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:37.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 1 -- 172.21.15.114:0/807402816 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feebc0ba070 con 0x7feecc09e000 2024-09-16T05:18:37.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:37.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:18:37.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 10 monclient: dump: 2024-09-16T05:18:37.790 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:37.790 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:37.791 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:37.791 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:37.791 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:37.791 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:37.791 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:37.791 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:37.791 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:37.791 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:37.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 10 monclient: _finish_auth 0 2024-09-16T05:18:37.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 10 monclient: _check_auth_tickets 2024-09-16T05:18:37.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:37.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 1 -- 172.21.15.114:0/807402816 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feebc0b5400 con 0x7feecc09e000 2024-09-16T05:18:37.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:37.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 1 -- 172.21.15.114:0/807402816 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feebc0b1040 con 0x7feecc09e000 2024-09-16T05:18:37.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 5 monclient: authenticate success, global_id 94133 2024-09-16T05:18:37.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient: _renew_subs 2024-09-16T05:18:37.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:18:37.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 1 -- 172.21.15.114:0/807402816 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feecc091e90 con 0x7feecc09e000 2024-09-16T05:18:37.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feecaffd640 4 set_mon_vals no callback set 2024-09-16T05:18:37.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:37.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:18:37.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 10 monclient: dump: 2024-09-16T05:18:37.792 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:37.792 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:37.792 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:37.792 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:37.792 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:37.793 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:37.793 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:37.793 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:37.793 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:37.793 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:37.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 client.94133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:37.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient: _renew_subs 2024-09-16T05:18:37.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:18:37.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 1 -- 172.21.15.114:0/807402816 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feecc0a44e0 con 0x7feecc09e000 2024-09-16T05:18:37.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 1 librados: init done 2024-09-16T05:18:37.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 1 -- 172.21.15.114:0/807402816 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7feebc0bd070 con 0x7feecc09e000 2024-09-16T05:18:37.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:37.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:37.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:18:37.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feed2f50640 1 -- 172.21.15.114:0/807402816 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feecc13bb50 con 0x7feecc09e000 2024-09-16T05:18:37.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 1 --2- 172.21.15.114:0/807402816 >> v2:172.21.15.149:6812/1185531135 conn(0x7feeb0037810 0x7feeb0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:37.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 1 -- 172.21.15.114:0/807402816 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(431..431 src has 1..431) ==== 5406+0+0 (secure 0 0 0) 0x7feebc0e2180 con 0x7feecc09e000 2024-09-16T05:18:37.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 10 client.94133.objecter ms_dispatch 0x7feecc000ba0 osd_map(431..431 src has 1..431) 2024-09-16T05:18:37.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 3 client.94133.objecter handle_osd_map got epochs [431,431] > 0 2024-09-16T05:18:37.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 3 client.94133.objecter handle_osd_map decoding full epoch 431 2024-09-16T05:18:37.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 20 client.94133.objecter dump_active .. 0 homeless 2024-09-16T05:18:37.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feecbfff640 1 --2- 172.21.15.114:0/807402816 >> v2:172.21.15.149:6812/1185531135 conn(0x7feeb0037810 0x7feeb0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:37.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feecbfff640 10 monclient: get_auth_request con 0x7feeb0037810 auth_method 0 2024-09-16T05:18:37.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feec97fa640 10 client.94133.objecter ms_handle_connect 0x7feeb0037810 2024-09-16T05:18:37.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.778+0000 7feecbfff640 1 --2- 172.21.15.114:0/807402816 >> v2:172.21.15.149:6812/1185531135 conn(0x7feeb0037810 0x7feeb0039cd0 secure :-1 s=READY pgs=3543 cs=0 l=1 rev1=1 crypto rx=0x7feecc090900 tx=0x7feec4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:37.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.782+0000 7feec97fa640 1 -- 172.21.15.114:0/807402816 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7feebc016130 con 0x7feecc09e000 2024-09-16T05:18:37.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.782+0000 7feec97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:37.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:37.782+0000 7feec97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.006+0000 7feed2f50640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:18:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.006+0000 7feed2f50640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.006+0000 7feed2f50640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:18:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.006+0000 7feed2f50640 1 -- 172.21.15.114:0/807402816 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7feecc09c6a0 con 0x7feecc09e000 2024-09-16T05:18:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.006+0000 7feec97fa640 1 -- 172.21.15.114:0/807402816 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7feebc0bd360 con 0x7feecc09e000 2024-09-16T05:18:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.006+0000 7feec97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:38.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.006+0000 7feec97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:38.007 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:18:38.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 10 librados: watch_flush enter 2024-09-16T05:18:38.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 10 librados: watch_flush exit 2024-09-16T05:18:38.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 20 client.94133.objecter shutdown clearing up homeless session... 2024-09-16T05:18:38.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 10 client.94133.objecter successfully canceled tick 2024-09-16T05:18:38.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 1 -- 172.21.15.114:0/807402816 >> v2:172.21.15.149:6812/1185531135 conn(0x7feeb0037810 msgr2=0x7feeb0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 1 --2- 172.21.15.114:0/807402816 >> v2:172.21.15.149:6812/1185531135 conn(0x7feeb0037810 0x7feeb0039cd0 secure :-1 s=READY pgs=3543 cs=0 l=1 rev1=1 crypto rx=0x7feecc090900 tx=0x7feec4009040 comp rx=0 tx=0).stop 2024-09-16T05:18:38.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 10 monclient: shutdown 2024-09-16T05:18:38.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:38.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 1 -- 172.21.15.114:0/807402816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feecc09e000 msgr2=0x7feecc138f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 1 --2- 172.21.15.114:0/807402816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feecc09e000 0x7feecc138f60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7feebc01ab60 tx=0x7feebc019250 comp rx=0 tx=0).stop 2024-09-16T05:18:38.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 1 -- 172.21.15.114:0/807402816 shutdown_connections 2024-09-16T05:18:38.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 1 --2- 172.21.15.114:0/807402816 >> v2:172.21.15.149:6812/1185531135 conn(0x7feeb0037810 0x7feeb0039cd0 unknown :-1 s=CLOSED pgs=3543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 1 --2- 172.21.15.114:0/807402816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feecc139640 0x7feecc08e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 1 --2- 172.21.15.114:0/807402816 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feecc0a0ec0 0x7feecc08e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 1 --2- 172.21.15.114:0/807402816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feecc09e000 0x7feecc138f60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 1 -- 172.21.15.114:0/807402816 >> 172.21.15.114:0/807402816 conn(0x7feecc0a1fb0 msgr2=0x7feecc13a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:38.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 1 -- 172.21.15.114:0/807402816 shutdown_connections 2024-09-16T05:18:38.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 1 -- 172.21.15.114:0/807402816 wait complete. 2024-09-16T05:18:38.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.010+0000 7feed2f50640 1 librados: shutdown 2024-09-16T05:18:38.023 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:18:38.023 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:18:38.023 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:18:38.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:18:38.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 10 monclient: get_monmap_and_config 2024-09-16T05:18:38.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 10 monclient: build_initial_monmap 2024-09-16T05:18:38.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 10 monclient: monmap: 2024-09-16T05:18:38.139 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:38.139 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.139 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:38.144422+0000 2024-09-16T05:18:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:38.144422+0000 2024-09-16T05:18:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:38.140 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 1 Processor -- start 2024-09-16T05:18:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 1 -- start start 2024-09-16T05:18:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 10 monclient: init 2024-09-16T05:18:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-16T05:18:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fac4c072580 0x7fac4c0a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 10 monclient(hunting): picked mon.noname-a con 0x7fac4c072580 addr v2:172.21.15.114:3300/0 2024-09-16T05:18:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac4c072960 addr v1:172.21.15.114:6789/0 2024-09-16T05:18:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac4c13c640 addr v1:172.21.15.149:6789/0 2024-09-16T05:18:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac4c0a1050 con 0x7fac4c072960 2024-09-16T05:18:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac4c0a1330 con 0x7fac4c13c640 2024-09-16T05:18:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fac4c0a1610 con 0x7fac4c072580 2024-09-16T05:18:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac52816640 10 monclient(hunting): authenticate will time out at 1602.773697s 2024-09-16T05:18:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fac4c072580 0x7fac4c0a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4b7fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fac4c072960 0x7fac4c072d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:41198/0 (socket says 172.21.15.114:41198) 2024-09-16T05:18:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4b7fe640 1 -- 172.21.15.114:0/3562116616 learned_addr learned my addr 172.21.15.114:0/3562116616 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:18:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 10 monclient(hunting): get_auth_request con 0x7fac4c072580 auth_method 0 2024-09-16T05:18:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4affd640 1 -- 172.21.15.114:0/3562116616 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1522291209 0 0) 0x7fac4c0a1050 con 0x7fac4c072960 2024-09-16T05:18:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4affd640 10 monclient(hunting): _negotiate 2024-09-16T05:18:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4affd640 10 monclient(hunting): my global_id is 74439 2024-09-16T05:18:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4affd640 1 -- 172.21.15.114:0/3562116616 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac34003530 con 0x7fac4c072960 2024-09-16T05:18:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 10 monclient(hunting): handle_auth_done global_id 74436 payload 274 2024-09-16T05:18:38.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 1 -- 172.21.15.114:0/3562116616 >> v1:172.21.15.149:6789/0 conn(0x7fac4c13c640 legacy=0x7fac4c13ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:38.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 1 -- 172.21.15.114:0/3562116616 >> v1:172.21.15.114:6789/0 conn(0x7fac4c072960 legacy=0x7fac4c072d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:18:38.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 1 monclient: found mon.noname-a 2024-09-16T05:18:38.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:38.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:38.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.142+0000 7fac4bfff640 1 -- 172.21.15.114:0/3562116616 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c09ca80 con 0x7fac4c072580 2024-09-16T05:18:38.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4bfff640 1 --2- 172.21.15.114:0/3562116616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c072580 0x7fac4c0a1d80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fac44009d90 tx=0x7fac4401c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9e2084ce5a3a00 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4affd640 1 -- 172.21.15.114:0/3562116616 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac440b2070 con 0x7fac4c072580 2024-09-16T05:18:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4affd640 10 monclient: dump: 2024-09-16T05:18:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.154 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.154 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.154 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4affd640 10 monclient: _finish_auth 0 2024-09-16T05:18:38.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4affd640 10 monclient: _check_auth_tickets 2024-09-16T05:18:38.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4affd640 1 -- 172.21.15.114:0/3562116616 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac440a58d0 con 0x7fac4c072580 2024-09-16T05:18:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 5 monclient: authenticate success, global_id 74436 2024-09-16T05:18:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4affd640 1 -- 172.21.15.114:0/3562116616 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac440a5ca0 con 0x7fac4c072580 2024-09-16T05:18:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 4 set_mon_vals no callback set 2024-09-16T05:18:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4affd640 10 monclient: dump: 2024-09-16T05:18:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.156 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient: shutdown 2024-09-16T05:18:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 -- 172.21.15.114:0/3562116616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c072580 msgr2=0x7fac4c0a1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 --2- 172.21.15.114:0/3562116616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c072580 0x7fac4c0a1d80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fac44009d90 tx=0x7fac4401c7f0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 -- 172.21.15.114:0/3562116616 shutdown_connections 2024-09-16T05:18:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 --2- 172.21.15.114:0/3562116616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c072580 0x7fac4c0a1d80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 -- 172.21.15.114:0/3562116616 >> 172.21.15.114:0/3562116616 conn(0x7fac4c0a6310 msgr2=0x7fac4c0a2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 -- 172.21.15.114:0/3562116616 shutdown_connections 2024-09-16T05:18:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 -- 172.21.15.114:0/3562116616 wait complete. 2024-09-16T05:18:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient: build_initial_monmap 2024-09-16T05:18:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient: monmap: 2024-09-16T05:18:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:38.158 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 librados: starting msgr at 2024-09-16T05:18:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 librados: starting objecter 2024-09-16T05:18:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 Processor -- start 2024-09-16T05:18:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 -- start start 2024-09-16T05:18:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 librados: setting wanted keys 2024-09-16T05:18:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 librados: calling monclient init 2024-09-16T05:18:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient: init 2024-09-16T05:18:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:18:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac4c072580 0x7fac4c12d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient(hunting): picked mon.noname-b con 0x7fac4c072580 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c072960 0x7fac4c12db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient(hunting): picked mon.noname-a con 0x7fac4c072960 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac4c13c640 0x7fac4c13cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient(hunting): picked mon.noname-c con 0x7fac4c13c640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c142d00 con 0x7fac4c072960 2024-09-16T05:18:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c142b80 con 0x7fac4c072580 2024-09-16T05:18:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fac4c142e80 con 0x7fac4c13c640 2024-09-16T05:18:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient(hunting): authenticate will time out at 1602.778599s 2024-09-16T05:18:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c072960 0x7fac4c12db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac50db3640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac4c13c640 0x7fac4c13cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c072960 0x7fac4c12db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34014/0 (socket says 172.21.15.114:34014) 2024-09-16T05:18:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 1 -- 172.21.15.114:0/3824175201 learned_addr learned my addr 172.21.15.114:0/3824175201 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request con 0x7fac4c072960 auth_method 0 2024-09-16T05:18:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac50db3640 10 monclient(hunting): get_auth_request con 0x7fac4c13c640 auth_method 0 2024-09-16T05:18:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac50db3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac50db3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac50db3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac50db3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac50db3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac50db3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_done global_id 74442 payload 824 2024-09-16T05:18:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4bfff640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac4c072580 0x7fac4c12d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 1 -- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac4c13c640 msgr2=0x7fac4c13cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac4c13c640 0x7fac4c13cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 1 -- 172.21.15.114:0/3824175201 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac4c072580 msgr2=0x7fac4c12d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac4c072580 0x7fac4c12d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:18:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 1 monclient: found mon.noname-a 2024-09-16T05:18:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 1 -- 172.21.15.114:0/3824175201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c06d180 con 0x7fac4c072960 2024-09-16T05:18:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac50db3640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac4c13c640 0x7fac4c13cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:18:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4b7fe640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c072960 0x7fac4c12db90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00c480 tx=0x7fac3c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac48ff9640 10 client.?.objecter ms_handle_connect 0x7fac4c072960 2024-09-16T05:18:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac48ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac48ff9640 1 -- 172.21.15.114:0/3824175201 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c017070 con 0x7fac4c072960 2024-09-16T05:18:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac48ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac48ff9640 10 monclient: dump: 2024-09-16T05:18:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.164 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac48ff9640 10 monclient: _finish_auth 0 2024-09-16T05:18:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac48ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:18:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac48ff9640 1 -- 172.21.15.114:0/3824175201 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac3c005cf0 con 0x7fac4c072960 2024-09-16T05:18:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 5 monclient: authenticate success, global_id 74442 2024-09-16T05:18:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient: _renew_subs 2024-09-16T05:18:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 -- 172.21.15.114:0/3824175201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac4c13e880 con 0x7fac4c072960 2024-09-16T05:18:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac48ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac48ff9640 1 -- 172.21.15.114:0/3824175201 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c007290 con 0x7fac4c072960 2024-09-16T05:18:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac48ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac48ff9640 10 monclient: dump: 2024-09-16T05:18:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.165 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.166 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 client.74442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient: _renew_subs 2024-09-16T05:18:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac4a7fc640 4 set_mon_vals no callback set 2024-09-16T05:18:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 -- 172.21.15.114:0/3824175201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac4c132320 con 0x7fac4c072960 2024-09-16T05:18:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 1 librados: init done 2024-09-16T05:18:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.146+0000 7fac52816640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac52816640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c13d160 0x7fac4c13d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac52816640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac52816640 1 -- 172.21.15.114:0/3824175201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c13f440 con 0x7fac4c13d160 2024-09-16T05:18:38.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac4bfff640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c13d160 0x7fac4c13d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac4bfff640 10 monclient: get_auth_request con 0x7fac4c13d160 auth_method 0 2024-09-16T05:18:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 1 -- 172.21.15.114:0/3824175201 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fac3c01a070 con 0x7fac4c072960 2024-09-16T05:18:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 1 --2- 172.21.15.114:0/3824175201 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac38037810 0x7fac38039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac4bfff640 10 monclient(hunting): handle_auth_done global_id 74445 payload 824 2024-09-16T05:18:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 1 -- 172.21.15.114:0/3824175201 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(432..432 src has 1..432) ==== 5462+0+0 (secure 0 0 0) 0x7fac3c048270 con 0x7fac4c072960 2024-09-16T05:18:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 10 client.74442.objecter ms_dispatch 0x7fac4c000ba0 osd_map(432..432 src has 1..432) 2024-09-16T05:18:38.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac4bfff640 1 -- 172.21.15.114:0/3824175201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac4c0a1d80 con 0x7fac4c13d160 2024-09-16T05:18:38.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 3 client.74442.objecter handle_osd_map got epochs [432,432] > 0 2024-09-16T05:18:38.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 3 client.74442.objecter handle_osd_map decoding full epoch 432 2024-09-16T05:18:38.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 20 client.74442.objecter dump_active .. 0 homeless 2024-09-16T05:18:38.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 10 client.74442.objecter ms_handle_connect 0x7fac4c13d160 2024-09-16T05:18:38.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 10 client.74442.objecter resend_mon_ops 2024-09-16T05:18:38.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac4bfff640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c13d160 0x7fac4c13d540 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fac4c138140 tx=0x7fac44019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 1 -- 172.21.15.114:0/3824175201 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac440bd030 con 0x7fac4c13d160 2024-09-16T05:18:38.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac50db3640 1 --2- 172.21.15.114:0/3824175201 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac38037810 0x7fac38039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:38.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 10 monclient: dump: 2024-09-16T05:18:38.168 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.169 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.169 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.169 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.169 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.169 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.169 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.169 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.169 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.169 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac50db3640 10 monclient: get_auth_request con 0x7fac38037810 auth_method 0 2024-09-16T05:18:38.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 1 -- 172.21.15.114:0/3824175201 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fac440b8470 con 0x7fac4c13d160 2024-09-16T05:18:38.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:38.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:38.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 1 -- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c13d160 msgr2=0x7fac4c13d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c13d160 0x7fac4c13d540 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fac4c138140 tx=0x7fac44019590 comp rx=0 tx=0).stop 2024-09-16T05:18:38.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 1 -- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c13d160 msgr2=0x7fac4c13d540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:38.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c13d160 0x7fac4c13d540 secure :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fac4c138140 tx=0x7fac44019590 comp rx=0 tx=0).stop 2024-09-16T05:18:38.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac48ff9640 10 client.74442.objecter ms_handle_connect 0x7fac38037810 2024-09-16T05:18:38.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.149+0000 7fac50db3640 1 --2- 172.21.15.114:0/3824175201 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac38037810 0x7fac38039cd0 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 crypto rx=0x7fac40007fd0 tx=0x7fac40007dc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac52816640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:18:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac52816640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c13da80 0x7fac4c134c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac52816640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac52816640 1 -- 172.21.15.114:0/3824175201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c132840 con 0x7fac4c13da80 2024-09-16T05:18:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac4bfff640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c13da80 0x7fac4c134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac4bfff640 10 monclient: get_auth_request con 0x7fac4c13da80 auth_method 0 2024-09-16T05:18:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac4bfff640 10 monclient(hunting): handle_auth_done global_id 74448 payload 824 2024-09-16T05:18:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac4bfff640 1 -- 172.21.15.114:0/3824175201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fac4c06bf60 con 0x7fac4c13da80 2024-09-16T05:18:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac4bfff640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c13da80 0x7fac4c134c10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fac44009d90 tx=0x7fac440a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac48ff9640 10 client.74442.objecter ms_handle_connect 0x7fac4c13da80 2024-09-16T05:18:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac48ff9640 10 client.74442.objecter resend_mon_ops 2024-09-16T05:18:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac48ff9640 1 -- 172.21.15.114:0/3824175201 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac440bd070 con 0x7fac4c13da80 2024-09-16T05:18:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac48ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac48ff9640 10 monclient: dump: 2024-09-16T05:18:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.173 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.174 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac48ff9640 1 -- 172.21.15.114:0/3824175201 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7fac440b8620 con 0x7fac4c13da80 2024-09-16T05:18:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac48ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:18:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac48ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac48ff9640 1 -- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c13da80 msgr2=0x7fac4c134c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac48ff9640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c13da80 0x7fac4c134c10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fac44009d90 tx=0x7fac440a7000 comp rx=0 tx=0).stop 2024-09-16T05:18:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac48ff9640 1 -- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c13da80 msgr2=0x7fac4c134c10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:38.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac48ff9640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c13da80 0x7fac4c134c10 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fac44009d90 tx=0x7fac440a7000 comp rx=0 tx=0).stop 2024-09-16T05:18:38.175 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:18:38.175 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:18:38.175 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:18:38.175 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:18:38.175 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 50, 2024-09-16T05:18:38.175 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 277209, 2024-09-16T05:18:38.175 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:18:38.175 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:18:38.175 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:18:38.175 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:18:38.175 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:38.176 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 4, 2024-09-16T05:18:38.176 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:38.176 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:18:38.176 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:18:38.176 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:38.176 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:38.176 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:38.176 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:38.176 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:38.176 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:38.176 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:38.176 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:38.176 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:38.177 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:38.177 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:38.177 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:38.177 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:18:38.177 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:18:38.177 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:38.177 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:38.177 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:38.177 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:38.177 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:38.177 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:38.177 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:38.177 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:38.178 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:38.178 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:38.178 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:38.178 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.178 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:18:38.178 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:18:38.178 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:18:38.178 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:18:38.178 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:18:38.179 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:18:38.179 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:38.179 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:38.179 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:18:38.179 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:18:38.179 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:18:38.179 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:18:38.179 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:18:38.179 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:18:38.179 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:18:38.179 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:38.179 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:18:38.179 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:38.180 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:38.180 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:38.180 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:38.180 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:38.180 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:38.180 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:38.180 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:38.180 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:38.180 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:38.180 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:38.181 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:38.181 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:18:38.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.181 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:18:38.181 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.181 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:18:38.181 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:18:38.181 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:38.181 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:38.181 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.181 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:38.181 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:18:38.181 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.182 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.182 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.182 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:38.182 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:18:38.182 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.182 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.182 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.182 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.182 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:18:38.182 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:18:38.182 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:38.182 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:38.182 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:38.182 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.183 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.183 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:18:38.183 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:18:38.183 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:38.183 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:38.183 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.183 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:38.183 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:18:38.183 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.183 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.183 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.183 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:38.183 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:18:38.184 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.184 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.184 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.184 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:18:38.184 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:18:38.184 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:38.184 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:38.184 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:38.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.184 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.184 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:18:38.184 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:18:38.184 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:38.185 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:38.185 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.185 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:38.185 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:18:38.185 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.185 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.185 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:38.185 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:18:38.185 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.185 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.185 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.186 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:18:38.186 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:18:38.186 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:38.186 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:38.186 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:38.186 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.186 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.186 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.186 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:18:38.186 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:18:38.186 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.186 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:38.187 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:38.187 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:18:38.187 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.187 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:38.187 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:18:38.187 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.188 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:38.188 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:38.188 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:18:38.188 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.188 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:38.188 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:18:38.188 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.188 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:38.188 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:38.188 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:18:38.188 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.188 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.189 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:18:38.189 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:18:38.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac52816640 10 librados: watch_flush enter 2024-09-16T05:18:38.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac52816640 10 librados: watch_flush exit 2024-09-16T05:18:38.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac52816640 20 client.74442.objecter shutdown clearing up homeless session... 2024-09-16T05:18:38.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.153+0000 7fac52816640 10 client.74442.objecter successfully canceled tick 2024-09-16T05:18:38.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.157+0000 7fac52816640 1 -- 172.21.15.114:0/3824175201 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac38037810 msgr2=0x7fac38039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.157+0000 7fac52816640 1 --2- 172.21.15.114:0/3824175201 >> v2:172.21.15.149:6812/1185531135 conn(0x7fac38037810 0x7fac38039cd0 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 crypto rx=0x7fac40007fd0 tx=0x7fac40007dc0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.157+0000 7fac52816640 10 monclient: shutdown 2024-09-16T05:18:38.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.157+0000 7fac52816640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:38.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.157+0000 7fac52816640 1 -- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c072960 msgr2=0x7fac4c12db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.157+0000 7fac52816640 1 --2- 172.21.15.114:0/3824175201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4c072960 0x7fac4c12db90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00c480 tx=0x7fac3c00c940 comp rx=0 tx=0).stop 2024-09-16T05:18:38.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.157+0000 7fac50db3640 1 -- 172.21.15.114:0/3824175201 reap_dead start 2024-09-16T05:18:38.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.157+0000 7fac50db3640 1 -- 172.21.15.114:0/3824175201 reap_dead start 2024-09-16T05:18:38.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.157+0000 7fac52816640 1 -- 172.21.15.114:0/3824175201 shutdown_connections 2024-09-16T05:18:38.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.157+0000 7fac52816640 1 -- 172.21.15.114:0/3824175201 >> 172.21.15.114:0/3824175201 conn(0x7fac4c0a6310 msgr2=0x7fac4c130f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:38.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.157+0000 7fac52816640 1 -- 172.21.15.114:0/3824175201 shutdown_connections 2024-09-16T05:18:38.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.157+0000 7fac52816640 1 -- 172.21.15.114:0/3824175201 wait complete. 2024-09-16T05:18:38.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.157+0000 7fac52816640 1 librados: shutdown 2024-09-16T05:18:38.192 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:18:38.192 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:18:38.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:18:38.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 10 monclient: get_monmap_and_config 2024-09-16T05:18:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 10 monclient: build_initial_monmap 2024-09-16T05:18:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 10 monclient: monmap: 2024-09-16T05:18:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:38.304921+0000 2024-09-16T05:18:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:38.304921+0000 2024-09-16T05:18:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:38.301 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:38.301 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:38.301 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 1 Processor -- start 2024-09-16T05:18:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 1 -- start start 2024-09-16T05:18:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 10 monclient: init 2024-09-16T05:18:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-16T05:18:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3754137730 addr v1:172.21.15.149:6789/0 2024-09-16T05:18:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37540a5d00 addr v1:172.21.15.114:6789/0 2024-09-16T05:18:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f375413dbe0 0x7f375413ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 10 monclient(hunting): picked mon.noname-a con 0x7f375413dbe0 addr v2:172.21.15.114:3300/0 2024-09-16T05:18:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37540a55e0 con 0x7f37540a5d00 2024-09-16T05:18:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37540a3710 con 0x7f3754137730 2024-09-16T05:18:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f375409c450 con 0x7f375413dbe0 2024-09-16T05:18:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f375a405640 10 monclient(hunting): authenticate will time out at 1602.934019s 2024-09-16T05:18:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f375413dbe0 0x7f375413ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37537fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f37540a5d00 0x7f375413c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:41212/0 (socket says 172.21.15.114:41212) 2024-09-16T05:18:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37537fe640 1 -- 172.21.15.114:0/1009947846 learned_addr learned my addr 172.21.15.114:0/1009947846 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:18:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 10 monclient(hunting): get_auth_request con 0x7f375413dbe0 auth_method 0 2024-09-16T05:18:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f3752ffd640 1 -- 172.21.15.114:0/1009947846 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2479658059 0 0) 0x7f37540a55e0 con 0x7f37540a5d00 2024-09-16T05:18:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f3752ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:18:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f3752ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f3752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f3752ffd640 10 monclient(hunting): my global_id is 74454 2024-09-16T05:18:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f3752ffd640 1 -- 172.21.15.114:0/1009947846 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3740003530 con 0x7f37540a5d00 2024-09-16T05:18:38.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 10 monclient(hunting): handle_auth_done global_id 74451 payload 274 2024-09-16T05:18:38.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 1 -- 172.21.15.114:0/1009947846 >> v1:172.21.15.149:6789/0 conn(0x7f3754137730 legacy=0x7f3754137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:38.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 1 -- 172.21.15.114:0/1009947846 >> v1:172.21.15.114:6789/0 conn(0x7f37540a5d00 legacy=0x7f375413c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 10 monclient: _finish_hunting 0 2024-09-16T05:18:38.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 1 monclient: found mon.noname-a 2024-09-16T05:18:38.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:38.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:38.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.301+0000 7f37589a2640 1 -- 172.21.15.114:0/1009947846 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f375409be00 con 0x7f375413dbe0 2024-09-16T05:18:38.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f37589a2640 1 --2- 172.21.15.114:0/1009947846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f375413dbe0 0x7f375413ffd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f374c009d90 tx=0x7f374c01c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a03aac2a595cb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3752ffd640 1 -- 172.21.15.114:0/1009947846 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c0b2070 con 0x7f375413dbe0 2024-09-16T05:18:38.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3752ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3752ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:38.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3752ffd640 10 monclient: dump: 2024-09-16T05:18:38.313 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.313 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.313 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.313 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.313 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.313 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.314 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.314 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.314 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.314 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3752ffd640 10 monclient: _finish_auth 0 2024-09-16T05:18:38.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3752ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:18:38.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3752ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:38.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3752ffd640 1 -- 172.21.15.114:0/1009947846 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f374c0a57c0 con 0x7f375413dbe0 2024-09-16T05:18:38.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3752ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:38.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 5 monclient: authenticate success, global_id 74451 2024-09-16T05:18:38.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:38.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 4 set_mon_vals no callback set 2024-09-16T05:18:38.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3752ffd640 1 -- 172.21.15.114:0/1009947846 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c0a5b70 con 0x7f375413dbe0 2024-09-16T05:18:38.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3752ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3752ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:38.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3752ffd640 10 monclient: dump: 2024-09-16T05:18:38.315 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.315 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.315 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.315 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.315 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.315 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.316 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.316 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.316 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.316 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 10 monclient: shutdown 2024-09-16T05:18:38.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:38.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 -- 172.21.15.114:0/1009947846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f375413dbe0 msgr2=0x7f375413ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 --2- 172.21.15.114:0/1009947846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f375413dbe0 0x7f375413ffd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f374c009d90 tx=0x7f374c01c7f0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 -- 172.21.15.114:0/1009947846 shutdown_connections 2024-09-16T05:18:38.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 --2- 172.21.15.114:0/1009947846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f375413dbe0 0x7f375413ffd0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 -- 172.21.15.114:0/1009947846 >> 172.21.15.114:0/1009947846 conn(0x7f37540a9c70 msgr2=0x7f37540aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:38.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 -- 172.21.15.114:0/1009947846 shutdown_connections 2024-09-16T05:18:38.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 -- 172.21.15.114:0/1009947846 wait complete. 2024-09-16T05:18:38.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 10 monclient: build_initial_monmap 2024-09-16T05:18:38.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 10 monclient: monmap: 2024-09-16T05:18:38.317 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:38.317 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:38.317 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:38.317 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:38.317 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:38.317 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.317 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:38.317 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:38.317 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:38.317 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 librados: starting msgr at 2024-09-16T05:18:38.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 librados: starting objecter 2024-09-16T05:18:38.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 Processor -- start 2024-09-16T05:18:38.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 -- start start 2024-09-16T05:18:38.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 librados: setting wanted keys 2024-09-16T05:18:38.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 librados: calling monclient init 2024-09-16T05:18:38.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 10 monclient: init 2024-09-16T05:18:38.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:38.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:18:38.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37540a5d00 0x7f37541cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 10 monclient(hunting): picked mon.noname-c con 0x7f37540a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:38.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3754137730 0x7f37541cf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 10 monclient(hunting): picked mon.noname-a con 0x7f3754137730 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:38.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f375413dbe0 0x7f375409e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 10 monclient(hunting): picked mon.noname-b con 0x7f375413dbe0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3754142eb0 con 0x7f3754137730 2024-09-16T05:18:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f3754142d30 con 0x7f375413dbe0 2024-09-16T05:18:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f3754143030 con 0x7f37540a5d00 2024-09-16T05:18:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f375a405640 10 monclient(hunting): authenticate will time out at 1602.939719s 2024-09-16T05:18:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f37537fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3754137730 0x7f37541cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3753fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37540a5d00 0x7f37541cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f37537fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3754137730 0x7f37541cf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34036/0 (socket says 172.21.15.114:34036) 2024-09-16T05:18:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f37537fe640 1 -- 172.21.15.114:0/826887717 learned_addr learned my addr 172.21.15.114:0/826887717 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:38.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f37537fe640 10 monclient(hunting): get_auth_request con 0x7f3754137730 auth_method 0 2024-09-16T05:18:38.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f37537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f37537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f37537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3753fff640 10 monclient(hunting): get_auth_request con 0x7f37540a5d00 auth_method 0 2024-09-16T05:18:38.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3753fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3753fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3753fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f37537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f37537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f37537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3753fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3753fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.305+0000 7f3753fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37589a2640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f375413dbe0 0x7f375409e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3753fff640 10 monclient(hunting): handle_auth_done global_id 94139 payload 824 2024-09-16T05:18:38.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3753fff640 1 -- 172.21.15.114:0/826887717 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f375413dbe0 msgr2=0x7f375409e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3753fff640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f375413dbe0 0x7f375409e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3753fff640 1 -- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3754137730 msgr2=0x7f37541cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3753fff640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3754137730 0x7f37541cf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3753fff640 10 monclient: _finish_hunting 0 2024-09-16T05:18:38.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3753fff640 1 monclient: found mon.noname-c 2024-09-16T05:18:38.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3753fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:38.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3753fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:18:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3753fff640 1 -- 172.21.15.114:0/826887717 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37540a2050 con 0x7f37540a5d00 2024-09-16T05:18:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37537fe640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3754137730 0x7f37541cf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:18:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3753fff640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37540a5d00 0x7f37541cf400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f373c00c3e0 tx=0x7f373c00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 client.?.objecter ms_handle_connect 0x7f37540a5d00 2024-09-16T05:18:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 1 -- 172.21.15.114:0/826887717 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c011790 con 0x7f37540a5d00 2024-09-16T05:18:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:18:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 monclient: dump: 2024-09-16T05:18:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.323 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.323 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.323 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.323 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.323 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.323 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.323 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.323 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.323 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 monclient: _finish_auth 0 2024-09-16T05:18:38.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:18:38.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:38.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 1 -- 172.21.15.114:0/826887717 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f373c017070 con 0x7f37540a5d00 2024-09-16T05:18:38.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f375a405640 5 monclient: authenticate success, global_id 94139 2024-09-16T05:18:38.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f375a405640 10 monclient: _renew_subs 2024-09-16T05:18:38.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f375a405640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:18:38.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f375a405640 1 -- 172.21.15.114:0/826887717 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37540a0c00 con 0x7f37540a5d00 2024-09-16T05:18:38.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:38.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 1 -- 172.21.15.114:0/826887717 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c011bb0 con 0x7f37540a5d00 2024-09-16T05:18:38.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:18:38.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 monclient: dump: 2024-09-16T05:18:38.324 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.324 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.324 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.324 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.324 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.324 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.325 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.325 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.325 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.325 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37527fc640 4 set_mon_vals no callback set 2024-09-16T05:18:38.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f375a405640 10 client.94139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:38.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f375a405640 10 monclient: _renew_subs 2024-09-16T05:18:38.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f375a405640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:18:38.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f375a405640 1 -- 172.21.15.114:0/826887717 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f37540a2330 con 0x7f37540a5d00 2024-09-16T05:18:38.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f375a405640 1 librados: init done 2024-09-16T05:18:38.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f375a405640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:38.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f375a405640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f375409ec10 0x7f375409eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f375a405640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f375a405640 1 -- 172.21.15.114:0/826887717 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f375413f8d0 con 0x7f375409ec10 2024-09-16T05:18:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 1 -- 172.21.15.114:0/826887717 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f373c01c460 con 0x7f37540a5d00 2024-09-16T05:18:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37537fe640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f375409ec10 0x7f375409eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37537fe640 10 monclient: get_auth_request con 0x7f375409ec10 auth_method 0 2024-09-16T05:18:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 1 --2- 172.21.15.114:0/826887717 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3734037810 0x7f3734039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 1 -- 172.21.15.114:0/826887717 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(432..432 src has 1..432) ==== 5462+0+0 (secure 0 0 0) 0x7f373c048c60 con 0x7f37540a5d00 2024-09-16T05:18:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 client.94139.objecter ms_dispatch 0x7f3754000ba0 osd_map(432..432 src has 1..432) 2024-09-16T05:18:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 3 client.94139.objecter handle_osd_map got epochs [432,432] > 0 2024-09-16T05:18:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 3 client.94139.objecter handle_osd_map decoding full epoch 432 2024-09-16T05:18:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37537fe640 10 monclient(hunting): handle_auth_done global_id 94142 payload 824 2024-09-16T05:18:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37537fe640 1 -- 172.21.15.114:0/826887717 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3754137b10 con 0x7f375409ec10 2024-09-16T05:18:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 20 client.94139.objecter dump_active .. 0 homeless 2024-09-16T05:18:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 client.94139.objecter ms_handle_connect 0x7f375409ec10 2024-09-16T05:18:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37537fe640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f375409ec10 0x7f375409eff0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f375409ff20 tx=0x7f3744001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 client.94139.objecter resend_mon_ops 2024-09-16T05:18:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37589a2640 1 --2- 172.21.15.114:0/826887717 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3734037810 0x7f3734039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37589a2640 10 monclient: get_auth_request con 0x7f3734037810 auth_method 0 2024-09-16T05:18:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 1 -- 172.21.15.114:0/826887717 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3744005ad0 con 0x7f375409ec10 2024-09-16T05:18:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:18:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 monclient: dump: 2024-09-16T05:18:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.328 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 1 -- 172.21.15.114:0/826887717 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3744018070 con 0x7f375409ec10 2024-09-16T05:18:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 1 -- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f375409ec10 msgr2=0x7f375409eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f375409ec10 0x7f375409eff0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f375409ff20 tx=0x7f3744001e70 comp rx=0 tx=0).stop 2024-09-16T05:18:38.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 1 -- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f375409ec10 msgr2=0x7f375409eff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:38.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f375409ec10 0x7f375409eff0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f3750ff9640 10 client.94139.objecter ms_handle_connect 0x7f3734037810 2024-09-16T05:18:38.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.309+0000 7f37589a2640 1 --2- 172.21.15.114:0/826887717 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3734037810 0x7f3734039cd0 secure :-1 s=READY pgs=3545 cs=0 l=1 rev1=1 crypto rx=0x7f374c0af000 tx=0x7f374c0a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:18:38.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f375409f530 0x7f375413ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 1 -- 172.21.15.114:0/826887717 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f3754068650 con 0x7f375409f530 2024-09-16T05:18:38.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f37537fe640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f375409f530 0x7f375413ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f37537fe640 10 monclient: get_auth_request con 0x7f375409f530 auth_method 0 2024-09-16T05:18:38.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f37537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f37537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f37537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f37537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f37537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f37537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f37537fe640 10 monclient(hunting): handle_auth_done global_id 94145 payload 824 2024-09-16T05:18:38.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f37537fe640 1 -- 172.21.15.114:0/826887717 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f37540a1d70 con 0x7f375409f530 2024-09-16T05:18:38.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f3750ff9640 10 client.94139.objecter ms_handle_connect 0x7f375409f530 2024-09-16T05:18:38.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f3750ff9640 10 client.94139.objecter resend_mon_ops 2024-09-16T05:18:38.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f37537fe640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f375409f530 0x7f375413ad40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f375413fa50 tx=0x7f374401ea00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f3750ff9640 1 -- 172.21.15.114:0/826887717 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374401ed20 con 0x7f375409f530 2024-09-16T05:18:38.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f3750ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f3750ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:18:38.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f3750ff9640 10 monclient: dump: 2024-09-16T05:18:38.332 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.332 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.332 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.332 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.333 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.333 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.333 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.333 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.333 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.333 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f3750ff9640 1 -- 172.21.15.114:0/826887717 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f37440070d0 con 0x7f375409f530 2024-09-16T05:18:38.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f3750ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:18:38.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f3750ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:38.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f3750ff9640 1 -- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f375409f530 msgr2=0x7f375413ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f3750ff9640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f375409f530 0x7f375413ad40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f375413fa50 tx=0x7f374401ea00 comp rx=0 tx=0).stop 2024-09-16T05:18:38.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f3750ff9640 1 -- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f375409f530 msgr2=0x7f375413ad40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:38.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f3750ff9640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f375409f530 0x7f375413ad40 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f375413fa50 tx=0x7f374401ea00 comp rx=0 tx=0).stop 2024-09-16T05:18:38.334 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:18:38.334 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:18:38.334 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:18:38.334 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:18:38.334 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 50, 2024-09-16T05:18:38.334 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 19867, 2024-09-16T05:18:38.334 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:18:38.334 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:18:38.334 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:18:38.334 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:18:38.335 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:38.335 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 5, 2024-09-16T05:18:38.335 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:38.335 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:18:38.335 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:18:38.335 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:38.335 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:38.335 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:38.335 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:38.335 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:38.335 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:38.335 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:38.335 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:38.335 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:38.336 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:38.336 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:38.336 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:38.336 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:18:38.336 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:18:38.336 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:38.336 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:38.336 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:38.336 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:38.336 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:38.336 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:38.336 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:38.336 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:38.336 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:38.337 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:38.337 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:38.337 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.337 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:18:38.337 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:18:38.337 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:18:38.337 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:18:38.337 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:18:38.337 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:18:38.337 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:38.337 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:38.337 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:18:38.338 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:18:38.338 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:18:38.338 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:18:38.338 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:18:38.338 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:18:38.338 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:18:38.338 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:38.338 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:18:38.338 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:38.338 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:38.338 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:38.338 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:38.338 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:38.338 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:18:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:18:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:18:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:18:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:38.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.340 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:38.340 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:18:38.340 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.340 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:38.340 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:18:38.340 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.340 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.340 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.341 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:18:38.341 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:18:38.341 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:38.341 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:38.341 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:38.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.341 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:18:38.341 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:18:38.341 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:38.341 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:38.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.342 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:38.342 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:18:38.342 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.342 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:38.342 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:18:38.342 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.342 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.342 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.342 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:18:38.342 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:18:38.343 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:38.343 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:38.343 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:38.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.343 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:18:38.343 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:18:38.343 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:38.343 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:38.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.343 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:38.343 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:18:38.343 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.344 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:38.344 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:18:38.344 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.344 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.344 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.344 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:18:38.344 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:18:38.344 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:38.344 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:38.344 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:38.345 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.345 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.345 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:18:38.345 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:18:38.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.345 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:38.345 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:38.345 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:18:38.345 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.345 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:38.346 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:18:38.346 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.346 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:38.346 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:38.346 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:18:38.346 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.346 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.346 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:18:38.346 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:18:38.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 10 librados: watch_flush enter 2024-09-16T05:18:38.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 10 librados: watch_flush exit 2024-09-16T05:18:38.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 20 client.94139.objecter shutdown clearing up homeless session... 2024-09-16T05:18:38.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 10 client.94139.objecter successfully canceled tick 2024-09-16T05:18:38.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 1 -- 172.21.15.114:0/826887717 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3734037810 msgr2=0x7f3734039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 1 --2- 172.21.15.114:0/826887717 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3734037810 0x7f3734039cd0 secure :-1 s=READY pgs=3545 cs=0 l=1 rev1=1 crypto rx=0x7f374c0af000 tx=0x7f374c0a7070 comp rx=0 tx=0).stop 2024-09-16T05:18:38.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 10 monclient: shutdown 2024-09-16T05:18:38.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:38.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 1 -- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37540a5d00 msgr2=0x7f37541cf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 1 --2- 172.21.15.114:0/826887717 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37540a5d00 0x7f37541cf400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f373c00c3e0 tx=0x7f373c00c8a0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f37589a2640 1 -- 172.21.15.114:0/826887717 reap_dead start 2024-09-16T05:18:38.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f37589a2640 1 -- 172.21.15.114:0/826887717 reap_dead start 2024-09-16T05:18:38.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 1 -- 172.21.15.114:0/826887717 shutdown_connections 2024-09-16T05:18:38.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 1 -- 172.21.15.114:0/826887717 >> 172.21.15.114:0/826887717 conn(0x7f37540a9c70 msgr2=0x7f37540a7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:38.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 1 -- 172.21.15.114:0/826887717 shutdown_connections 2024-09-16T05:18:38.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 1 -- 172.21.15.114:0/826887717 wait complete. 2024-09-16T05:18:38.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.317+0000 7f375a405640 1 librados: shutdown 2024-09-16T05:18:38.350 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:18:38.350 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:18:38.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:18:38.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.461+0000 7fc5ec1e3640 10 monclient: get_monmap_and_config 2024-09-16T05:18:38.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.461+0000 7fc5ec1e3640 10 monclient: build_initial_monmap 2024-09-16T05:18:38.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.461+0000 7fc5ec1e3640 10 monclient: monmap: 2024-09-16T05:18:38.463 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:38.463 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.463 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:38.468231+0000 2024-09-16T05:18:38.463 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:38.468231+0000 2024-09-16T05:18:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:38.464 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:38.464 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 1 Processor -- start 2024-09-16T05:18:38.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 1 -- start start 2024-09-16T05:18:38.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 10 monclient: init 2024-09-16T05:18:38.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:38.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-16T05:18:38.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc5e41376e0 0x7fc5e4137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 10 monclient(hunting): picked mon.noname-a con 0x7fc5e41376e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:18:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5e41370a0 0x7fc5e40a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 10 monclient(hunting): picked mon.noname-b con 0x7fc5e41370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5e40a5d00 addr v1:172.21.15.149:6789/0 2024-09-16T05:18:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5e409bd20 con 0x7fc5e40a5d00 2024-09-16T05:18:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fc5e4138000 con 0x7fc5e41376e0 2024-09-16T05:18:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc5e409c5c0 con 0x7fc5e41370a0 2024-09-16T05:18:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:38.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5ec1e3640 10 monclient(hunting): authenticate will time out at 1603.097450s 2024-09-16T05:18:38.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e977e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5e41370a0 0x7fc5e40a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc5e41376e0 0x7fc5e4137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e977e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5e41370a0 0x7fc5e40a4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:46620/0 (socket says 172.21.15.114:46620) 2024-09-16T05:18:38.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e977e640 1 -- 172.21.15.114:0/896782615 learned_addr learned my addr 172.21.15.114:0/896782615 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:38.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e977e640 10 monclient(hunting): get_auth_request con 0x7fc5e41370a0 auth_method 0 2024-09-16T05:18:38.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e977e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e977e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e977e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 10 monclient(hunting): get_auth_request con 0x7fc5e41376e0 auth_method 0 2024-09-16T05:18:38.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e977e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e977e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e977e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 10 monclient(hunting): handle_auth_done global_id 74460 payload 274 2024-09-16T05:18:38.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 1 -- 172.21.15.114:0/896782615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5e41370a0 msgr2=0x7fc5e40a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:38.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 1 --2- 172.21.15.114:0/896782615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5e41370a0 0x7fc5e40a4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 1 -- 172.21.15.114:0/896782615 >> v1:172.21.15.149:6789/0 conn(0x7fc5e40a5d00 legacy=0x7fc5e40a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:38.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 10 monclient: _finish_hunting 0 2024-09-16T05:18:38.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 1 monclient: found mon.noname-a 2024-09-16T05:18:38.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:38.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:38.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e9f7f640 1 -- 172.21.15.114:0/896782615 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e409c000 con 0x7fc5e41376e0 2024-09-16T05:18:38.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.465+0000 7fc5e977e640 1 --2- 172.21.15.114:0/896782615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5e41370a0 0x7fc5e40a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:18:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 1 --2- 172.21.15.114:0/896782615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e41376e0 0x7fc5e4137ac0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40097c0 tx=0x7fc5d401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb50f3b6921fa395 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e8f7d640 1 -- 172.21.15.114:0/896782615 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d40b0070 con 0x7fc5e41376e0 2024-09-16T05:18:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e8f7d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e8f7d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e8f7d640 10 monclient: dump: 2024-09-16T05:18:38.477 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.477 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.477 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.477 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.477 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.477 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.477 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.477 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.477 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.477 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e8f7d640 10 monclient: _finish_auth 0 2024-09-16T05:18:38.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e8f7d640 10 monclient: _check_auth_tickets 2024-09-16T05:18:38.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e8f7d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:38.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e8f7d640 1 -- 172.21.15.114:0/896782615 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d40ab600 con 0x7fc5e41376e0 2024-09-16T05:18:38.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e8f7d640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:38.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e8f7d640 1 -- 172.21.15.114:0/896782615 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d40ab9d0 con 0x7fc5e41376e0 2024-09-16T05:18:38.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 5 monclient: authenticate success, global_id 74460 2024-09-16T05:18:38.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:38.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 4 set_mon_vals no callback set 2024-09-16T05:18:38.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e8f7d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e8f7d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:38.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e8f7d640 10 monclient: dump: 2024-09-16T05:18:38.478 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.478 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.478 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.478 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.478 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.479 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.479 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.479 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.479 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.479 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient: shutdown 2024-09-16T05:18:38.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:38.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 -- 172.21.15.114:0/896782615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e41376e0 msgr2=0x7fc5e4137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 --2- 172.21.15.114:0/896782615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e41376e0 0x7fc5e4137ac0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40097c0 tx=0x7fc5d401c320 comp rx=0 tx=0).stop 2024-09-16T05:18:38.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 -- 172.21.15.114:0/896782615 shutdown_connections 2024-09-16T05:18:38.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 --2- 172.21.15.114:0/896782615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5e41370a0 0x7fc5e40a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 --2- 172.21.15.114:0/896782615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e41376e0 0x7fc5e4137ac0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 -- 172.21.15.114:0/896782615 >> 172.21.15.114:0/896782615 conn(0x7fc5e40a9a40 msgr2=0x7fc5e40a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:38.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 -- 172.21.15.114:0/896782615 shutdown_connections 2024-09-16T05:18:38.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 -- 172.21.15.114:0/896782615 wait complete. 2024-09-16T05:18:38.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient: build_initial_monmap 2024-09-16T05:18:38.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient: monmap: 2024-09-16T05:18:38.480 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:38.480 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:38.480 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:38.480 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:38.480 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:38.480 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.480 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:38.480 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:38.480 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:38.480 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 librados: starting msgr at 2024-09-16T05:18:38.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 librados: starting objecter 2024-09-16T05:18:38.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 Processor -- start 2024-09-16T05:18:38.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 -- start start 2024-09-16T05:18:38.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 librados: setting wanted keys 2024-09-16T05:18:38.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 librados: calling monclient init 2024-09-16T05:18:38.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient: init 2024-09-16T05:18:38.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:38.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:18:38.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e40a5d00 0x7fc5e41cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient(hunting): picked mon.noname-a con 0x7fc5e40a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:38.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e41370a0 0x7fc5e4136170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient(hunting): picked mon.noname-b con 0x7fc5e41370a0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:38.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5e41376e0 0x7fc5e413c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient(hunting): picked mon.noname-c con 0x7fc5e41376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:38.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e4141d80 con 0x7fc5e40a5d00 2024-09-16T05:18:38.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e4141c00 con 0x7fc5e41370a0 2024-09-16T05:18:38.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc5e4141f00 con 0x7fc5e41376e0 2024-09-16T05:18:38.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:38.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient(hunting): authenticate will time out at 1603.102377s 2024-09-16T05:18:38.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e40a5d00 0x7fc5e41cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e40a5d00 0x7fc5e41cf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34046/0 (socket says 172.21.15.114:34046) 2024-09-16T05:18:38.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ea780640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5e41376e0 0x7fc5e413c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 1 -- 172.21.15.114:0/2126812224 learned_addr learned my addr 172.21.15.114:0/2126812224 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:38.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 10 monclient(hunting): get_auth_request con 0x7fc5e40a5d00 auth_method 0 2024-09-16T05:18:38.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ea780640 10 monclient(hunting): get_auth_request con 0x7fc5e41376e0 auth_method 0 2024-09-16T05:18:38.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ea780640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ea780640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ea780640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ea780640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ea780640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ea780640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 10 monclient(hunting): handle_auth_done global_id 74463 payload 824 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e977e640 1 --2- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e41370a0 0x7fc5e4136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 1 -- 172.21.15.114:0/2126812224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5e41376e0 msgr2=0x7fc5e413c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 1 --2- 172.21.15.114:0/2126812224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5e41376e0 0x7fc5e413c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 1 -- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e41370a0 msgr2=0x7fc5e4136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 1 --2- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e41370a0 0x7fc5e4136170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 10 monclient: _finish_hunting 0 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 1 monclient: found mon.noname-a 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 1 -- 172.21.15.114:0/2126812224 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e4120b00 con 0x7fc5e40a5d00 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5e9f7f640 1 --2- 172.21.15.114:0/2126812224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e40a5d00 0x7fc5e41cf0c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc5d401a9d0 tx=0x7fc5d40190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5da7fc640 10 client.?.objecter ms_handle_connect 0x7fc5e40a5d00 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5da7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5da7fc640 1 -- 172.21.15.114:0/2126812224 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d40b8020 con 0x7fc5e40a5d00 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5da7fc640 10 monclient: dump: 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5da7fc640 10 monclient: _finish_auth 0 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5da7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 5 monclient: authenticate success, global_id 74463 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5da7fc640 1 -- 172.21.15.114:0/2126812224 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d40b4410 con 0x7fc5e40a5d00 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient: _renew_subs 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 -- 172.21.15.114:0/2126812224 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5e40a2530 con 0x7fc5e40a5d00 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5da7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5da7fc640 1 -- 172.21.15.114:0/2126812224 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d40b0040 con 0x7fc5e40a5d00 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5da7fc640 10 monclient: dump: 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5dbfff640 4 set_mon_vals no callback set 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 client.74463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient: _renew_subs 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 -- 172.21.15.114:0/2126812224 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5e411fec0 con 0x7fc5e40a5d00 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 librados: init done 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 --2- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e41366b0 0x7fc5e413c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.469+0000 7fc5ec1e3640 1 -- 172.21.15.114:0/2126812224 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e4137ac0 con 0x7fc5e41366b0 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 1 -- 172.21.15.114:0/2126812224 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc5d40bb020 con 0x7fc5e40a5d00 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5e977e640 1 --2- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e41366b0 0x7fc5e413c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 1 --2- 172.21.15.114:0/2126812224 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc5c8037810 0x7fc5c8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5e977e640 10 monclient: get_auth_request con 0x7fc5e41366b0 auth_method 0 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5e977e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5e977e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5e977e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 1 -- 172.21.15.114:0/2126812224 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(432..432 src has 1..432) ==== 5462+0+0 (secure 0 0 0) 0x7fc5d40e2970 con 0x7fc5e40a5d00 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 10 client.74463.objecter ms_dispatch 0x7fc5e4000ba0 osd_map(432..432 src has 1..432) 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 3 client.74463.objecter handle_osd_map got epochs [432,432] > 0 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 3 client.74463.objecter handle_osd_map decoding full epoch 432 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5e977e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5e977e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5e977e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 20 client.74463.objecter dump_active .. 0 homeless 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5ea780640 1 --2- 172.21.15.114:0/2126812224 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc5c8037810 0x7fc5c8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5ea780640 10 monclient: get_auth_request con 0x7fc5c8037810 auth_method 0 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5e977e640 10 monclient(hunting): handle_auth_done global_id 64540 payload 824 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5e977e640 1 -- 172.21.15.114:0/2126812224 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5e413e920 con 0x7fc5e41366b0 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 10 client.74463.objecter ms_handle_connect 0x7fc5e41366b0 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 10 client.74463.objecter resend_mon_ops 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5e977e640 1 --2- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e41366b0 0x7fc5e413c5d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc5e413d930 tx=0x7fc5e0010b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 1 -- 172.21.15.114:0/2126812224 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e000f040 con 0x7fc5e41366b0 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 10 monclient: dump: 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 10 client.74463.objecter ms_handle_connect 0x7fc5c8037810 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5ea780640 1 --2- 172.21.15.114:0/2126812224 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc5c8037810 0x7fc5c8039cd0 secure :-1 s=READY pgs=3546 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc005600 tx=0x7fc5dc003d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 1 -- 172.21.15.114:0/2126812224 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc5e001c400 con 0x7fc5e41366b0 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 1 -- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e41366b0 msgr2=0x7fc5e413c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 1 --2- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e41366b0 0x7fc5e413c5d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc5e413d930 tx=0x7fc5e0010b60 comp rx=0 tx=0).stop 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 1 -- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e41366b0 msgr2=0x7fc5e413c5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.473+0000 7fc5da7fc640 1 --2- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e41366b0 0x7fc5e413c5d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 1 --2- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e413cb10 0x7fc5e4138190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 1 -- 172.21.15.114:0/2126812224 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e413cf20 con 0x7fc5e413cb10 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5e977e640 1 --2- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e413cb10 0x7fc5e4138190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5e977e640 10 monclient: get_auth_request con 0x7fc5e413cb10 auth_method 0 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5e977e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5e977e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5e977e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5e977e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5e977e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5e977e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5e977e640 10 monclient(hunting): handle_auth_done global_id 64543 payload 824 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5e977e640 1 -- 172.21.15.114:0/2126812224 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc5e41cff40 con 0x7fc5e413cb10 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5e977e640 1 --2- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e413cb10 0x7fc5e4138190 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc5e001b000 tx=0x7fc5e00046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5da7fc640 10 client.74463.objecter ms_handle_connect 0x7fc5e413cb10 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5da7fc640 10 client.74463.objecter resend_mon_ops 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5da7fc640 1 -- 172.21.15.114:0/2126812224 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0021770 con 0x7fc5e413cb10 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5da7fc640 10 monclient: dump: 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5da7fc640 1 -- 172.21.15.114:0/2126812224 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fc5e0025070 con 0x7fc5e413cb10 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5da7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5da7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5da7fc640 1 -- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e413cb10 msgr2=0x7fc5e4138190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5da7fc640 1 --2- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e413cb10 0x7fc5e4138190 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc5e001b000 tx=0x7fc5e00046d0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5da7fc640 1 -- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e413cb10 msgr2=0x7fc5e4138190 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5da7fc640 1 --2- 172.21.15.114:0/2126812224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5e413cb10 0x7fc5e4138190 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 50, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 402618, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 5, 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:18:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:18:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:18:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:18:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:18:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:18:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:18:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:18:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 10 librados: watch_flush enter 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 10 librados: watch_flush exit 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 20 client.74463.objecter shutdown clearing up homeless session... 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 10 client.74463.objecter successfully canceled tick 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 1 -- 172.21.15.114:0/2126812224 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc5c8037810 msgr2=0x7fc5c8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 1 --2- 172.21.15.114:0/2126812224 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc5c8037810 0x7fc5c8039cd0 secure :-1 s=READY pgs=3546 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc005600 tx=0x7fc5dc003d50 comp rx=0 tx=0).stop 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 10 monclient: shutdown 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 1 -- 172.21.15.114:0/2126812224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e40a5d00 msgr2=0x7fc5e41cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 1 --2- 172.21.15.114:0/2126812224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e40a5d00 0x7fc5e41cf0c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc5d401a9d0 tx=0x7fc5d40190c0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ea780640 1 -- 172.21.15.114:0/2126812224 reap_dead start 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ea780640 1 -- 172.21.15.114:0/2126812224 reap_dead start 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 1 -- 172.21.15.114:0/2126812224 shutdown_connections 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 1 -- 172.21.15.114:0/2126812224 >> 172.21.15.114:0/2126812224 conn(0x7fc5e40a9a40 msgr2=0x7fc5e40a7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 1 -- 172.21.15.114:0/2126812224 shutdown_connections 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 1 -- 172.21.15.114:0/2126812224 wait complete. 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.481+0000 7fc5ec1e3640 1 librados: shutdown 2024-09-16T05:18:38.514 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:18:38.514 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T05:18:38.515 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T05:18:38.515 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T05:18:38.515 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T05:18:38.515 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T05:18:38.515 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:18:38.524 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T05:18:38.524 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:18:38.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:18:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 10 monclient: get_monmap_and_config 2024-09-16T05:18:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 10 monclient: build_initial_monmap 2024-09-16T05:18:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 10 monclient: monmap: 2024-09-16T05:18:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:38.682347+0000 2024-09-16T05:18:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:38.682347+0000 2024-09-16T05:18:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:38.678 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.678 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:38.678 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:38.678 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:38.678 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:38.678 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:38.678 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 1 Processor -- start 2024-09-16T05:18:38.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 1 -- start start 2024-09-16T05:18:38.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 10 monclient: init 2024-09-16T05:18:38.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:38.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-16T05:18:38.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb0f81376e0 addr v1:172.21.15.114:6789/0 2024-09-16T05:18:38.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb0f81370a0 0x7fb0f8137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 10 monclient(hunting): picked mon.noname-c con 0x7fb0f81370a0 addr v2:172.21.15.149:3300/0 2024-09-16T05:18:38.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f80a5d00 0x7fb0f80a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 10 monclient(hunting): picked mon.noname-b con 0x7fb0f80a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:38.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0f809bd20 con 0x7fb0f81376e0 2024-09-16T05:18:38.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fb0f8138000 con 0x7fb0f81370a0 2024-09-16T05:18:38.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb0f809c5c0 con 0x7fb0f80a5d00 2024-09-16T05:18:38.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:38.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.677+0000 7fb0fd8cc640 10 monclient(hunting): authenticate will time out at 1603.311994s 2024-09-16T05:18:38.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f80a5d00 msgr2=0x7fb0f80a4980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:18:38.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f80a5d00 0x7fb0f80a4980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:38.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f6ffd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb0f81376e0 0x7fb0f80a4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:41218/0 (socket says 172.21.15.114:41218) 2024-09-16T05:18:38.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f6ffd640 1 -- 172.21.15.114:0/2580636384 learned_addr learned my addr 172.21.15.114:0/2580636384 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:18:38.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 -- 172.21.15.114:0/2580636384 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1220158102 0 0) 0x7fb0f809bd20 con 0x7fb0f81376e0 2024-09-16T05:18:38.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:18:38.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient(hunting): my global_id is 74466 2024-09-16T05:18:38.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f67fc640 1 --2- 172.21.15.114:0/2580636384 >> v2:172.21.15.149:3300/0 conn(0x7fb0f81370a0 0x7fb0f8137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 -- 172.21.15.114:0/2580636384 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0e0003530 con 0x7fb0f81376e0 2024-09-16T05:18:38.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f67fc640 10 monclient(hunting): get_auth_request con 0x7fb0f81370a0 auth_method 0 2024-09-16T05:18:38.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 -- 172.21.15.114:0/2580636384 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3587986321 0 0) 0x7fb0e0003530 con 0x7fb0f81376e0 2024-09-16T05:18:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 -- 172.21.15.114:0/2580636384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f80a5d00 msgr2=0x7fb0f80a4980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:18:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 --2- 172.21.15.114:0/2580636384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f80a5d00 0x7fb0f80a4980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 -- 172.21.15.114:0/2580636384 >> v2:172.21.15.149:3300/0 conn(0x7fb0f81370a0 msgr2=0x7fb0f8137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 --2- 172.21.15.114:0/2580636384 >> v2:172.21.15.149:3300/0 conn(0x7fb0f81370a0 0x7fb0f8137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:18:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:18:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:18:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 -- 172.21.15.114:0/2580636384 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f809c000 con 0x7fb0f81376e0 2024-09-16T05:18:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:18:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:18:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 -- 172.21.15.114:0/2580636384 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7fb0e8001680 con 0x7fb0f81376e0 2024-09-16T05:18:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0fd8cc640 5 monclient: authenticate success, global_id 74466 2024-09-16T05:18:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0fd8cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:18:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: dump: 2024-09-16T05:18:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.696 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:18:38.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:38.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 -- 172.21.15.114:0/2580636384 >> v1:172.21.15.114:6789/0 conn(0x7fb0f81376e0 legacy=0x7fb0f80a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:18:38.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 --2- 172.21.15.114:0/2580636384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f80a5d00 0x7fb0e00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient(hunting): picked mon.c con 0x7fb0f80a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:38.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 --2- 172.21.15.114:0/2580636384 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb0e0003f60 0x7fb0e0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient(hunting): picked mon.b con 0x7fb0e0003f60 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:38.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 --2- 172.21.15.114:0/2580636384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e0004d50 0x7fb0e00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient(hunting): picked mon.a con 0x7fb0e0004d50 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:38.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 -- 172.21.15.114:0/2580636384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f809c5c0 con 0x7fb0e0004d50 2024-09-16T05:18:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 -- 172.21.15.114:0/2580636384 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e000bbc0 con 0x7fb0e0003f60 2024-09-16T05:18:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 -- 172.21.15.114:0/2580636384 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb0e000bdc0 con 0x7fb0f80a5d00 2024-09-16T05:18:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f6ffd640 1 -- 172.21.15.114:0/2580636384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f80a5d00 msgr2=0x7fb0e00039e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:18:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:18:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0fd8cc640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f6ffd640 1 --2- 172.21.15.114:0/2580636384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f80a5d00 0x7fb0e00039e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 1 --2- 172.21.15.114:0/2580636384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e0004d50 0x7fb0e00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 10 monclient(hunting): get_auth_request con 0x7fb0e0004d50 auth_method 0 2024-09-16T05:18:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:18:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f67fc640 1 --2- 172.21.15.114:0/2580636384 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb0e0003f60 0x7fb0e0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 10 monclient(hunting): handle_auth_done global_id 74466 payload 293 2024-09-16T05:18:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 1 -- 172.21.15.114:0/2580636384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f80a5d00 msgr2=0x7fb0e00039e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:18:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 1 --2- 172.21.15.114:0/2580636384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f80a5d00 0x7fb0e00039e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 1 -- 172.21.15.114:0/2580636384 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb0e0003f60 msgr2=0x7fb0e0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 1 --2- 172.21.15.114:0/2580636384 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb0e0003f60 0x7fb0e0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 10 monclient: _finish_hunting 0 2024-09-16T05:18:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 1 monclient: found mon.a 2024-09-16T05:18:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:38.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 1 -- 172.21.15.114:0/2580636384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e0003530 con 0x7fb0e0004d50 2024-09-16T05:18:38.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 10 monclient: _finish_auth 0 2024-09-16T05:18:38.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 10 monclient: _check_auth_tickets 2024-09-16T05:18:38.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:38.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f67fc640 10 monclient: get_auth_request con 0x7fb0e0003f60 auth_method 0 2024-09-16T05:18:38.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f67fc640 1 --2- 172.21.15.114:0/2580636384 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb0e0003f60 0x7fb0e0004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:18:38.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f77fe640 1 --2- 172.21.15.114:0/2580636384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e0004d50 0x7fb0e00051d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009e30 tx=0x7fb0f001caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8440696b3093522 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 -- 172.21.15.114:0/2580636384 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0009a30 con 0x7fb0e0004d50 2024-09-16T05:18:38.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:38.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: dump: 2024-09-16T05:18:38.699 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.700 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 -- 172.21.15.114:0/2580636384 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f00a9070 con 0x7fb0e0004d50 2024-09-16T05:18:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0fd8cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 1 -- 172.21.15.114:0/2580636384 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f00a5a30 con 0x7fb0e0004d50 2024-09-16T05:18:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0fd8cc640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0fd8cc640 4 set_mon_vals no callback set 2024-09-16T05:18:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.681+0000 7fb0f5ffb640 10 monclient: dump: 2024-09-16T05:18:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.702 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient: shutdown 2024-09-16T05:18:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 -- 172.21.15.114:0/2580636384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e0004d50 msgr2=0x7fb0e00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 --2- 172.21.15.114:0/2580636384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e0004d50 0x7fb0e00051d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009e30 tx=0x7fb0f001caf0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 1 -- 172.21.15.114:0/2580636384 reap_dead start 2024-09-16T05:18:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 -- 172.21.15.114:0/2580636384 shutdown_connections 2024-09-16T05:18:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 -- 172.21.15.114:0/2580636384 >> 172.21.15.114:0/2580636384 conn(0x7fb0f80a9a40 msgr2=0x7fb0f80a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 -- 172.21.15.114:0/2580636384 shutdown_connections 2024-09-16T05:18:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 -- 172.21.15.114:0/2580636384 wait complete. 2024-09-16T05:18:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient: build_initial_monmap 2024-09-16T05:18:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient: monmap: 2024-09-16T05:18:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:38.703 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 librados: starting msgr at 2024-09-16T05:18:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 librados: starting objecter 2024-09-16T05:18:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 Processor -- start 2024-09-16T05:18:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 -- start start 2024-09-16T05:18:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 librados: setting wanted keys 2024-09-16T05:18:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 librados: calling monclient init 2024-09-16T05:18:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient: init 2024-09-16T05:18:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:18:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f81ceee0 0x7fb0f81357a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient(hunting): picked mon.noname-c con 0x7fb0f81ceee0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb0f8135ce0 0x7fb0f80a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient(hunting): picked mon.noname-b con 0x7fb0f8135ce0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f80a1350 0x7fb0f80a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient(hunting): picked mon.noname-a con 0x7fb0f80a1350 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f813f830 con 0x7fb0f80a1350 2024-09-16T05:18:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f813f6b0 con 0x7fb0f8135ce0 2024-09-16T05:18:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb0f813f9b0 con 0x7fb0f81ceee0 2024-09-16T05:18:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f6ffd640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f81ceee0 msgr2=0x7fb0f81357a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:18:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient(hunting): authenticate will time out at 1603.317548s 2024-09-16T05:18:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f6ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f81ceee0 0x7fb0f81357a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f80a1350 0x7fb0f80a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f80a1350 0x7fb0f80a17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34056/0 (socket says 172.21.15.114:34056) 2024-09-16T05:18:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 1 -- 172.21.15.114:0/2386516273 learned_addr learned my addr 172.21.15.114:0/2386516273 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 10 monclient(hunting): get_auth_request con 0x7fb0f80a1350 auth_method 0 2024-09-16T05:18:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f67fc640 1 --2- 172.21.15.114:0/2386516273 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb0f8135ce0 0x7fb0f80a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f67fc640 10 monclient(hunting): get_auth_request con 0x7fb0f8135ce0 auth_method 0 2024-09-16T05:18:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 10 monclient(hunting): handle_auth_done global_id 74469 payload 824 2024-09-16T05:18:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 1 -- 172.21.15.114:0/2386516273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f81ceee0 msgr2=0x7fb0f81357a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:18:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 1 --2- 172.21.15.114:0/2386516273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f81ceee0 0x7fb0f81357a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 1 -- 172.21.15.114:0/2386516273 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb0f8135ce0 msgr2=0x7fb0f80a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 1 --2- 172.21.15.114:0/2386516273 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb0f8135ce0 0x7fb0f80a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 10 monclient: _finish_hunting 0 2024-09-16T05:18:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 1 monclient: found mon.noname-a 2024-09-16T05:18:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 1 -- 172.21.15.114:0/2386516273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f8137f80 con 0x7fb0f80a1350 2024-09-16T05:18:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f77fe640 1 --2- 172.21.15.114:0/2386516273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f80a1350 0x7fb0f80a17d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb0f001cfd0 tx=0x7fb0e0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 10 client.?.objecter ms_handle_connect 0x7fb0f80a1350 2024-09-16T05:18:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 1 -- 172.21.15.114:0/2386516273 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0009a30 con 0x7fb0f80a1350 2024-09-16T05:18:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 10 monclient: dump: 2024-09-16T05:18:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.709 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 10 monclient: _finish_auth 0 2024-09-16T05:18:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 10 monclient: _check_auth_tickets 2024-09-16T05:18:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 1 -- 172.21.15.114:0/2386516273 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f00a5d50 con 0x7fb0f80a1350 2024-09-16T05:18:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 5 monclient: authenticate success, global_id 74469 2024-09-16T05:18:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient: _renew_subs 2024-09-16T05:18:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 -- 172.21.15.114:0/2386516273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0f813d100 con 0x7fb0f80a1350 2024-09-16T05:18:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 1 -- 172.21.15.114:0/2386516273 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f00092d0 con 0x7fb0f80a1350 2024-09-16T05:18:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 10 monclient: dump: 2024-09-16T05:18:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:38.710 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 client.74469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f57fa640 4 set_mon_vals no callback set 2024-09-16T05:18:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient: _renew_subs 2024-09-16T05:18:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 -- 172.21.15.114:0/2386516273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0f8121e70 con 0x7fb0f80a1350 2024-09-16T05:18:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 librados: init done 2024-09-16T05:18:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0fd8cc640 1 -- 172.21.15.114:0/2386516273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0f8137ac0 con 0x7fb0f80a1350 2024-09-16T05:18:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 1 -- 172.21.15.114:0/2386516273 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb0f00ae670 con 0x7fb0f80a1350 2024-09-16T05:18:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 1 --2- 172.21.15.114:0/2386516273 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb0e40377c0 0x7fb0e4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 1 -- 172.21.15.114:0/2386516273 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(432..432 src has 1..432) ==== 5462+0+0 (secure 0 0 0) 0x7fb0f00e2c20 con 0x7fb0f80a1350 2024-09-16T05:18:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 10 client.74469.objecter ms_dispatch 0x7fb0f8000ba0 osd_map(432..432 src has 1..432) 2024-09-16T05:18:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 3 client.74469.objecter handle_osd_map got epochs [432,432] > 0 2024-09-16T05:18:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 3 client.74469.objecter handle_osd_map decoding full epoch 432 2024-09-16T05:18:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0dffff640 20 client.74469.objecter dump_active .. 0 homeless 2024-09-16T05:18:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f6ffd640 1 --2- 172.21.15.114:0/2386516273 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb0e40377c0 0x7fb0e4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.685+0000 7fb0f6ffd640 10 monclient: get_auth_request con 0x7fb0e40377c0 auth_method 0 2024-09-16T05:18:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.689+0000 7fb0dffff640 10 client.74469.objecter ms_handle_connect 0x7fb0e40377c0 2024-09-16T05:18:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.689+0000 7fb0f6ffd640 1 --2- 172.21.15.114:0/2386516273 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb0e40377c0 0x7fb0e4039c80 secure :-1 s=READY pgs=3547 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8001a50 tx=0x7fb0e8001bb0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.689+0000 7fb0dffff640 1 -- 172.21.15.114:0/2386516273 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb0f0016ac0 con 0x7fb0f80a1350 2024-09-16T05:18:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.693+0000 7fb0dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.693+0000 7fb0dffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:38.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.909+0000 7fb0fd8cc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:18:38.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.909+0000 7fb0fd8cc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:38.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.909+0000 7fb0fd8cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:38.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.909+0000 7fb0fd8cc640 1 -- 172.21.15.114:0/2386516273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb0f8120740 con 0x7fb0f80a1350 2024-09-16T05:18:38.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.909+0000 7fb0dffff640 1 -- 172.21.15.114:0/2386516273 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fb0f00054d0 con 0x7fb0f80a1350 2024-09-16T05:18:38.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.909+0000 7fb0dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:38.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.909+0000 7fb0dffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:38.910 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:18:38.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 10 librados: watch_flush enter 2024-09-16T05:18:38.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 10 librados: watch_flush exit 2024-09-16T05:18:38.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 20 client.74469.objecter shutdown clearing up homeless session... 2024-09-16T05:18:38.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 10 client.74469.objecter successfully canceled tick 2024-09-16T05:18:38.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 1 -- 172.21.15.114:0/2386516273 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb0e40377c0 msgr2=0x7fb0e4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 1 --2- 172.21.15.114:0/2386516273 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb0e40377c0 0x7fb0e4039c80 secure :-1 s=READY pgs=3547 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8001a50 tx=0x7fb0e8001bb0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 10 monclient: shutdown 2024-09-16T05:18:38.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:38.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 1 -- 172.21.15.114:0/2386516273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f80a1350 msgr2=0x7fb0f80a17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:38.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 1 --2- 172.21.15.114:0/2386516273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f80a1350 0x7fb0f80a17d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb0f001cfd0 tx=0x7fb0e0005840 comp rx=0 tx=0).stop 2024-09-16T05:18:38.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 1 -- 172.21.15.114:0/2386516273 shutdown_connections 2024-09-16T05:18:38.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 1 --2- 172.21.15.114:0/2386516273 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb0e40377c0 0x7fb0e4039c80 unknown :-1 s=CLOSED pgs=3547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 1 --2- 172.21.15.114:0/2386516273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f80a1350 0x7fb0f80a17d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 1 --2- 172.21.15.114:0/2386516273 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb0f8135ce0 0x7fb0f80a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 1 --2- 172.21.15.114:0/2386516273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f81ceee0 0x7fb0f81357a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 1 -- 172.21.15.114:0/2386516273 >> 172.21.15.114:0/2386516273 conn(0x7fb0f80a9a40 msgr2=0x7fb0f80a8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 1 -- 172.21.15.114:0/2386516273 shutdown_connections 2024-09-16T05:18:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 1 -- 172.21.15.114:0/2386516273 wait complete. 2024-09-16T05:18:38.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:38.913+0000 7fb0fd8cc640 1 librados: shutdown 2024-09-16T05:18:41.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:18:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 10 monclient: get_monmap_and_config 2024-09-16T05:18:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 10 monclient: build_initial_monmap 2024-09-16T05:18:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 10 monclient: monmap: 2024-09-16T05:18:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:42.060716+0000 2024-09-16T05:18:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:42.060716+0000 2024-09-16T05:18:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:42.057 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 1 Processor -- start 2024-09-16T05:18:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 1 -- start start 2024-09-16T05:18:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 10 monclient: init 2024-09-16T05:18:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-16T05:18:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7febbc1376e0 addr v1:172.21.15.149:6789/0 2024-09-16T05:18:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7febbc1370a0 addr v1:172.21.15.114:6789/0 2024-09-16T05:18:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7febbc0a5d00 0x7febbc137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 10 monclient(hunting): picked mon.noname-c con 0x7febbc0a5d00 addr v2:172.21.15.149:3300/0 2024-09-16T05:18:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febbc0a5090 con 0x7febbc1370a0 2024-09-16T05:18:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febbc09bd20 con 0x7febbc1376e0 2024-09-16T05:18:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7febbc138000 con 0x7febbc0a5d00 2024-09-16T05:18:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 10 monclient(hunting): authenticate will time out at 1606.690607s 2024-09-16T05:18:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbb7fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7febbc1370a0 0x7febbc0a4980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:41220/0 (socket says 172.21.15.114:41220) 2024-09-16T05:18:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbb7fe640 1 -- 172.21.15.114:0/1961364280 learned_addr learned my addr 172.21.15.114:0/1961364280 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:18:42.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 -- 172.21.15.114:0/1961364280 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 456509213 0 0) 0x7febbc0a5090 con 0x7febbc1370a0 2024-09-16T05:18:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 1 --2- 172.21.15.114:0/1961364280 >> v2:172.21.15.149:3300/0 conn(0x7febbc0a5d00 0x7febbc137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:18:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient(hunting): my global_id is 74472 2024-09-16T05:18:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 -- 172.21.15.114:0/1961364280 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feba4003530 con 0x7febbc1370a0 2024-09-16T05:18:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 10 monclient(hunting): get_auth_request con 0x7febbc0a5d00 auth_method 0 2024-09-16T05:18:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 -- 172.21.15.114:0/1961364280 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2304568775 0 0) 0x7feba4003530 con 0x7febbc1370a0 2024-09-16T05:18:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 -- 172.21.15.114:0/1961364280 >> v2:172.21.15.149:3300/0 conn(0x7febbc0a5d00 msgr2=0x7febbc137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 --2- 172.21.15.114:0/1961364280 >> v2:172.21.15.149:3300/0 conn(0x7febbc0a5d00 0x7febbc137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 -- 172.21.15.114:0/1961364280 >> v1:172.21.15.149:6789/0 conn(0x7febbc1376e0 legacy=0x7febbc0a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient: _finish_hunting 0 2024-09-16T05:18:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:18:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:18:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 -- 172.21.15.114:0/1961364280 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febbc09c000 con 0x7febbc1370a0 2024-09-16T05:18:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient: _finish_auth 0 2024-09-16T05:18:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient: _check_auth_tickets 2024-09-16T05:18:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 5 monclient: authenticate success, global_id 74472 2024-09-16T05:18:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 -- 172.21.15.114:0/1961364280 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7febb0001680 con 0x7febbc1370a0 2024-09-16T05:18:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:18:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient: dump: 2024-09-16T05:18:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:42.068 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:42.068 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:42.068 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:42.068 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:42.068 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:42.068 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:42.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:18:42.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:42.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 -- 172.21.15.114:0/1961364280 >> v1:172.21.15.114:6789/0 conn(0x7febbc1370a0 legacy=0x7febbc0a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:42.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:18:42.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 --2- 172.21.15.114:0/1961364280 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feba40039c0 0x7feba4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:42.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient(hunting): picked mon.b con 0x7feba40039c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:42.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 --2- 172.21.15.114:0/1961364280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feba4004810 0x7feba4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:42.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient(hunting): picked mon.c con 0x7feba4004810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:42.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 --2- 172.21.15.114:0/1961364280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba40052f0 0x7feba400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:42.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient(hunting): picked mon.a con 0x7feba40052f0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:42.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:42.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 -- 172.21.15.114:0/1961364280 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feba400c190 con 0x7feba40052f0 2024-09-16T05:18:42.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:42.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 -- 172.21.15.114:0/1961364280 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7feba400c340 con 0x7feba40039c0 2024-09-16T05:18:42.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:42.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 1 -- 172.21.15.114:0/1961364280 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7feba400c540 con 0x7feba4004810 2024-09-16T05:18:42.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:42.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:18:42.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc240f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:42.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbb7fe640 1 -- 172.21.15.114:0/1961364280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feba4004810 msgr2=0x7feba4004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:18:42.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbb7fe640 1 --2- 172.21.15.114:0/1961364280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feba4004810 0x7feba4004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:42.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 1 --2- 172.21.15.114:0/1961364280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba40052f0 0x7feba400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:42.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 10 monclient(hunting): get_auth_request con 0x7feba40052f0 auth_method 0 2024-09-16T05:18:42.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:42.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:42.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:42.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbbfff640 1 --2- 172.21.15.114:0/1961364280 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feba40039c0 0x7feba4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:42.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:42.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:42.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:18:42.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbbfff640 10 monclient(hunting): get_auth_request con 0x7feba40039c0 auth_method 0 2024-09-16T05:18:42.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:42.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:42.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febbbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:42.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 10 monclient(hunting): handle_auth_done global_id 74472 payload 293 2024-09-16T05:18:42.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 1 -- 172.21.15.114:0/1961364280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feba4004810 msgr2=0x7feba4004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:18:42.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.057+0000 7febc09ac640 1 --2- 172.21.15.114:0/1961364280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feba4004810 0x7feba4004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:42.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc09ac640 1 -- 172.21.15.114:0/1961364280 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feba40039c0 msgr2=0x7feba4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:42.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc09ac640 1 --2- 172.21.15.114:0/1961364280 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feba40039c0 0x7feba4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:42.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc09ac640 10 monclient: _finish_hunting 0 2024-09-16T05:18:42.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc09ac640 1 monclient: found mon.a 2024-09-16T05:18:42.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc09ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:42.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc09ac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:42.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc09ac640 1 -- 172.21.15.114:0/1961364280 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba4003530 con 0x7feba40052f0 2024-09-16T05:18:42.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc09ac640 10 monclient: _finish_auth 0 2024-09-16T05:18:42.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc09ac640 10 monclient: _check_auth_tickets 2024-09-16T05:18:42.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc09ac640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:42.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febbbfff640 1 --2- 172.21.15.114:0/1961364280 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feba40039c0 0x7feba4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:18:42.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc09ac640 1 -- 172.21.15.114:0/1961364280 reap_dead start 2024-09-16T05:18:42.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc09ac640 1 --2- 172.21.15.114:0/1961364280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba40052f0 0x7feba400b9d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7febb4009ab0 tx=0x7febb401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f5955bc66db34b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:42.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febbaffd640 1 -- 172.21.15.114:0/1961364280 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb40096b0 con 0x7feba40052f0 2024-09-16T05:18:42.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:42.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febbaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:42.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febbaffd640 10 monclient: dump: 2024-09-16T05:18:42.072 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:42.072 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:42.072 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:42.073 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:42.073 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:42.073 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:42.073 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:42.073 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:42.073 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:42.073 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:42.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:42.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febbaffd640 1 -- 172.21.15.114:0/1961364280 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febb4004020 con 0x7feba40052f0 2024-09-16T05:18:42.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febbaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:42.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:42.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 4 set_mon_vals no callback set 2024-09-16T05:18:42.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febbaffd640 1 -- 172.21.15.114:0/1961364280 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb4004410 con 0x7feba40052f0 2024-09-16T05:18:42.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:42.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febbaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:42.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febbaffd640 10 monclient: dump: 2024-09-16T05:18:42.074 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:42.074 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:42.074 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:42.074 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:42.074 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:42.074 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:42.074 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:42.074 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:42.074 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:42.074 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:42.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 10 monclient: shutdown 2024-09-16T05:18:42.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 1 -- 172.21.15.114:0/1961364280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba40052f0 msgr2=0x7feba400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 1 --2- 172.21.15.114:0/1961364280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba40052f0 0x7feba400b9d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7febb4009ab0 tx=0x7febb401ca70 comp rx=0 tx=0).stop 2024-09-16T05:18:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 1 -- 172.21.15.114:0/1961364280 shutdown_connections 2024-09-16T05:18:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 1 --2- 172.21.15.114:0/1961364280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba40052f0 0x7feba400b9d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 1 -- 172.21.15.114:0/1961364280 >> 172.21.15.114:0/1961364280 conn(0x7febbc0a9a40 msgr2=0x7febbc0a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 1 -- 172.21.15.114:0/1961364280 shutdown_connections 2024-09-16T05:18:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 1 -- 172.21.15.114:0/1961364280 wait complete. 2024-09-16T05:18:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 10 monclient: build_initial_monmap 2024-09-16T05:18:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 10 monclient: monmap: 2024-09-16T05:18:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:42.076 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:42.076 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:42.076 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:42.076 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:42.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 1 librados: starting msgr at 2024-09-16T05:18:42.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.061+0000 7febc240f640 1 librados: starting objecter 2024-09-16T05:18:42.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 1 Processor -- start 2024-09-16T05:18:42.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 1 -- start start 2024-09-16T05:18:42.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 1 librados: setting wanted keys 2024-09-16T05:18:42.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 1 librados: calling monclient init 2024-09-16T05:18:42.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient: init 2024-09-16T05:18:42.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:42.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:18:42.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feba40052f0 0x7febbc1ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:42.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient(hunting): picked mon.noname-c con 0x7feba40052f0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:42.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febbc136300 0x7febbc136780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:42.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient(hunting): picked mon.noname-a con 0x7febbc136300 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:42.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7febbc13c110 0x7febbc13c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:42.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient(hunting): picked mon.noname-b con 0x7febbc13c110 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:42.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:42.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7febbc137e50 con 0x7febbc136300 2024-09-16T05:18:42.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:42.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7febbc137cd0 con 0x7febbc13c110 2024-09-16T05:18:42.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:42.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7febbc0a4270 con 0x7feba40052f0 2024-09-16T05:18:42.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:42.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient(hunting): authenticate will time out at 1606.696603s 2024-09-16T05:18:42.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbbfff640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feba40052f0 msgr2=0x7febbc1ceb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:18:42.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feba40052f0 0x7febbc1ceb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:42.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febbc136300 0x7febbc136780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:42.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febbc136300 0x7febbc136780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34080/0 (socket says 172.21.15.114:34080) 2024-09-16T05:18:42.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 1 -- 172.21.15.114:0/472486610 learned_addr learned my addr 172.21.15.114:0/472486610 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:42.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 10 monclient(hunting): get_auth_request con 0x7febbc136300 auth_method 0 2024-09-16T05:18:42.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:42.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:42.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:42.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:42.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:42.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:42.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 10 monclient(hunting): handle_auth_done global_id 74475 payload 824 2024-09-16T05:18:42.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 1 -- 172.21.15.114:0/472486610 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feba40052f0 msgr2=0x7febbc1ceb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:18:42.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 1 --2- 172.21.15.114:0/472486610 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feba40052f0 0x7febbc1ceb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:42.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 1 -- 172.21.15.114:0/472486610 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7febbc13c110 msgr2=0x7febbc13c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:42.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 1 --2- 172.21.15.114:0/472486610 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7febbc13c110 0x7febbc13c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:42.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:18:42.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 1 monclient: found mon.noname-a 2024-09-16T05:18:42.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:42.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:42.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 1 -- 172.21.15.114:0/472486610 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febbc073d20 con 0x7febbc136300 2024-09-16T05:18:42.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbb7fe640 1 --2- 172.21.15.114:0/472486610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febbc136300 0x7febbc136780 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7febb00011f0 tx=0x7febb000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:42.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 10 client.?.objecter ms_handle_connect 0x7febbc136300 2024-09-16T05:18:42.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:42.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 1 -- 172.21.15.114:0/472486610 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb0018070 con 0x7febbc136300 2024-09-16T05:18:42.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:42.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:42.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 10 monclient: dump: 2024-09-16T05:18:42.084 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:42.084 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:42.084 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:42.084 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:42.084 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:42.084 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:42.084 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:42.084 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:42.084 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:42.084 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:42.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 10 monclient: _finish_auth 0 2024-09-16T05:18:42.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:18:42.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:42.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 1 -- 172.21.15.114:0/472486610 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febb00072d0 con 0x7febbc136300 2024-09-16T05:18:42.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 5 monclient: authenticate success, global_id 74475 2024-09-16T05:18:42.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient: _renew_subs 2024-09-16T05:18:42.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:42.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 1 -- 172.21.15.114:0/472486610 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febbc13ec00 con 0x7febbc136300 2024-09-16T05:18:42.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:42.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 1 -- 172.21.15.114:0/472486610 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb00047f0 con 0x7febbc136300 2024-09-16T05:18:42.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:42.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:42.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 10 monclient: dump: 2024-09-16T05:18:42.085 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:42.085 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:42.085 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:42.085 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:42.086 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:42.086 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:42.086 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:42.086 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:42.086 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:42.086 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:42.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febba7fc640 4 set_mon_vals no callback set 2024-09-16T05:18:42.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 client.74475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:42.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient: _renew_subs 2024-09-16T05:18:42.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:42.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 1 -- 172.21.15.114:0/472486610 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febbc13e960 con 0x7febbc136300 2024-09-16T05:18:42.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 1 librados: init done 2024-09-16T05:18:42.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:42.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:42.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:42.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febc240f640 1 -- 172.21.15.114:0/472486610 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febbc0a4600 con 0x7febbc136300 2024-09-16T05:18:42.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 1 -- 172.21.15.114:0/472486610 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7febb001b070 con 0x7febbc136300 2024-09-16T05:18:42.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 1 --2- 172.21.15.114:0/472486610 >> v2:172.21.15.149:6812/1185531135 conn(0x7feba80377c0 0x7feba8039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:42.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 1 -- 172.21.15.114:0/472486610 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(432..432 src has 1..432) ==== 5462+0+0 (secure 0 0 0) 0x7febb00482d0 con 0x7febbc136300 2024-09-16T05:18:42.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 10 client.74475.objecter ms_dispatch 0x7febbc000ba0 osd_map(432..432 src has 1..432) 2024-09-16T05:18:42.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 3 client.74475.objecter handle_osd_map got epochs [432,432] > 0 2024-09-16T05:18:42.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 3 client.74475.objecter handle_osd_map decoding full epoch 432 2024-09-16T05:18:42.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 20 client.74475.objecter dump_active .. 0 homeless 2024-09-16T05:18:42.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbbfff640 1 --2- 172.21.15.114:0/472486610 >> v2:172.21.15.149:6812/1185531135 conn(0x7feba80377c0 0x7feba8039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:42.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbbfff640 10 monclient: get_auth_request con 0x7feba80377c0 auth_method 0 2024-09-16T05:18:42.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febb8ff9640 10 client.74475.objecter ms_handle_connect 0x7feba80377c0 2024-09-16T05:18:42.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.065+0000 7febbbfff640 1 --2- 172.21.15.114:0/472486610 >> v2:172.21.15.149:6812/1185531135 conn(0x7feba80377c0 0x7feba8039c80 secure :-1 s=READY pgs=3548 cs=0 l=1 rev1=1 crypto rx=0x7febac007fd0 tx=0x7febac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:42.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.069+0000 7febb8ff9640 1 -- 172.21.15.114:0/472486610 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7febb0010300 con 0x7febbc136300 2024-09-16T05:18:42.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.069+0000 7febb8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:42.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.069+0000 7febb8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:42.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:18:42.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:42.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:42.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 1 -- 172.21.15.114:0/472486610 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7febbc068820 con 0x7febbc136300 2024-09-16T05:18:42.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febb8ff9640 1 -- 172.21.15.114:0/472486610 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7febb001be30 con 0x7febbc136300 2024-09-16T05:18:42.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febb8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:42.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febb8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:42.301 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:18:42.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 10 librados: watch_flush enter 2024-09-16T05:18:42.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 10 librados: watch_flush exit 2024-09-16T05:18:42.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 20 client.74475.objecter shutdown clearing up homeless session... 2024-09-16T05:18:42.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 10 client.74475.objecter successfully canceled tick 2024-09-16T05:18:42.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 1 -- 172.21.15.114:0/472486610 >> v2:172.21.15.149:6812/1185531135 conn(0x7feba80377c0 msgr2=0x7feba8039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:42.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 1 --2- 172.21.15.114:0/472486610 >> v2:172.21.15.149:6812/1185531135 conn(0x7feba80377c0 0x7feba8039c80 secure :-1 s=READY pgs=3548 cs=0 l=1 rev1=1 crypto rx=0x7febac007fd0 tx=0x7febac009040 comp rx=0 tx=0).stop 2024-09-16T05:18:42.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 10 monclient: shutdown 2024-09-16T05:18:42.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:42.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 1 -- 172.21.15.114:0/472486610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febbc136300 msgr2=0x7febbc136780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:42.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 1 --2- 172.21.15.114:0/472486610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febbc136300 0x7febbc136780 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7febb00011f0 tx=0x7febb000cb70 comp rx=0 tx=0).stop 2024-09-16T05:18:42.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 1 -- 172.21.15.114:0/472486610 shutdown_connections 2024-09-16T05:18:42.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 1 --2- 172.21.15.114:0/472486610 >> v2:172.21.15.149:6812/1185531135 conn(0x7feba80377c0 0x7feba8039c80 unknown :-1 s=CLOSED pgs=3548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:42.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 1 --2- 172.21.15.114:0/472486610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febbc136300 0x7febbc136780 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:42.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 1 --2- 172.21.15.114:0/472486610 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7febbc13c110 0x7febbc13c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:42.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 1 --2- 172.21.15.114:0/472486610 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feba40052f0 0x7febbc1ceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:42.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 1 -- 172.21.15.114:0/472486610 >> 172.21.15.114:0/472486610 conn(0x7febbc0a9a40 msgr2=0x7febbc09fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:42.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 1 -- 172.21.15.114:0/472486610 shutdown_connections 2024-09-16T05:18:42.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 1 -- 172.21.15.114:0/472486610 wait complete. 2024-09-16T05:18:42.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:42.301+0000 7febc240f640 1 librados: shutdown 2024-09-16T05:18:45.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:18:45.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 10 monclient: get_monmap_and_config 2024-09-16T05:18:45.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 10 monclient: build_initial_monmap 2024-09-16T05:18:45.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 10 monclient: monmap: 2024-09-16T05:18:45.452 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:45.452 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:45.457368+0000 2024-09-16T05:18:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:45.457368+0000 2024-09-16T05:18:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:45.453 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:45.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 1 Processor -- start 2024-09-16T05:18:45.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 1 -- start start 2024-09-16T05:18:45.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 10 monclient: init 2024-09-16T05:18:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T05:18:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feadc137730 0x7feadc137b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 10 monclient(hunting): picked mon.noname-a con 0x7feadc137730 addr v2:172.21.15.114:3300/0 2024-09-16T05:18:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7feadc0a5d00 0x7feadc0a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 10 monclient(hunting): picked mon.noname-c con 0x7feadc0a5d00 addr v2:172.21.15.149:3300/0 2024-09-16T05:18:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feadc0a47b0 addr v1:172.21.15.149:6789/0 2024-09-16T05:18:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feadc0a55e0 con 0x7feadc0a47b0 2024-09-16T05:18:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7feadc09c450 con 0x7feadc137730 2024-09-16T05:18:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7feadc0a3710 con 0x7feadc0a5d00 2024-09-16T05:18:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feae0c3a640 10 monclient(hunting): authenticate will time out at 1610.086568s 2024-09-16T05:18:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feadc137730 0x7feadc137b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feadc137730 0x7feadc137b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34086/0 (socket says 172.21.15.114:34086) 2024-09-16T05:18:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 1 -- 172.21.15.114:0/2033011409 learned_addr learned my addr 172.21.15.114:0/2033011409 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 10 monclient(hunting): get_auth_request con 0x7feadc137730 auth_method 0 2024-09-16T05:18:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7fead9d9b640 1 --2- 172.21.15.114:0/2033011409 >> v2:172.21.15.149:3300/0 conn(0x7feadc0a5d00 0x7feadc0a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 10 monclient(hunting): handle_auth_done global_id 74478 payload 274 2024-09-16T05:18:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 1 -- 172.21.15.114:0/2033011409 >> v2:172.21.15.149:3300/0 conn(0x7feadc0a5d00 msgr2=0x7feadc0a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:45.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 1 --2- 172.21.15.114:0/2033011409 >> v2:172.21.15.149:3300/0 conn(0x7feadc0a5d00 0x7feadc0a4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:45.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 1 -- 172.21.15.114:0/2033011409 >> v1:172.21.15.149:6789/0 conn(0x7feadc0a47b0 legacy=0x7feadc13e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:45.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 10 monclient: _finish_hunting 0 2024-09-16T05:18:45.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 1 monclient: found mon.noname-a 2024-09-16T05:18:45.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:45.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:45.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7feada59c640 1 -- 172.21.15.114:0/2033011409 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feadc09be00 con 0x7feadc137730 2024-09-16T05:18:45.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7fead9d9b640 10 monclient: get_auth_request con 0x7feadc0a5d00 auth_method 0 2024-09-16T05:18:45.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.453+0000 7fead9d9b640 1 --2- 172.21.15.114:0/2033011409 >> v2:172.21.15.149:3300/0 conn(0x7feadc0a5d00 0x7feadc0a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:18:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feada59c640 1 --2- 172.21.15.114:0/2033011409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feadc137730 0x7feadc137b10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feadc073240 tx=0x7feac401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30b75a73e87cfd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7fead959a640 1 -- 172.21.15.114:0/2033011409 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac40b0070 con 0x7feadc137730 2024-09-16T05:18:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7fead959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7fead959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7fead959a640 10 monclient: dump: 2024-09-16T05:18:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:45.464 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7fead959a640 10 monclient: _finish_auth 0 2024-09-16T05:18:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7fead959a640 10 monclient: _check_auth_tickets 2024-09-16T05:18:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7fead959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7fead959a640 1 -- 172.21.15.114:0/2033011409 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feac40ab600 con 0x7feadc137730 2024-09-16T05:18:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7fead959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7fead959a640 1 -- 172.21.15.114:0/2033011409 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac40ab9d0 con 0x7feadc137730 2024-09-16T05:18:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 5 monclient: authenticate success, global_id 74478 2024-09-16T05:18:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 4 set_mon_vals no callback set 2024-09-16T05:18:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7fead959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7fead959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:45.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7fead959a640 10 monclient: dump: 2024-09-16T05:18:45.465 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:45.465 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:45.465 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:45.465 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:45.465 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:45.465 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:45.465 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:45.465 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:45.465 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:45.465 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:45.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 10 monclient: shutdown 2024-09-16T05:18:45.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:45.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 1 -- 172.21.15.114:0/2033011409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feadc137730 msgr2=0x7feadc137b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:45.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 1 --2- 172.21.15.114:0/2033011409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feadc137730 0x7feadc137b10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feadc073240 tx=0x7feac401c400 comp rx=0 tx=0).stop 2024-09-16T05:18:45.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 1 -- 172.21.15.114:0/2033011409 shutdown_connections 2024-09-16T05:18:45.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 1 --2- 172.21.15.114:0/2033011409 >> v2:172.21.15.149:3300/0 conn(0x7feadc0a5d00 0x7feadc0a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:45.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 1 --2- 172.21.15.114:0/2033011409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feadc137730 0x7feadc137b10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:45.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 1 -- 172.21.15.114:0/2033011409 >> 172.21.15.114:0/2033011409 conn(0x7feadc0a9c70 msgr2=0x7feadc0aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:45.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 1 -- 172.21.15.114:0/2033011409 shutdown_connections 2024-09-16T05:18:45.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 1 -- 172.21.15.114:0/2033011409 wait complete. 2024-09-16T05:18:45.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 10 monclient: build_initial_monmap 2024-09-16T05:18:45.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 10 monclient: monmap: 2024-09-16T05:18:45.466 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:45.466 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:45.466 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:45.466 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:45.466 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:45.467 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:45.467 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:45.467 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:45.467 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:45.467 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:45.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 1 librados: starting msgr at 2024-09-16T05:18:45.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 1 librados: starting objecter 2024-09-16T05:18:45.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 1 Processor -- start 2024-09-16T05:18:45.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 1 -- start start 2024-09-16T05:18:45.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 1 librados: setting wanted keys 2024-09-16T05:18:45.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 1 librados: calling monclient init 2024-09-16T05:18:45.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.457+0000 7feae0c3a640 10 monclient: init 2024-09-16T05:18:45.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:45.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:18:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feadc0a47b0 0x7feadc136140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient(hunting): picked mon.noname-a con 0x7feadc0a47b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feadc0a5d00 0x7feadc13c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient(hunting): picked mon.noname-c con 0x7feadc0a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feadc137730 0x7feadc13c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient(hunting): picked mon.noname-b con 0x7feadc137730 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feadc072a20 con 0x7feadc0a47b0 2024-09-16T05:18:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7feadc0728a0 con 0x7feadc137730 2024-09-16T05:18:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7feadc1371a0 con 0x7feadc0a5d00 2024-09-16T05:18:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient(hunting): authenticate will time out at 1610.092328s 2024-09-16T05:18:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead9d9b640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feadc0a5d00 msgr2=0x7feadc13c230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:18:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead9d9b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feadc0a5d00 0x7feadc13c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feadc0a47b0 0x7feadc136140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feadc0a47b0 0x7feadc136140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34100/0 (socket says 172.21.15.114:34100) 2024-09-16T05:18:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 1 -- 172.21.15.114:0/980986665 learned_addr learned my addr 172.21.15.114:0/980986665 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 10 monclient(hunting): get_auth_request con 0x7feadc0a47b0 auth_method 0 2024-09-16T05:18:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feadad9d640 1 --2- 172.21.15.114:0/980986665 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feadc137730 0x7feadc13c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feadad9d640 10 monclient(hunting): get_auth_request con 0x7feadc137730 auth_method 0 2024-09-16T05:18:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feadad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feadad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feadad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 10 monclient(hunting): handle_auth_done global_id 74481 payload 824 2024-09-16T05:18:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 1 -- 172.21.15.114:0/980986665 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feadc0a5d00 msgr2=0x7feadc13c230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:18:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 1 --2- 172.21.15.114:0/980986665 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feadc0a5d00 0x7feadc13c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 1 -- 172.21.15.114:0/980986665 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feadc137730 msgr2=0x7feadc13c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 1 --2- 172.21.15.114:0/980986665 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feadc137730 0x7feadc13c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 10 monclient: _finish_hunting 0 2024-09-16T05:18:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 1 monclient: found mon.noname-a 2024-09-16T05:18:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 1 -- 172.21.15.114:0/980986665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feadc1205f0 con 0x7feadc0a47b0 2024-09-16T05:18:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feada59c640 1 --2- 172.21.15.114:0/980986665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feadc0a47b0 0x7feadc136140 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7feac40ae040 tx=0x7feac40191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 10 client.?.objecter ms_handle_connect 0x7feadc0a47b0 2024-09-16T05:18:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 1 -- 172.21.15.114:0/980986665 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac40b0040 con 0x7feadc0a47b0 2024-09-16T05:18:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 10 monclient: dump: 2024-09-16T05:18:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:45.472 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:18:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:18:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 1 -- 172.21.15.114:0/980986665 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feac40b9070 con 0x7feadc0a47b0 2024-09-16T05:18:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 5 monclient: authenticate success, global_id 74481 2024-09-16T05:18:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient: _renew_subs 2024-09-16T05:18:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 1 -- 172.21.15.114:0/980986665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feadc13ea30 con 0x7feadc0a47b0 2024-09-16T05:18:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 1 -- 172.21.15.114:0/980986665 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac40b48a0 con 0x7feadc0a47b0 2024-09-16T05:18:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 10 monclient: dump: 2024-09-16T05:18:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:45.473 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead8d99640 4 set_mon_vals no callback set 2024-09-16T05:18:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 client.74481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient: _renew_subs 2024-09-16T05:18:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 1 -- 172.21.15.114:0/980986665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feadc09cec0 con 0x7feadc0a47b0 2024-09-16T05:18:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 1 librados: init done 2024-09-16T05:18:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7feae0c3a640 1 -- 172.21.15.114:0/980986665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feadc0a4270 con 0x7feadc0a47b0 2024-09-16T05:18:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 1 -- 172.21.15.114:0/980986665 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7feac40ab4c0 con 0x7feadc0a47b0 2024-09-16T05:18:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 1 --2- 172.21.15.114:0/980986665 >> v2:172.21.15.149:6812/1185531135 conn(0x7feabc0377c0 0x7feabc039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 1 -- 172.21.15.114:0/980986665 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(432..432 src has 1..432) ==== 5462+0+0 (secure 0 0 0) 0x7feac40e2c30 con 0x7feadc0a47b0 2024-09-16T05:18:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 10 client.74481.objecter ms_dispatch 0x7feadc000ba0 osd_map(432..432 src has 1..432) 2024-09-16T05:18:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 3 client.74481.objecter handle_osd_map got epochs [432,432] > 0 2024-09-16T05:18:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 3 client.74481.objecter handle_osd_map decoding full epoch 432 2024-09-16T05:18:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 20 client.74481.objecter dump_active .. 0 homeless 2024-09-16T05:18:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead9d9b640 1 --2- 172.21.15.114:0/980986665 >> v2:172.21.15.149:6812/1185531135 conn(0x7feabc0377c0 0x7feabc039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead9d9b640 10 monclient: get_auth_request con 0x7feabc0377c0 auth_method 0 2024-09-16T05:18:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead9d9b640 1 --2- 172.21.15.114:0/980986665 >> v2:172.21.15.149:6812/1185531135 conn(0x7feabc0377c0 0x7feabc039c80 secure :-1 s=READY pgs=3549 cs=0 l=1 rev1=1 crypto rx=0x7feadc13f070 tx=0x7feacc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.461+0000 7fead2ffd640 10 client.74481.objecter ms_handle_connect 0x7feabc0377c0 2024-09-16T05:18:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.465+0000 7fead2ffd640 1 -- 172.21.15.114:0/980986665 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7feac40e9080 con 0x7feadc0a47b0 2024-09-16T05:18:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.465+0000 7fead2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.465+0000 7fead2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.677+0000 7feae0c3a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:18:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.677+0000 7feae0c3a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.677+0000 7feae0c3a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.677+0000 7feae0c3a640 1 -- 172.21.15.114:0/980986665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7feadc13e7b0 con 0x7feadc0a47b0 2024-09-16T05:18:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.677+0000 7fead2ffd640 1 -- 172.21.15.114:0/980986665 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7feac40160b0 con 0x7feadc0a47b0 2024-09-16T05:18:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.677+0000 7fead2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.677+0000 7fead2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:45.679 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:18:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 10 librados: watch_flush enter 2024-09-16T05:18:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 10 librados: watch_flush exit 2024-09-16T05:18:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 20 client.74481.objecter shutdown clearing up homeless session... 2024-09-16T05:18:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 10 client.74481.objecter successfully canceled tick 2024-09-16T05:18:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 1 -- 172.21.15.114:0/980986665 >> v2:172.21.15.149:6812/1185531135 conn(0x7feabc0377c0 msgr2=0x7feabc039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 1 --2- 172.21.15.114:0/980986665 >> v2:172.21.15.149:6812/1185531135 conn(0x7feabc0377c0 0x7feabc039c80 secure :-1 s=READY pgs=3549 cs=0 l=1 rev1=1 crypto rx=0x7feadc13f070 tx=0x7feacc008040 comp rx=0 tx=0).stop 2024-09-16T05:18:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 10 monclient: shutdown 2024-09-16T05:18:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 1 -- 172.21.15.114:0/980986665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feadc0a47b0 msgr2=0x7feadc136140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 1 --2- 172.21.15.114:0/980986665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feadc0a47b0 0x7feadc136140 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7feac40ae040 tx=0x7feac40191a0 comp rx=0 tx=0).stop 2024-09-16T05:18:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 1 -- 172.21.15.114:0/980986665 shutdown_connections 2024-09-16T05:18:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 1 --2- 172.21.15.114:0/980986665 >> v2:172.21.15.149:6812/1185531135 conn(0x7feabc0377c0 0x7feabc039c80 unknown :-1 s=CLOSED pgs=3549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 1 --2- 172.21.15.114:0/980986665 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feadc137730 0x7feadc13c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 1 --2- 172.21.15.114:0/980986665 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feadc0a5d00 0x7feadc13c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 1 --2- 172.21.15.114:0/980986665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feadc0a47b0 0x7feadc136140 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 1 -- 172.21.15.114:0/980986665 >> 172.21.15.114:0/980986665 conn(0x7feadc0a9c70 msgr2=0x7feadc0a8600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 1 -- 172.21.15.114:0/980986665 shutdown_connections 2024-09-16T05:18:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 1 -- 172.21.15.114:0/980986665 wait complete. 2024-09-16T05:18:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:45.681+0000 7feae0c3a640 1 librados: shutdown 2024-09-16T05:18:48.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:18:48.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 10 monclient: get_monmap_and_config 2024-09-16T05:18:48.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 10 monclient: build_initial_monmap 2024-09-16T05:18:48.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 10 monclient: monmap: 2024-09-16T05:18:48.822 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:48.822 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:48.822 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:48.826072+0000 2024-09-16T05:18:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:48.826072+0000 2024-09-16T05:18:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:48.824 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:48.824 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:48.824 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:48.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 1 Processor -- start 2024-09-16T05:18:48.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 1 -- start start 2024-09-16T05:18:48.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 10 monclient: init 2024-09-16T05:18:48.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:48.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T05:18:48.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43600a0ec0 addr v1:172.21.15.114:6789/0 2024-09-16T05:18:48.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 0x7f436009e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:48.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 10 monclient(hunting): picked mon.noname-a con 0x7f436009e1d0 addr v2:172.21.15.114:3300/0 2024-09-16T05:18:48.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 0x7f436013ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:48.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 10 monclient(hunting): picked mon.noname-c con 0x7f4360139640 addr v2:172.21.15.149:3300/0 2024-09-16T05:18:48.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f436009a760 con 0x7f43600a0ec0 2024-09-16T05:18:48.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:48.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f43600994b0 con 0x7f436009e1d0 2024-09-16T05:18:48.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:48.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f43600a1870 con 0x7f4360139640 2024-09-16T05:18:48.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:48.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4366be8640 10 monclient(hunting): authenticate will time out at 1613.455199s 2024-09-16T05:18:48.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4364984640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f43600a0ec0 0x7f436009dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38822/0 (socket says 172.21.15.114:38822) 2024-09-16T05:18:48.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f435ffff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 0x7f436009e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:48.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4364984640 1 -- 172.21.15.114:0/2150656935 learned_addr learned my addr 172.21.15.114:0/2150656935 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:18:48.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f435ffff640 10 monclient(hunting): get_auth_request con 0x7f436009e1d0 auth_method 0 2024-09-16T05:18:48.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f435ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:48.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f435ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:48.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f435ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:48.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f435ffff640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 msgr2=0x7f436009e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:18:48.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f435ffff640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 msgr2=0x7f436009e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:48.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f435ffff640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 0x7f436009e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:48.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f435ffff640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 0x7f436009e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:48.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4365185640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 0x7f436013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:48.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4365185640 10 monclient(hunting): get_auth_request con 0x7f4360139640 auth_method 0 2024-09-16T05:18:48.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4365185640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:48.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4365185640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:48.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4365185640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:48.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4365185640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 msgr2=0x7f436013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:18:48.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4365185640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 msgr2=0x7f436013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:48.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4365185640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 0x7f436013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:48.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:48.821+0000 7f4365185640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 0x7f436013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:49.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f435ffff640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 0x7f436009e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:49.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f435ffff640 10 monclient(hunting): get_auth_request con 0x7f436009e1d0 auth_method 0 2024-09-16T05:18:49.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f435ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:49.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f435ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:49.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f435ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:49.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f435ffff640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 msgr2=0x7f436009e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:18:49.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f435ffff640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 msgr2=0x7f436009e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:49.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f435ffff640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 0x7f436009e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:49.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f4365185640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 0x7f436013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:49.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f4365185640 10 monclient(hunting): get_auth_request con 0x7f4360139640 auth_method 0 2024-09-16T05:18:49.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f4365185640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:49.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f4365185640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:49.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f4365185640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:49.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f435ffff640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 0x7f436009e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:18:49.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f4365185640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 msgr2=0x7f436013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:18:49.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f4365185640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 msgr2=0x7f436013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:49.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f4365185640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 0x7f436013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:49.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.025+0000 7f4365185640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 0x7f436013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:18:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f435ffff640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 0x7f436009e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f435ffff640 10 monclient(hunting): get_auth_request con 0x7f436009e1d0 auth_method 0 2024-09-16T05:18:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f435ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f435ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f435ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f4365185640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 0x7f436013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f435ffff640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 msgr2=0x7f436009e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:18:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f435ffff640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 msgr2=0x7f436009e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f435ffff640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 0x7f436009e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f4365185640 10 monclient(hunting): get_auth_request con 0x7f4360139640 auth_method 0 2024-09-16T05:18:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f4365185640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f4365185640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f4365185640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f435ffff640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 0x7f436009e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:18:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f4365185640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 msgr2=0x7f436013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:18:49.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f4365185640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 msgr2=0x7f436013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:49.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f4365185640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 0x7f436013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:49.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:49.429+0000 7f4365185640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 0x7f436013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:18:50.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f435ffff640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 0x7f436009e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:50.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f435ffff640 10 monclient(hunting): get_auth_request con 0x7f436009e1d0 auth_method 0 2024-09-16T05:18:50.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f435ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:50.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f435ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:50.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f435ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:50.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f435ffff640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 msgr2=0x7f436009e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:18:50.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f435ffff640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 msgr2=0x7f436009e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:50.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f435ffff640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 0x7f436009e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:50.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f4365185640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 0x7f436013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:50.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f4365185640 10 monclient(hunting): get_auth_request con 0x7f4360139640 auth_method 0 2024-09-16T05:18:50.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f435ffff640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 0x7f436009e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:18:50.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f4365185640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:50.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f4365185640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:50.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f4365185640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:50.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f4365185640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 msgr2=0x7f436013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:18:50.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f4365185640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 msgr2=0x7f436013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:18:50.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f4365185640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 0x7f436013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:18:50.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:50.233+0000 7f4365185640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 0x7f436013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:18:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 10 monclient(hunting): tick 2024-09-16T05:18:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:18:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 1 monclient(hunting): continuing hunt 2024-09-16T05:18:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:18:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 msgr2=0x7f436013ba30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:18:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.149:3300/0 conn(0x7f4360139640 0x7f436013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 1 -- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 msgr2=0x7f436009e5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:18:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 1 --2- 172.21.15.114:0/2150656935 >> v2:172.21.15.114:3300/0 conn(0x7f436009e1d0 0x7f436009e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 1 -- 172.21.15.114:0/2150656935 >> v1:172.21.15.114:6789/0 conn(0x7f43600a0ec0 legacy=0x7f436009dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 10 monclient: _add_conns ranks=[2,1] 2024-09-16T05:18:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f434c0058d0 0x7f434c007ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:51.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f434c0058d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:51.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f434c008370 addr v1:172.21.15.149:6789/0 2024-09-16T05:18:51.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 1 -- 172.21.15.114:0/2150656935 --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4360099790 con 0x7f434c008370 2024-09-16T05:18:51.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:51.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 1 -- 172.21.15.114:0/2150656935 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f43600994b0 con 0x7f434c0058d0 2024-09-16T05:18:51.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f435e7fc640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:51.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f4364984640 1 -- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f434c0058d0 msgr2=0x7f434c007ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:18:51.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:51.821+0000 7f4364984640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f434c0058d0 0x7f434c007ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:52.025+0000 7f4364984640 1 -- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f434c0058d0 msgr2=0x7f434c007ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:18:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:52.025+0000 7f4364984640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f434c0058d0 0x7f434c007ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:18:52.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:52.429+0000 7f4364984640 1 -- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f434c0058d0 msgr2=0x7f434c007ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:18:52.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:52.429+0000 7f4364984640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f434c0058d0 0x7f434c007ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:18:53.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 -- 172.21.15.114:0/2150656935 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2939420155 0 0) 0x7f4360099790 con 0x7f434c008370 2024-09-16T05:18:53.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:18:53.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:53.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:53.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient(hunting): my global_id is 64558 2024-09-16T05:18:53.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 -- 172.21.15.114:0/2150656935 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4348003530 con 0x7f434c008370 2024-09-16T05:18:53.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 -- 172.21.15.114:0/2150656935 <== mon.1 v1:172.21.15.149:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1467586244 0 0) 0x7f4348003530 con 0x7f434c008370 2024-09-16T05:18:53.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 -- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f434c0058d0 msgr2=0x7f434c007ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:18:53.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f434c0058d0 0x7f434c007ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:18:53.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 monclient: found mon.noname-c-legacy 2024-09-16T05:18:53.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:53.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.149:6789/0 2024-09-16T05:18:53.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 -- 172.21.15.114:0/2150656935 --> v1:172.21.15.149:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434c00d1b0 con 0x7f434c008370 2024-09-16T05:18:53.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:18:53.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:18:53.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:53.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 -- 172.21.15.114:0/2150656935 <== mon.1 v1:172.21.15.149:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f435400d5d0 con 0x7f434c008370 2024-09-16T05:18:53.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f4366be8640 5 monclient: authenticate success, global_id 64558 2024-09-16T05:18:53.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f4366be8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:53.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T05:18:53.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient: dump: 2024-09-16T05:18:53.127 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.127 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.127 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.127 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.127 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.128 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.128 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.128 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.128 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.128 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] but i'm connected to v1:172.21.15.149:6789/0, reconnecting 2024-09-16T05:18:53.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:53.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 -- 172.21.15.114:0/2150656935 >> v1:172.21.15.149:6789/0 conn(0x7f434c008370 legacy=0x7f434c00a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f4365185640 1 -- 172.21.15.114:0/2150656935 reap_dead start 2024-09-16T05:18:53.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:18:53.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43480039c0 0x7f4348003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient(hunting): picked mon.a con 0x7f43480039c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:53.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4348004810 0x7f4348004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient(hunting): picked mon.b con 0x7f4348004810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:53.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43480052f0 0x7f434800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient(hunting): picked mon.c con 0x7f43480052f0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:53.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 -- 172.21.15.114:0/2150656935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f43600994b0 con 0x7f43480039c0 2024-09-16T05:18:53.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 -- 172.21.15.114:0/2150656935 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f434800c240 con 0x7f4348004810 2024-09-16T05:18:53.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 1 -- 172.21.15.114:0/2150656935 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f434800c440 con 0x7f43480052f0 2024-09-16T05:18:53.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:53.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:18:53.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f4366be8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:53.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f4365185640 1 -- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43480052f0 msgr2=0x7f434800b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:18:53.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f4365185640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43480052f0 0x7f434800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:53.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4348004810 0x7f4348004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 10 monclient(hunting): get_auth_request con 0x7f4348004810 auth_method 0 2024-09-16T05:18:53.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:53.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:53.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:18:53.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:53.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:53.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:18:53.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 10 monclient(hunting): handle_auth_done global_id 64558 payload 293 2024-09-16T05:18:53.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 1 -- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43480052f0 msgr2=0x7f434800b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:18:53.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43480052f0 0x7f434800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 1 -- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43480039c0 msgr2=0x7f4348003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:53.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43480039c0 0x7f4348003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:18:53.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 1 monclient: found mon.b 2024-09-16T05:18:53.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:53.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.149:3300/0 2024-09-16T05:18:53.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 1 -- 172.21.15.114:0/2150656935 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4348003530 con 0x7f4348004810 2024-09-16T05:18:53.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 10 monclient: _finish_auth 0 2024-09-16T05:18:53.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:18:53.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.125+0000 7f435ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:53.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.133+0000 7f435ffff640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4348004810 0x7f4348004c90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4354006a40 tx=0x7f435401d9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c83b57713ee9778a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:53.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.133+0000 7f435f7fe640 1 -- 172.21.15.114:0/2150656935 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435400ddc0 con 0x7f4348004810 2024-09-16T05:18:53.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.133+0000 7f435f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.133+0000 7f435f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:18:53.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.133+0000 7f435f7fe640 10 monclient: dump: 2024-09-16T05:18:53.135 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.135 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.135 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.135 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.135 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.135 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.135 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.135 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.135 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.136 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.133+0000 7f435f7fe640 1 -- 172.21.15.114:0/2150656935 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43540059a0 con 0x7f4348004810 2024-09-16T05:18:53.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.133+0000 7f435f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:53.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.133+0000 7f435f7fe640 1 -- 172.21.15.114:0/2150656935 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354005d90 con 0x7f4348004810 2024-09-16T05:18:53.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.133+0000 7f4366be8640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:53.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.133+0000 7f4366be8640 4 set_mon_vals no callback set 2024-09-16T05:18:53.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.133+0000 7f435f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.133+0000 7f435f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:18:53.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.133+0000 7f435f7fe640 10 monclient: dump: 2024-09-16T05:18:53.136 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.136 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.136 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.136 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.136 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.137 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.137 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.137 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.137 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.137 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 10 monclient: shutdown 2024-09-16T05:18:53.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:53.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 1 -- 172.21.15.114:0/2150656935 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4348004810 msgr2=0x7f4348004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4348004810 0x7f4348004c90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4354006a40 tx=0x7f435401d9d0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 1 -- 172.21.15.114:0/2150656935 shutdown_connections 2024-09-16T05:18:53.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43480052f0 0x7f434800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4348004810 0x7f4348004c90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 1 --2- 172.21.15.114:0/2150656935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43480039c0 0x7f4348003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 1 -- 172.21.15.114:0/2150656935 >> 172.21.15.114:0/2150656935 conn(0x7f43600a1fb0 msgr2=0x7f43600a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:53.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 1 -- 172.21.15.114:0/2150656935 shutdown_connections 2024-09-16T05:18:53.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 1 -- 172.21.15.114:0/2150656935 wait complete. 2024-09-16T05:18:53.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 10 monclient: build_initial_monmap 2024-09-16T05:18:53.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 10 monclient: monmap: 2024-09-16T05:18:53.139 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:53.139 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:53.139 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:53.139 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:53.139 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:53.139 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.139 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:53.139 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:53.140 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:53.140 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 1 librados: starting msgr at 2024-09-16T05:18:53.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 1 librados: starting objecter 2024-09-16T05:18:53.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 1 Processor -- start 2024-09-16T05:18:53.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 1 -- start start 2024-09-16T05:18:53.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 1 librados: setting wanted keys 2024-09-16T05:18:53.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 1 librados: calling monclient init 2024-09-16T05:18:53.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.137+0000 7f4366be8640 10 monclient: init 2024-09-16T05:18:53.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:53.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:18:53.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43480039c0 0x7f43601337c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient(hunting): picked mon.noname-c con 0x7f43480039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:53.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4348004810 0x7f4360133d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient(hunting): picked mon.noname-a con 0x7f4348004810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:53.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f43480052f0 0x7f436012a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4364984640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43480039c0 msgr2=0x7f43601337c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:18:53.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4364984640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43480039c0 0x7f43601337c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:53.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient(hunting): picked mon.noname-b con 0x7f43480052f0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:53.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f436009d1c0 con 0x7f4348004810 2024-09-16T05:18:53.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f436009e9c0 con 0x7f43480052f0 2024-09-16T05:18:53.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f436009d340 con 0x7f43480039c0 2024-09-16T05:18:53.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:53.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient(hunting): authenticate will time out at 1617.772615s 2024-09-16T05:18:53.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4348004810 0x7f4360133d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4348004810 0x7f4360133d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48070/0 (socket says 172.21.15.114:48070) 2024-09-16T05:18:53.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 1 -- 172.21.15.114:0/4225341160 learned_addr learned my addr 172.21.15.114:0/4225341160 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:53.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 10 monclient(hunting): get_auth_request con 0x7f4348004810 auth_method 0 2024-09-16T05:18:53.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:53.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:53.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:53.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:53.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:53.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:53.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4365185640 1 --2- 172.21.15.114:0/4225341160 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f43480052f0 0x7f436012a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4365185640 10 monclient(hunting): get_auth_request con 0x7f43480052f0 auth_method 0 2024-09-16T05:18:53.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4365185640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:53.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4365185640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:53.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4365185640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:53.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 10 monclient(hunting): handle_auth_done global_id 74484 payload 824 2024-09-16T05:18:53.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 1 -- 172.21.15.114:0/4225341160 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43480039c0 msgr2=0x7f43601337c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:18:53.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 1 --2- 172.21.15.114:0/4225341160 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43480039c0 0x7f43601337c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 1 -- 172.21.15.114:0/4225341160 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f43480052f0 msgr2=0x7f436012a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 1 --2- 172.21.15.114:0/4225341160 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f43480052f0 0x7f436012a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:18:53.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 1 monclient: found mon.noname-a 2024-09-16T05:18:53.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:53.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:53.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 1 -- 172.21.15.114:0/4225341160 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4360073440 con 0x7f4348004810 2024-09-16T05:18:53.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435ffff640 1 --2- 172.21.15.114:0/4225341160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4348004810 0x7f4360133d00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f435401d800 tx=0x7f4348005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:53.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 10 client.?.objecter ms_handle_connect 0x7f4348004810 2024-09-16T05:18:53.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:53.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 1 -- 172.21.15.114:0/4225341160 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435400d7b0 con 0x7f4348004810 2024-09-16T05:18:53.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:53.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 10 monclient: dump: 2024-09-16T05:18:53.149 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.149 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.149 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.149 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.149 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.149 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.149 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.149 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.149 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.149 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 10 monclient: _finish_auth 0 2024-09-16T05:18:53.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:18:53.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:53.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 5 monclient: authenticate success, global_id 74484 2024-09-16T05:18:53.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient: _renew_subs 2024-09-16T05:18:53.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:53.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 1 -- 172.21.15.114:0/4225341160 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4360134240 con 0x7f4348004810 2024-09-16T05:18:53.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 1 -- 172.21.15.114:0/4225341160 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4354000e50 con 0x7f4348004810 2024-09-16T05:18:53.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:53.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 1 -- 172.21.15.114:0/4225341160 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435400d3d0 con 0x7f4348004810 2024-09-16T05:18:53.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:53.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 10 monclient: dump: 2024-09-16T05:18:53.150 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.150 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.150 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.151 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.151 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.151 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.151 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.151 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.151 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.151 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 client.74484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:53.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient: _renew_subs 2024-09-16T05:18:53.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:53.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 1 -- 172.21.15.114:0/4225341160 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43600a3a90 con 0x7f4348004810 2024-09-16T05:18:53.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 1 librados: init done 2024-09-16T05:18:53.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435effd640 4 set_mon_vals no callback set 2024-09-16T05:18:53.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 1 -- 172.21.15.114:0/4225341160 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f43540a7bb0 con 0x7f4348004810 2024-09-16T05:18:53.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:53.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:53.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 1 --2- 172.21.15.114:0/4225341160 >> v2:172.21.15.149:6812/1185531135 conn(0x7f434c03c360 0x7f434c03e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:53.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4366be8640 1 -- 172.21.15.114:0/4225341160 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f436009d670 con 0x7f4348004810 2024-09-16T05:18:53.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 1 -- 172.21.15.114:0/4225341160 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(432..432 src has 1..432) ==== 5462+0+0 (secure 0 0 0) 0x7f43540e3710 con 0x7f4348004810 2024-09-16T05:18:53.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 10 client.74484.objecter ms_dispatch 0x7f4360000ba0 osd_map(432..432 src has 1..432) 2024-09-16T05:18:53.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 3 client.74484.objecter handle_osd_map got epochs [432,432] > 0 2024-09-16T05:18:53.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 3 client.74484.objecter handle_osd_map decoding full epoch 432 2024-09-16T05:18:53.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f435d7fa640 20 client.74484.objecter dump_active .. 0 homeless 2024-09-16T05:18:53.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4364984640 1 --2- 172.21.15.114:0/4225341160 >> v2:172.21.15.149:6812/1185531135 conn(0x7f434c03c360 0x7f434c03e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.141+0000 7f4364984640 10 monclient: get_auth_request con 0x7f434c03c360 auth_method 0 2024-09-16T05:18:53.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.145+0000 7f435d7fa640 10 client.74484.objecter ms_handle_connect 0x7f434c03c360 2024-09-16T05:18:53.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.145+0000 7f4364984640 1 --2- 172.21.15.114:0/4225341160 >> v2:172.21.15.149:6812/1185531135 conn(0x7f434c03c360 0x7f434c03e820 secure :-1 s=READY pgs=3550 cs=0 l=1 rev1=1 crypto rx=0x7f4350007fd0 tx=0x7f4350007920 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:53.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.149+0000 7f435d7fa640 1 -- 172.21.15.114:0/4225341160 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f43540176b0 con 0x7f4348004810 2024-09-16T05:18:53.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.149+0000 7f435d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:53.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.149+0000 7f435d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:53.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.389+0000 7f4366be8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:18:53.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.389+0000 7f4366be8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:53.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.389+0000 7f4366be8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:53.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.389+0000 7f4366be8640 1 -- 172.21.15.114:0/4225341160 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f436012ba40 con 0x7f4348004810 2024-09-16T05:18:53.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.389+0000 7f435d7fa640 1 -- 172.21.15.114:0/4225341160 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f43540ca020 con 0x7f4348004810 2024-09-16T05:18:53.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.389+0000 7f435d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:18:53.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.389+0000 7f435d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:53.390 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:18:53.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 10 librados: watch_flush enter 2024-09-16T05:18:53.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 10 librados: watch_flush exit 2024-09-16T05:18:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 20 client.74484.objecter shutdown clearing up homeless session... 2024-09-16T05:18:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 10 client.74484.objecter successfully canceled tick 2024-09-16T05:18:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 1 -- 172.21.15.114:0/4225341160 >> v2:172.21.15.149:6812/1185531135 conn(0x7f434c03c360 msgr2=0x7f434c03e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 1 --2- 172.21.15.114:0/4225341160 >> v2:172.21.15.149:6812/1185531135 conn(0x7f434c03c360 0x7f434c03e820 secure :-1 s=READY pgs=3550 cs=0 l=1 rev1=1 crypto rx=0x7f4350007fd0 tx=0x7f4350007920 comp rx=0 tx=0).stop 2024-09-16T05:18:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 10 monclient: shutdown 2024-09-16T05:18:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 1 -- 172.21.15.114:0/4225341160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4348004810 msgr2=0x7f4360133d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 1 --2- 172.21.15.114:0/4225341160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4348004810 0x7f4360133d00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f435401d800 tx=0x7f4348005840 comp rx=0 tx=0).stop 2024-09-16T05:18:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 1 -- 172.21.15.114:0/4225341160 shutdown_connections 2024-09-16T05:18:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 1 --2- 172.21.15.114:0/4225341160 >> v2:172.21.15.149:6812/1185531135 conn(0x7f434c03c360 0x7f434c03e820 unknown :-1 s=CLOSED pgs=3550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 1 --2- 172.21.15.114:0/4225341160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4348004810 0x7f4360133d00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 1 --2- 172.21.15.114:0/4225341160 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f43480052f0 0x7f436012a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 1 --2- 172.21.15.114:0/4225341160 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43480039c0 0x7f43601337c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 1 -- 172.21.15.114:0/4225341160 >> 172.21.15.114:0/4225341160 conn(0x7f43600a1fb0 msgr2=0x7f43601be730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:53.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 1 -- 172.21.15.114:0/4225341160 shutdown_connections 2024-09-16T05:18:53.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 1 -- 172.21.15.114:0/4225341160 wait complete. 2024-09-16T05:18:53.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.393+0000 7f4366be8640 1 librados: shutdown 2024-09-16T05:18:53.409 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:18:53.409 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T05:18:53.409 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:18:53.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 10 monclient: get_monmap_and_config 2024-09-16T05:18:53.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 10 monclient: build_initial_monmap 2024-09-16T05:18:53.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 10 monclient: monmap: 2024-09-16T05:18:53.528 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:53.528 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.528 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:53.532891+0000 2024-09-16T05:18:53.528 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:53.532891+0000 2024-09-16T05:18:53.529 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:53.529 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.529 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:53.529 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:53.529 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:53.529 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:53.529 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:53.530 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 1 Processor -- start 2024-09-16T05:18:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 1 -- start start 2024-09-16T05:18:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 10 monclient: init 2024-09-16T05:18:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-16T05:18:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4d00a0ec0 0x7fd4d009dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 10 monclient(hunting): picked mon.noname-b con 0x7fd4d00a0ec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd4d009e000 0x7fd4d009e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 10 monclient(hunting): picked mon.noname-c con 0x7fd4d009e000 addr v2:172.21.15.149:3300/0 2024-09-16T05:18:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd4d0138630 0x7fd4d013aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 10 monclient(hunting): picked mon.noname-a con 0x7fd4d0138630 addr v2:172.21.15.114:3300/0 2024-09-16T05:18:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd4d00994b0 con 0x7fd4d0138630 2024-09-16T05:18:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fd4d009a760 con 0x7fd4d009e000 2024-09-16T05:18:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d00a1870 con 0x7fd4d00a0ec0 2024-09-16T05:18:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4d58be640 10 monclient(hunting): authenticate will time out at 1618.162015s 2024-09-16T05:18:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4ceffd640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4d00a0ec0 msgr2=0x7fd4d009dac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:18:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4ceffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4d00a0ec0 0x7fd4d009dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:53.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd4d0138630 0x7fd4d013aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd4d0138630 0x7fd4d013aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48078/0 (socket says 172.21.15.114:48078) 2024-09-16T05:18:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 1 -- 172.21.15.114:0/542230454 learned_addr learned my addr 172.21.15.114:0/542230454 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 10 monclient(hunting): get_auth_request con 0x7fd4d0138630 auth_method 0 2024-09-16T05:18:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4ce7fc640 1 --2- 172.21.15.114:0/542230454 >> v2:172.21.15.149:3300/0 conn(0x7fd4d009e000 0x7fd4d009e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4ce7fc640 10 monclient(hunting): get_auth_request con 0x7fd4d009e000 auth_method 0 2024-09-16T05:18:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:53.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 10 monclient(hunting): handle_auth_done global_id 74487 payload 274 2024-09-16T05:18:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 1 -- 172.21.15.114:0/542230454 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4d00a0ec0 msgr2=0x7fd4d009dac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:18:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 1 --2- 172.21.15.114:0/542230454 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4d00a0ec0 0x7fd4d009dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 1 -- 172.21.15.114:0/542230454 >> v2:172.21.15.149:3300/0 conn(0x7fd4d009e000 msgr2=0x7fd4d009e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:18:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 1 --2- 172.21.15.114:0/542230454 >> v2:172.21.15.149:3300/0 conn(0x7fd4d009e000 0x7fd4d009e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:18:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 1 monclient: found mon.noname-a 2024-09-16T05:18:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:53.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.529+0000 7fd4cf7fe640 1 -- 172.21.15.114:0/542230454 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d009e920 con 0x7fd4d0138630 2024-09-16T05:18:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cf7fe640 1 --2- 172.21.15.114:0/542230454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0138630 0x7fd4d013aa20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd4d006cf30 tx=0x7fd4c801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca83ed7a0ce87bd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cdffb640 1 -- 172.21.15.114:0/542230454 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c80a5760 con 0x7fd4d0138630 2024-09-16T05:18:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cdffb640 10 monclient: dump: 2024-09-16T05:18:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.546 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.547 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cdffb640 10 monclient: _finish_auth 0 2024-09-16T05:18:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cdffb640 10 monclient: _check_auth_tickets 2024-09-16T05:18:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cdffb640 1 -- 172.21.15.114:0/542230454 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4c80a5900 con 0x7fd4d0138630 2024-09-16T05:18:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 5 monclient: authenticate success, global_id 74487 2024-09-16T05:18:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 4 set_mon_vals no callback set 2024-09-16T05:18:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cdffb640 1 -- 172.21.15.114:0/542230454 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c80a5cd0 con 0x7fd4d0138630 2024-09-16T05:18:53.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:53.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cdffb640 10 monclient: dump: 2024-09-16T05:18:53.548 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.548 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.548 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.548 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.548 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.548 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.548 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.548 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.548 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.548 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 10 monclient: shutdown 2024-09-16T05:18:53.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:53.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 -- 172.21.15.114:0/542230454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0138630 msgr2=0x7fd4d013aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 --2- 172.21.15.114:0/542230454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0138630 0x7fd4d013aa20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd4d006cf30 tx=0x7fd4c801c9f0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 -- 172.21.15.114:0/542230454 shutdown_connections 2024-09-16T05:18:53.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 --2- 172.21.15.114:0/542230454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0138630 0x7fd4d013aa20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 --2- 172.21.15.114:0/542230454 >> v2:172.21.15.149:3300/0 conn(0x7fd4d009e000 0x7fd4d009e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 --2- 172.21.15.114:0/542230454 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4d00a0ec0 0x7fd4d009dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 -- 172.21.15.114:0/542230454 >> 172.21.15.114:0/542230454 conn(0x7fd4d00a1fb0 msgr2=0x7fd4d00a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:53.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 -- 172.21.15.114:0/542230454 shutdown_connections 2024-09-16T05:18:53.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 -- 172.21.15.114:0/542230454 wait complete. 2024-09-16T05:18:53.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 10 monclient: build_initial_monmap 2024-09-16T05:18:53.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 10 monclient: monmap: 2024-09-16T05:18:53.550 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:53.550 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:53.550 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:53.550 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:53.550 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:53.550 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.550 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:53.550 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:53.550 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:53.550 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 librados: starting msgr at 2024-09-16T05:18:53.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 librados: starting objecter 2024-09-16T05:18:53.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 Processor -- start 2024-09-16T05:18:53.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 -- start start 2024-09-16T05:18:53.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 librados: setting wanted keys 2024-09-16T05:18:53.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 librados: calling monclient init 2024-09-16T05:18:53.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 10 monclient: init 2024-09-16T05:18:53.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:53.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:18:53.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4d009e000 0x7fd4d0137640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 10 monclient(hunting): picked mon.noname-b con 0x7fd4d009e000 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:53.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d00a0ec0 0x7fd4d0137b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 10 monclient(hunting): picked mon.noname-a con 0x7fd4d00a0ec0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:53.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4d0138630 0x7fd4d01380c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 10 monclient(hunting): picked mon.noname-c con 0x7fd4d0138630 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:53.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d009c3b0 con 0x7fd4d00a0ec0 2024-09-16T05:18:53.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d009c230 con 0x7fd4d009e000 2024-09-16T05:18:53.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d009c9b0 con 0x7fd4d0138630 2024-09-16T05:18:53.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:53.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4d58be640 10 monclient(hunting): authenticate will time out at 1618.167882s 2024-09-16T05:18:53.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cf7fe640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4d0138630 msgr2=0x7fd4d01380c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:18:53.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4cf7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4d0138630 0x7fd4d01380c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:53.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4ce7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d00a0ec0 0x7fd4d0137b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4ce7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d00a0ec0 0x7fd4d0137b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48084/0 (socket says 172.21.15.114:48084) 2024-09-16T05:18:53.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.533+0000 7fd4ce7fc640 1 -- 172.21.15.114:0/1205150565 learned_addr learned my addr 172.21.15.114:0/1205150565 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:53.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 10 monclient(hunting): get_auth_request con 0x7fd4d00a0ec0 auth_method 0 2024-09-16T05:18:53.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:53.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:53.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:53.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:53.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:53.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:53.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ceffd640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4d009e000 0x7fd4d0137640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 10 monclient(hunting): handle_auth_done global_id 74490 payload 824 2024-09-16T05:18:53.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 1 -- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4d0138630 msgr2=0x7fd4d01380c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:18:53.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4d0138630 0x7fd4d01380c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 1 -- 172.21.15.114:0/1205150565 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4d009e000 msgr2=0x7fd4d0137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4d009e000 0x7fd4d0137640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:18:53.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 1 monclient: found mon.noname-a 2024-09-16T05:18:53.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:53.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:53.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 1 -- 172.21.15.114:0/1205150565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d009c6a0 con 0x7fd4d00a0ec0 2024-09-16T05:18:53.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ceffd640 10 monclient: get_auth_request con 0x7fd4d009e000 auth_method 0 2024-09-16T05:18:53.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ceffd640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4d009e000 0x7fd4d0137640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:18:53.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ce7fc640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d00a0ec0 0x7fd4d0137b80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd4c400d7d0 tx=0x7fd4c400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:53.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 client.?.objecter ms_handle_connect 0x7fd4d00a0ec0 2024-09-16T05:18:53.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:53.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 1 -- 172.21.15.114:0/1205150565 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c400b6c0 con 0x7fd4d00a0ec0 2024-09-16T05:18:53.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:53.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 monclient: dump: 2024-09-16T05:18:53.555 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.555 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.555 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.555 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.555 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.555 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.555 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.555 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.555 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.555 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 monclient: _finish_auth 0 2024-09-16T05:18:53.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:18:53.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:53.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 1 -- 172.21.15.114:0/1205150565 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4c4019070 con 0x7fd4d00a0ec0 2024-09-16T05:18:53.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4d58be640 5 monclient: authenticate success, global_id 74490 2024-09-16T05:18:53.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4d58be640 10 monclient: _renew_subs 2024-09-16T05:18:53.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4d58be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:53.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4d58be640 1 -- 172.21.15.114:0/1205150565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4d0094e50 con 0x7fd4d00a0ec0 2024-09-16T05:18:53.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:53.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 1 -- 172.21.15.114:0/1205150565 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c400baa0 con 0x7fd4d00a0ec0 2024-09-16T05:18:53.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:53.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 monclient: dump: 2024-09-16T05:18:53.557 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.557 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.557 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.557 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.557 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.557 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.557 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.557 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.557 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.557 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4cd7fa640 4 set_mon_vals no callback set 2024-09-16T05:18:53.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4d58be640 10 client.74490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:53.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4d58be640 10 monclient: _renew_subs 2024-09-16T05:18:53.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4d58be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:53.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4d58be640 1 -- 172.21.15.114:0/1205150565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4d00942a0 con 0x7fd4d00a0ec0 2024-09-16T05:18:53.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4d58be640 1 librados: init done 2024-09-16T05:18:53.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4d58be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:53.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4d58be640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d00923b0 0x7fd4d002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4d58be640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4d58be640 1 -- 172.21.15.114:0/1205150565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d0069390 con 0x7fd4d00923b0 2024-09-16T05:18:53.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 1 -- 172.21.15.114:0/1205150565 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd4c400bc40 con 0x7fd4d00a0ec0 2024-09-16T05:18:53.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ceffd640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d00923b0 0x7fd4d002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ceffd640 10 monclient: get_auth_request con 0x7fd4d00923b0 auth_method 0 2024-09-16T05:18:53.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:53.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:53.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:53.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:53.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:53.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 1 --2- 172.21.15.114:0/1205150565 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd4bc0377c0 0x7fd4bc039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:53.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 1 -- 172.21.15.114:0/1205150565 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(432..432 src has 1..432) ==== 5462+0+0 (secure 0 0 0) 0x7fd4c40492d0 con 0x7fd4d00a0ec0 2024-09-16T05:18:53.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 client.74490.objecter ms_dispatch 0x7fd4d0000ba0 osd_map(432..432 src has 1..432) 2024-09-16T05:18:53.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 3 client.74490.objecter handle_osd_map got epochs [432,432] > 0 2024-09-16T05:18:53.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 3 client.74490.objecter handle_osd_map decoding full epoch 432 2024-09-16T05:18:53.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ceffd640 10 monclient(hunting): handle_auth_done global_id 74493 payload 824 2024-09-16T05:18:53.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ceffd640 1 -- 172.21.15.114:0/1205150565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd4d009e3e0 con 0x7fd4d00923b0 2024-09-16T05:18:53.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 20 client.74490.objecter dump_active .. 0 homeless 2024-09-16T05:18:53.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 client.74490.objecter ms_handle_connect 0x7fd4d00923b0 2024-09-16T05:18:53.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 client.74490.objecter resend_mon_ops 2024-09-16T05:18:53.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4ceffd640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d00923b0 0x7fd4d002c650 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000c6f0 tx=0x7fd4c000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:53.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4cf7fe640 1 --2- 172.21.15.114:0/1205150565 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd4bc0377c0 0x7fd4bc039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4cf7fe640 10 monclient: get_auth_request con 0x7fd4bc0377c0 auth_method 0 2024-09-16T05:18:53.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 1 -- 172.21.15.114:0/1205150565 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0005c50 con 0x7fd4d00923b0 2024-09-16T05:18:53.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:53.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 monclient: dump: 2024-09-16T05:18:53.560 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.560 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.560 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.560 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.560 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.560 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.560 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.561 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.561 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.561 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 1 -- 172.21.15.114:0/1205150565 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd4c0005c50 con 0x7fd4d00923b0 2024-09-16T05:18:53.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:53.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:53.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 1 -- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d00923b0 msgr2=0x7fd4d002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d00923b0 0x7fd4d002c650 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000c6f0 tx=0x7fd4c000cbb0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 1 -- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d00923b0 msgr2=0x7fd4d002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:53.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d00923b0 0x7fd4d002c650 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000c6f0 tx=0x7fd4c000cbb0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4b7fff640 10 client.74490.objecter ms_handle_connect 0x7fd4bc0377c0 2024-09-16T05:18:53.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.537+0000 7fd4cf7fe640 1 --2- 172.21.15.114:0/1205150565 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd4bc0377c0 0x7fd4bc039c80 secure :-1 s=READY pgs=3551 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80ae000 tx=0x7fd4c80a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:53.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:18:53.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0092790 0x7fd4d002cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 1 -- 172.21.15.114:0/1205150565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d008e710 con 0x7fd4d0092790 2024-09-16T05:18:53.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4ceffd640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0092790 0x7fd4d002cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4ceffd640 10 monclient: get_auth_request con 0x7fd4d0092790 auth_method 0 2024-09-16T05:18:53.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:53.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:53.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:53.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:53.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:53.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:53.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4ceffd640 10 monclient(hunting): handle_auth_done global_id 74496 payload 824 2024-09-16T05:18:53.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4ceffd640 1 -- 172.21.15.114:0/1205150565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd4d00686f0 con 0x7fd4d0092790 2024-09-16T05:18:53.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4ceffd640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0092790 0x7fd4d002cb90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0000c10 tx=0x7fd4c00059e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:53.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4b7fff640 10 client.74490.objecter ms_handle_connect 0x7fd4d0092790 2024-09-16T05:18:53.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4b7fff640 10 client.74490.objecter resend_mon_ops 2024-09-16T05:18:53.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4b7fff640 1 -- 172.21.15.114:0/1205150565 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0005e00 con 0x7fd4d0092790 2024-09-16T05:18:53.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:53.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4b7fff640 10 monclient: dump: 2024-09-16T05:18:53.565 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.565 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.565 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.565 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.565 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.565 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.565 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.565 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.566 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.566 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4b7fff640 1 -- 172.21.15.114:0/1205150565 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fd4c0005e00 con 0x7fd4d0092790 2024-09-16T05:18:53.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4b7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:18:53.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4b7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:53.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4b7fff640 1 -- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0092790 msgr2=0x7fd4d002cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4b7fff640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0092790 0x7fd4d002cb90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0000c10 tx=0x7fd4c00059e0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4b7fff640 1 -- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0092790 msgr2=0x7fd4d002cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:53.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4b7fff640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0092790 0x7fd4d002cb90 secure :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0000c10 tx=0x7fd4c00059e0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.566 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:18:53.567 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:18:53.567 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:18:53.567 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:18:53.567 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 54, 2024-09-16T05:18:53.567 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 292600, 2024-09-16T05:18:53.567 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:18:53.567 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:18:53.567 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-16T05:18:53.567 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:53.567 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:18:53.567 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:53.567 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:18:53.567 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:18:53.568 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:53.568 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:53.568 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:53.568 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:53.568 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:53.568 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:53.568 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:53.568 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:53.568 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:53.568 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:53.568 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:53.568 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:53.569 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:18:53.569 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:18:53.569 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:53.569 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:53.569 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:53.569 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:53.569 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:53.569 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:53.569 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:53.569 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:53.569 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:53.569 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:53.569 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:53.569 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:53.570 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.570 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:18:53.570 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:18:53.570 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:18:53.570 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:18:53.570 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:18:53.570 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:18:53.570 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:53.570 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:53.570 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:18:53.570 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:18:53.570 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:18:53.571 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:18:53.571 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:18:53.571 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:18:53.571 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:18:53.571 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:53.571 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:18:53.571 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:53.571 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:53.571 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:53.571 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:53.571 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:53.571 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:53.571 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:53.572 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:53.572 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:53.572 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:53.572 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:53.572 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:53.572 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:18:53.572 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.572 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:18:53.572 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.572 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:18:53.572 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:18:53.572 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:53.572 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:53.573 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.573 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:53.573 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:18:53.573 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:53.573 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.573 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.573 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:53.573 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:18:53.573 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:53.573 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:53.573 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:53.573 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.573 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:18:53.573 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:18:53.574 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:53.574 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:53.574 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:53.574 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.574 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.574 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:18:53.574 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:18:53.574 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:53.574 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:53.574 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.574 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:53.574 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:18:53.574 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:53.574 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.575 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.575 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:53.575 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:18:53.575 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:53.575 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:53.575 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:53.575 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.575 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:18:53.575 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:18:53.575 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:53.575 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:53.575 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:53.575 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:18:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:18:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:18:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:18:53.576 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:18:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:18:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:18:53.577 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:18:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:18:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:18:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:18:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:53.578 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:18:53.579 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.579 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:53.579 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:53.579 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:18:53.579 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:53.579 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:53.579 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.579 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:18:53.579 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:18:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 10 librados: watch_flush enter 2024-09-16T05:18:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 10 librados: watch_flush exit 2024-09-16T05:18:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 20 client.74490.objecter shutdown clearing up homeless session... 2024-09-16T05:18:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 10 client.74490.objecter successfully canceled tick 2024-09-16T05:18:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 1 -- 172.21.15.114:0/1205150565 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd4bc0377c0 msgr2=0x7fd4bc039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 1 --2- 172.21.15.114:0/1205150565 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd4bc0377c0 0x7fd4bc039c80 secure :-1 s=READY pgs=3551 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80ae000 tx=0x7fd4c80a7070 comp rx=0 tx=0).stop 2024-09-16T05:18:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 10 monclient: shutdown 2024-09-16T05:18:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 1 -- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d00a0ec0 msgr2=0x7fd4d0137b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 1 --2- 172.21.15.114:0/1205150565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d00a0ec0 0x7fd4d0137b80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd4c400d7d0 tx=0x7fd4c400dc90 comp rx=0 tx=0).stop 2024-09-16T05:18:53.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4cf7fe640 1 -- 172.21.15.114:0/1205150565 reap_dead start 2024-09-16T05:18:53.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4cf7fe640 1 -- 172.21.15.114:0/1205150565 reap_dead start 2024-09-16T05:18:53.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 1 -- 172.21.15.114:0/1205150565 shutdown_connections 2024-09-16T05:18:53.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 1 -- 172.21.15.114:0/1205150565 >> 172.21.15.114:0/1205150565 conn(0x7fd4d00a1fb0 msgr2=0x7fd4d00a4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:53.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 1 -- 172.21.15.114:0/1205150565 shutdown_connections 2024-09-16T05:18:53.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 1 -- 172.21.15.114:0/1205150565 wait complete. 2024-09-16T05:18:53.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.545+0000 7fd4d58be640 1 librados: shutdown 2024-09-16T05:18:53.582 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:18:53.582 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T05:18:53.583 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:18:53.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 10 monclient: get_monmap_and_config 2024-09-16T05:18:53.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 10 monclient: build_initial_monmap 2024-09-16T05:18:53.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 10 monclient: monmap: 2024-09-16T05:18:53.699 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:53.699 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.700 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:18:53.704578+0000 2024-09-16T05:18:53.700 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:18:53.704578+0000 2024-09-16T05:18:53.700 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:53.700 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.700 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:18:53.700 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:18:53.700 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:18:53.700 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:18:53.700 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:18:53.700 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 1 Processor -- start 2024-09-16T05:18:53.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 1 -- start start 2024-09-16T05:18:53.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 10 monclient: init 2024-09-16T05:18:53.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:53.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-16T05:18:53.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f24b809b6c0 addr v1:172.21.15.149:6789/0 2024-09-16T05:18:53.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24b80a15d0 0x7f24b813a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 10 monclient(hunting): picked mon.noname-b con 0x7f24b80a15d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:53.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f24b80a19e0 0x7f24b813cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 10 monclient(hunting): picked mon.noname-c con 0x7f24b80a19e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:18:53.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24b809d4e0 con 0x7f24b809b6c0 2024-09-16T05:18:53.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f24b809e390 con 0x7f24b80a19e0 2024-09-16T05:18:53.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f24b809ece0 con 0x7f24b80a15d0 2024-09-16T05:18:53.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:53.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24bcb58640 10 monclient(hunting): authenticate will time out at 1618.333933s 2024-09-16T05:18:53.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b5d9b640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24b80a15d0 msgr2=0x7f24b813a650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:18:53.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b5d9b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24b80a15d0 0x7f24b813a650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:53.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f24b80a19e0 0x7f24b813cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f24b80a19e0 0x7f24b813cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.114:39750/0 (socket says 172.21.15.114:39750) 2024-09-16T05:18:53.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b659c640 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f24b809b6c0 0x7f24b80a0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.114:38518/0 (socket says 172.21.15.114:38518) 2024-09-16T05:18:53.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 1 -- 172.21.15.114:0/471917093 learned_addr learned my addr 172.21.15.114:0/471917093 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:53.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 10 monclient(hunting): get_auth_request con 0x7f24b80a19e0 auth_method 0 2024-09-16T05:18:53.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:53.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:53.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:53.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:53.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:53.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:53.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 10 monclient(hunting): handle_auth_done global_id 64567 payload 274 2024-09-16T05:18:53.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 1 -- 172.21.15.114:0/471917093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24b80a15d0 msgr2=0x7f24b813a650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:18:53.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 1 --2- 172.21.15.114:0/471917093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24b80a15d0 0x7f24b813a650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 1 -- 172.21.15.114:0/471917093 >> v1:172.21.15.149:6789/0 conn(0x7f24b809b6c0 legacy=0x7f24b80a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 10 monclient: _finish_hunting 0 2024-09-16T05:18:53.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 1 monclient: found mon.noname-c 2024-09-16T05:18:53.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:53.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:18:53.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b559a640 1 -- 172.21.15.114:0/471917093 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1435573615 0 0) 0x7f24b809d4e0 con 0x7f24b809b6c0 2024-09-16T05:18:53.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b6d9d640 1 -- 172.21.15.114:0/471917093 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b809f430 con 0x7f24b80a19e0 2024-09-16T05:18:53.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.701+0000 7f24b559a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:18:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24b6d9d640 1 --2- 172.21.15.114:0/471917093 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b80a19e0 0x7f24b813cba0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009870 tx=0x7f24a801c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5380357dbe704cac server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24b559a640 1 -- 172.21.15.114:0/471917093 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a80b0070 con 0x7f24b80a19e0 2024-09-16T05:18:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24b559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:18:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24b559a640 10 monclient: dump: 2024-09-16T05:18:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.714 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24b559a640 10 monclient: _finish_auth 0 2024-09-16T05:18:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24b559a640 10 monclient: _check_auth_tickets 2024-09-16T05:18:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24b559a640 1 -- 172.21.15.114:0/471917093 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24a80ab600 con 0x7f24b80a19e0 2024-09-16T05:18:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24bcb58640 5 monclient: authenticate success, global_id 64567 2024-09-16T05:18:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24bcb58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:18:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24b559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24bcb58640 10 monclient: get_monmap_and_config success 2024-09-16T05:18:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24bcb58640 4 set_mon_vals no callback set 2024-09-16T05:18:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24b559a640 1 -- 172.21.15.114:0/471917093 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a80ab9d0 con 0x7f24b80a19e0 2024-09-16T05:18:53.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24b559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:18:53.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24b559a640 10 monclient: dump: 2024-09-16T05:18:53.715 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.715 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.715 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.715 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.715 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.715 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.715 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.715 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.715 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.715 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24bcb58640 10 monclient: shutdown 2024-09-16T05:18:53.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24bcb58640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:53.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24bcb58640 1 -- 172.21.15.114:0/471917093 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b80a19e0 msgr2=0x7f24b813cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24bcb58640 1 --2- 172.21.15.114:0/471917093 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b80a19e0 0x7f24b813cba0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009870 tx=0x7f24a801c440 comp rx=0 tx=0).stop 2024-09-16T05:18:53.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.705+0000 7f24bcb58640 1 -- 172.21.15.114:0/471917093 shutdown_connections 2024-09-16T05:18:53.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 --2- 172.21.15.114:0/471917093 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b80a19e0 0x7f24b813cba0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 --2- 172.21.15.114:0/471917093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24b80a15d0 0x7f24b813a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 -- 172.21.15.114:0/471917093 >> 172.21.15.114:0/471917093 conn(0x7f24b80a1fb0 msgr2=0x7f24b80a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:53.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 -- 172.21.15.114:0/471917093 shutdown_connections 2024-09-16T05:18:53.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 -- 172.21.15.114:0/471917093 wait complete. 2024-09-16T05:18:53.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient: build_initial_monmap 2024-09-16T05:18:53.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient: monmap: 2024-09-16T05:18:53.716 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:18:53.716 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:18:53.717 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:18:53.717 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:18:53.717 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:18:53.717 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.717 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:18:53.717 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:18:53.717 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:18:53.717 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 librados: starting msgr at 2024-09-16T05:18:53.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 librados: starting objecter 2024-09-16T05:18:53.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 Processor -- start 2024-09-16T05:18:53.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 -- start start 2024-09-16T05:18:53.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 librados: setting wanted keys 2024-09-16T05:18:53.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 librados: calling monclient init 2024-09-16T05:18:53.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient: init 2024-09-16T05:18:53.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient: _reopen_session rank -1 2024-09-16T05:18:53.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:18:53.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24b809b6c0 0x7f24b813b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient(hunting): picked mon.noname-c con 0x7f24b809b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:18:53.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b80a15d0 0x7f24b813b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient(hunting): picked mon.noname-a con 0x7f24b80a15d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:18:53.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b80a19e0 0x7f24b813baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient(hunting): picked mon.noname-b con 0x7f24b80a19e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:18:53.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f24b813f900 con 0x7f24b80a15d0 2024-09-16T05:18:53.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f24b809cad0 con 0x7f24b80a19e0 2024-09-16T05:18:53.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f24b813fa80 con 0x7f24b809b6c0 2024-09-16T05:18:53.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient(hunting): _renew_subs 2024-09-16T05:18:53.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient(hunting): authenticate will time out at 1618.341730s 2024-09-16T05:18:53.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b659c640 1 -- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24b809b6c0 msgr2=0x7f24b813b020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3301/0 2024-09-16T05:18:53.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b659c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24b809b6c0 0x7f24b813b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:18:53.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b80a15d0 0x7f24b813b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b80a15d0 0x7f24b813b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48104/0 (socket says 172.21.15.114:48104) 2024-09-16T05:18:53.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 1 -- 172.21.15.114:0/2990208856 learned_addr learned my addr 172.21.15.114:0/2990208856 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:18:53.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 10 monclient(hunting): get_auth_request con 0x7f24b80a15d0 auth_method 0 2024-09-16T05:18:53.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:53.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:53.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:53.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:53.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:53.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:53.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b6d9d640 1 --2- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b80a19e0 0x7f24b813baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b6d9d640 10 monclient(hunting): get_auth_request con 0x7f24b80a19e0 auth_method 0 2024-09-16T05:18:53.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:53.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:53.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:53.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 10 monclient(hunting): handle_auth_done global_id 74499 payload 824 2024-09-16T05:18:53.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 1 -- 172.21.15.114:0/2990208856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24b809b6c0 msgr2=0x7f24b813b020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:18:53.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 1 --2- 172.21.15.114:0/2990208856 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24b809b6c0 0x7f24b813b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 1 -- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b80a19e0 msgr2=0x7f24b813baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 1 --2- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b80a19e0 0x7f24b813baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:18:53.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 1 monclient: found mon.noname-a 2024-09-16T05:18:53.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:18:53.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:18:53.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 1 -- 172.21.15.114:0/2990208856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b806ded0 con 0x7f24b80a15d0 2024-09-16T05:18:53.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b5d9b640 1 --2- 172.21.15.114:0/2990208856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b80a15d0 0x7f24b813b560 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f24a000b550 tx=0x7f24a000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:53.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 10 client.?.objecter ms_handle_connect 0x7f24b80a15d0 2024-09-16T05:18:53.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:18:53.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 1 -- 172.21.15.114:0/2990208856 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a0003dd0 con 0x7f24b80a15d0 2024-09-16T05:18:53.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:18:53.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 10 monclient: dump: 2024-09-16T05:18:53.721 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.722 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.722 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.722 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.722 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.722 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.722 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.722 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.722 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.722 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 10 monclient: _finish_auth 0 2024-09-16T05:18:53.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 10 monclient: _check_auth_tickets 2024-09-16T05:18:53.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:18:53.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 1 -- 172.21.15.114:0/2990208856 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24a0015070 con 0x7f24b80a15d0 2024-09-16T05:18:53.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:18:53.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 1 -- 172.21.15.114:0/2990208856 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a00052b0 con 0x7f24b80a15d0 2024-09-16T05:18:53.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 5 monclient: authenticate success, global_id 74499 2024-09-16T05:18:53.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient: _renew_subs 2024-09-16T05:18:53.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:53.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 -- 172.21.15.114:0/2990208856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b8072ae0 con 0x7f24b80a15d0 2024-09-16T05:18:53.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:18:53.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 10 monclient: dump: 2024-09-16T05:18:53.723 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.723 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.723 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.723 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.724 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.724 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.724 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.724 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.724 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.724 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b4d99640 4 set_mon_vals no callback set 2024-09-16T05:18:53.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 client.74499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:18:53.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient: _renew_subs 2024-09-16T05:18:53.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:18:53.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 -- 172.21.15.114:0/2990208856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b812bee0 con 0x7f24b80a15d0 2024-09-16T05:18:53.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 librados: init done 2024-09-16T05:18:53.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:53.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 --2- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b81ccb20 0x7f24b81ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24bcb58640 1 -- 172.21.15.114:0/2990208856 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f24b812ac50 con 0x7f24b81ccb20 2024-09-16T05:18:53.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 1 -- 172.21.15.114:0/2990208856 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f24a0005450 con 0x7f24b80a15d0 2024-09-16T05:18:53.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b659c640 1 --2- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b81ccb20 0x7f24b81ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b659c640 10 monclient: get_auth_request con 0x7f24b81ccb20 auth_method 0 2024-09-16T05:18:53.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:53.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:53.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:53.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 1 --2- 172.21.15.114:0/2990208856 >> v2:172.21.15.149:6812/1185531135 conn(0x7f24980377c0 0x7f2498039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 1 -- 172.21.15.114:0/2990208856 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(432..432 src has 1..432) ==== 5462+0+0 (secure 0 0 0) 0x7f24a0048cb0 con 0x7f24b80a15d0 2024-09-16T05:18:53.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 10 client.74499.objecter ms_dispatch 0x7f24b8000ba0 osd_map(432..432 src has 1..432) 2024-09-16T05:18:53.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 3 client.74499.objecter handle_osd_map got epochs [432,432] > 0 2024-09-16T05:18:53.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24aeffd640 3 client.74499.objecter handle_osd_map decoding full epoch 432 2024-09-16T05:18:53.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:53.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:53.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.709+0000 7f24b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:53.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24aeffd640 20 client.74499.objecter dump_active .. 0 homeless 2024-09-16T05:18:53.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24b6d9d640 1 --2- 172.21.15.114:0/2990208856 >> v2:172.21.15.149:6812/1185531135 conn(0x7f24980377c0 0x7f2498039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24b659c640 10 monclient(hunting): handle_auth_done global_id 64576 payload 824 2024-09-16T05:18:53.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24b659c640 1 -- 172.21.15.114:0/2990208856 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f24b80a0ec0 con 0x7f24b81ccb20 2024-09-16T05:18:53.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24b6d9d640 10 monclient: get_auth_request con 0x7f24980377c0 auth_method 0 2024-09-16T05:18:53.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24aeffd640 10 client.74499.objecter ms_handle_connect 0x7f24b81ccb20 2024-09-16T05:18:53.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24b659c640 1 --2- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b81ccb20 0x7f24b81ccf00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f24b000c7f0 tx=0x7f24b000ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:53.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24aeffd640 10 client.74499.objecter resend_mon_ops 2024-09-16T05:18:53.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24aeffd640 1 -- 172.21.15.114:0/2990208856 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b0017070 con 0x7f24b81ccb20 2024-09-16T05:18:53.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:18:53.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24aeffd640 10 monclient: dump: 2024-09-16T05:18:53.727 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.727 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.727 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.727 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.727 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.727 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.727 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.727 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.727 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.727 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24aeffd640 10 client.74499.objecter ms_handle_connect 0x7f24980377c0 2024-09-16T05:18:53.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24b6d9d640 1 --2- 172.21.15.114:0/2990208856 >> v2:172.21.15.149:6812/1185531135 conn(0x7f24980377c0 0x7f2498039c80 secure :-1 s=READY pgs=3552 cs=0 l=1 rev1=1 crypto rx=0x7f24a80ad000 tx=0x7f24a80a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:53.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24aeffd640 1 -- 172.21.15.114:0/2990208856 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f24b000a6b0 con 0x7f24b81ccb20 2024-09-16T05:18:53.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24aeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:18:53.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24aeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:18:53.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24aeffd640 1 -- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b81ccb20 msgr2=0x7f24b81ccf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24aeffd640 1 --2- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b81ccb20 0x7f24b81ccf00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f24b000c7f0 tx=0x7f24b000ccb0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24aeffd640 1 -- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b81ccb20 msgr2=0x7f24b81ccf00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:53.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.713+0000 7f24aeffd640 1 --2- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b81ccb20 0x7f24b81ccf00 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.717+0000 7f24bcb58640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:18:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.717+0000 7f24bcb58640 1 --2- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b81cd440 0x7f24b806f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:18:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.717+0000 7f24bcb58640 10 monclient(hunting): start opening mon connection 2024-09-16T05:18:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.717+0000 7f24bcb58640 1 -- 172.21.15.114:0/2990208856 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f24b812ba30 con 0x7f24b81cd440 2024-09-16T05:18:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24b659c640 1 --2- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b81cd440 0x7f24b806f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:18:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24b659c640 10 monclient: get_auth_request con 0x7f24b81cd440 auth_method 0 2024-09-16T05:18:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:18:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24b659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:18:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:18:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:18:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:18:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:18:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24b659c640 10 monclient(hunting): handle_auth_done global_id 64579 payload 824 2024-09-16T05:18:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24b659c640 1 -- 172.21.15.114:0/2990208856 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f24b812c0e0 con 0x7f24b81cd440 2024-09-16T05:18:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24b659c640 1 --2- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b81cd440 0x7f24b806f900 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f24b0014000 tx=0x7f24b000f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:18:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24aeffd640 10 client.74499.objecter ms_handle_connect 0x7f24b81cd440 2024-09-16T05:18:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24aeffd640 10 client.74499.objecter resend_mon_ops 2024-09-16T05:18:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24aeffd640 1 -- 172.21.15.114:0/2990208856 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b001c320 con 0x7f24b81cd440 2024-09-16T05:18:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:18:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:18:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24aeffd640 10 monclient: dump: 2024-09-16T05:18:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:18:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:18:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:18:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:18:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:18:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:18:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:18:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:18:53.732 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:18:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24aeffd640 1 -- 172.21.15.114:0/2990208856 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f24b001c320 con 0x7f24b81cd440 2024-09-16T05:18:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24aeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:18:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24aeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:18:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24aeffd640 1 -- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b81cd440 msgr2=0x7f24b806f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24aeffd640 1 --2- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b81cd440 0x7f24b806f900 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f24b0014000 tx=0x7f24b000f000 comp rx=0 tx=0).stop 2024-09-16T05:18:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24aeffd640 1 -- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b81cd440 msgr2=0x7f24b806f900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:18:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24aeffd640 1 --2- 172.21.15.114:0/2990208856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24b81cd440 0x7f24b806f900 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:18:53.733 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:18:53.733 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:18:53.733 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:18:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:18:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 54, 2024-09-16T05:18:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 417858, 2024-09-16T05:18:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:18:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:18:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-16T05:18:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:18:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:18:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:18:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:53.734 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:18:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:18:53.735 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:18:53.736 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:18:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:18:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:18:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:18:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:18:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:18:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:18:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:18:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:18:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:18:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:18:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:18:53.737 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:18:53.738 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:18:53.738 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:18:53.738 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:18:53.738 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:18:53.738 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:18:53.738 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:18:53.738 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:18:53.738 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:18:53.738 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:18:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:18:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:18:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:18:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:18:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:18:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:18:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:18:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:18:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:53.739 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:18:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:18:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.740 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:18:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:18:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:18:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:18:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:53.741 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:18:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:18:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:18:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:18:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:53.742 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:18:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:18:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:18:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:18:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:18:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:18:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.743 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:18:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:18:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:18:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:18:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:18:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:18:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:18:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:18:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.744 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:18:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:18:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:18:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:18:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:18:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:53.745 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:18:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:18:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:18:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:18:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:18:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:18:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:18:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:18:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:18:53.746 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:18:53.747 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:18:53.747 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:18:53.747 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:18:53.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24bcb58640 10 librados: watch_flush enter 2024-09-16T05:18:53.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24bcb58640 10 librados: watch_flush exit 2024-09-16T05:18:53.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24bcb58640 20 client.74499.objecter shutdown clearing up homeless session... 2024-09-16T05:18:53.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24bcb58640 10 client.74499.objecter successfully canceled tick 2024-09-16T05:18:53.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24bcb58640 1 -- 172.21.15.114:0/2990208856 >> v2:172.21.15.149:6812/1185531135 conn(0x7f24980377c0 msgr2=0x7f2498039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24bcb58640 1 --2- 172.21.15.114:0/2990208856 >> v2:172.21.15.149:6812/1185531135 conn(0x7f24980377c0 0x7f2498039c80 secure :-1 s=READY pgs=3552 cs=0 l=1 rev1=1 crypto rx=0x7f24a80ad000 tx=0x7f24a80a6000 comp rx=0 tx=0).stop 2024-09-16T05:18:53.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24bcb58640 10 monclient: shutdown 2024-09-16T05:18:53.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24bcb58640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:18:53.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24bcb58640 1 -- 172.21.15.114:0/2990208856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b80a15d0 msgr2=0x7f24b813b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:18:53.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24bcb58640 1 --2- 172.21.15.114:0/2990208856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b80a15d0 0x7f24b813b560 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f24a000b550 tx=0x7f24a000ba10 comp rx=0 tx=0).stop 2024-09-16T05:18:53.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24b6d9d640 1 -- 172.21.15.114:0/2990208856 reap_dead start 2024-09-16T05:18:53.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24b6d9d640 1 -- 172.21.15.114:0/2990208856 reap_dead start 2024-09-16T05:18:53.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24bcb58640 1 -- 172.21.15.114:0/2990208856 shutdown_connections 2024-09-16T05:18:53.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24bcb58640 1 -- 172.21.15.114:0/2990208856 >> 172.21.15.114:0/2990208856 conn(0x7f24b80a1fb0 msgr2=0x7f24b80a4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:18:53.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24bcb58640 1 -- 172.21.15.114:0/2990208856 shutdown_connections 2024-09-16T05:18:53.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24bcb58640 1 -- 172.21.15.114:0/2990208856 wait complete. 2024-09-16T05:18:53.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:18:53.721+0000 7f24bcb58640 1 librados: shutdown 2024-09-16T05:18:53.750 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:18:53.750 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:19:07.088 INFO:tasks.ceph.osd.2.smithi114.stderr:2024-09-16T05:19:07.089+0000 7fadcfd15640 -1 snap_mapper.add_oid found existing snaps mapped on 104:bfbcb972:test-rados-api-smithi149-29623-97::foo:21, removing 2024-09-16T05:19:07.089 INFO:tasks.ceph.osd.4.smithi149.stderr:2024-09-16T05:19:07.090+0000 7f59bebf7640 -1 snap_mapper.add_oid found existing snaps mapped on 104:bfbcb972:test-rados-api-smithi149-29623-97::foo:21, removing 2024-09-16T05:20:23.751 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T05:20:23.752 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T05:20:23.752 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T05:20:23.752 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T05:20:23.756 INFO:tasks.ceph.mon.c:Started 2024-09-16T05:20:23.756 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:20:23.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:20:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 10 monclient: get_monmap_and_config 2024-09-16T05:20:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 10 monclient: build_initial_monmap 2024-09-16T05:20:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 10 monclient: monmap: 2024-09-16T05:20:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:20:23.882251+0000 2024-09-16T05:20:23.877 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:20:23.882251+0000 2024-09-16T05:20:23.878 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:23.878 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:23.878 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:20:23.878 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:20:23.878 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:20:23.878 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:20:23.878 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:20:23.878 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:23.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 1 Processor -- start 2024-09-16T05:20:23.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 1 -- start start 2024-09-16T05:20:23.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 10 monclient: init 2024-09-16T05:20:23.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:23.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T05:20:23.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 0x7fb368137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:23.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 10 monclient(hunting): picked mon.noname-b con 0x7fb3681376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:23.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3681370a0 addr v1:172.21.15.149:6789/0 2024-09-16T05:20:23.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb3680a5d00 0x7fb3680a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:23.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 10 monclient(hunting): picked mon.noname-c con 0x7fb3680a5d00 addr v2:172.21.15.149:3300/0 2024-09-16T05:20:23.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb36809bd20 con 0x7fb3681370a0 2024-09-16T05:20:23.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:23.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fb368138000 con 0x7fb3680a5d00 2024-09-16T05:20:23.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:23.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb36809c5c0 con 0x7fb3681376e0 2024-09-16T05:20:23.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:23.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb36d29d640 10 monclient(hunting): authenticate will time out at 1708.511134s 2024-09-16T05:20:23.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb366ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 0x7fb368137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:23.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb366ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 0x7fb368137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:60166/0 (socket says 172.21.15.114:60166) 2024-09-16T05:20:23.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb366ffd640 1 -- 172.21.15.114:0/2080270735 learned_addr learned my addr 172.21.15.114:0/2080270735 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:23.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb366ffd640 10 monclient(hunting): get_auth_request con 0x7fb3681376e0 auth_method 0 2024-09-16T05:20:23.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb366ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:23.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb366ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:23.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:23.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb366ffd640 1 -- 172.21.15.114:0/2080270735 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 msgr2=0x7fb368137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:20:23.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb366ffd640 1 -- 172.21.15.114:0/2080270735 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 msgr2=0x7fb368137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:20:23.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb366ffd640 1 --2- 172.21.15.114:0/2080270735 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 0x7fb368137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:23.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb366ffd640 1 --2- 172.21.15.114:0/2080270735 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 0x7fb368137ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:20:23.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb3677fe640 1 --2- 172.21.15.114:0/2080270735 >> v2:172.21.15.149:3300/0 conn(0x7fb3680a5d00 0x7fb3680a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:23.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb3677fe640 10 monclient(hunting): get_auth_request con 0x7fb3680a5d00 auth_method 0 2024-09-16T05:20:23.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb3677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:23.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb3677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:23.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb3677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:23.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb3677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:23.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb3677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:23.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.879+0000 7fb3677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:23.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb3677fe640 10 monclient(hunting): handle_auth_done global_id 64606 payload 274 2024-09-16T05:20:23.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb3677fe640 1 -- 172.21.15.114:0/2080270735 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 msgr2=0x7fb368137ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:20:23.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb3677fe640 1 --2- 172.21.15.114:0/2080270735 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 0x7fb368137ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:23.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb3677fe640 1 -- 172.21.15.114:0/2080270735 >> v1:172.21.15.149:6789/0 conn(0x7fb3681370a0 legacy=0x7fb3680a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:23.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb3677fe640 10 monclient: _finish_hunting 0 2024-09-16T05:20:23.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb3677fe640 1 monclient: found mon.noname-c 2024-09-16T05:20:23.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb3677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:23.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb3677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:20:23.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb3677fe640 1 -- 172.21.15.114:0/2080270735 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36809c000 con 0x7fb3680a5d00 2024-09-16T05:20:23.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb3677fe640 1 --2- 172.21.15.114:0/2080270735 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb3680a5d00 0x7fb3680a4980 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb3580097c0 tx=0x7fb35801c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe1de749c2039030 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:23.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb365ffb640 1 -- 172.21.15.114:0/2080270735 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3580b0070 con 0x7fb3680a5d00 2024-09-16T05:20:23.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb365ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:23.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb365ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:20:23.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb365ffb640 10 monclient: dump: 2024-09-16T05:20:23.891 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:23.891 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:23.891 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:23.891 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:23.891 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:23.891 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:23.891 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:23.891 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:23.891 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:23.891 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:23.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb365ffb640 10 monclient: _finish_auth 0 2024-09-16T05:20:23.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb365ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:20:23.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb365ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:23.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb365ffb640 1 -- 172.21.15.114:0/2080270735 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3580ab600 con 0x7fb3680a5d00 2024-09-16T05:20:23.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb36d29d640 5 monclient: authenticate success, global_id 64606 2024-09-16T05:20:23.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb36d29d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:23.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb365ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:23.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb36d29d640 10 monclient: get_monmap_and_config success 2024-09-16T05:20:23.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb365ffb640 1 -- 172.21.15.114:0/2080270735 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3580ab9d0 con 0x7fb3680a5d00 2024-09-16T05:20:23.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb365ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:23.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb36d29d640 4 set_mon_vals no callback set 2024-09-16T05:20:23.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb365ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:20:23.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb365ffb640 10 monclient: dump: 2024-09-16T05:20:23.893 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:23.893 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:23.893 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:23.893 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:23.893 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:23.893 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:23.893 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:23.893 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:23.893 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:23.893 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:23.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb36d29d640 10 monclient: shutdown 2024-09-16T05:20:23.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb36d29d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:23.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb36d29d640 1 -- 172.21.15.114:0/2080270735 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb3680a5d00 msgr2=0x7fb3680a4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:23.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb36d29d640 1 --2- 172.21.15.114:0/2080270735 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb3680a5d00 0x7fb3680a4980 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb3580097c0 tx=0x7fb35801c360 comp rx=0 tx=0).stop 2024-09-16T05:20:23.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb36d29d640 1 -- 172.21.15.114:0/2080270735 shutdown_connections 2024-09-16T05:20:23.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb36d29d640 1 --2- 172.21.15.114:0/2080270735 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb3680a5d00 0x7fb3680a4980 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:23.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb36d29d640 1 --2- 172.21.15.114:0/2080270735 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 0x7fb368137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:23.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb36d29d640 1 -- 172.21.15.114:0/2080270735 >> 172.21.15.114:0/2080270735 conn(0x7fb3680a9a40 msgr2=0x7fb3680a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:23.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb36d29d640 1 -- 172.21.15.114:0/2080270735 shutdown_connections 2024-09-16T05:20:23.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.883+0000 7fb36d29d640 1 -- 172.21.15.114:0/2080270735 wait complete. 2024-09-16T05:20:23.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient: build_initial_monmap 2024-09-16T05:20:23.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient: monmap: 2024-09-16T05:20:23.894 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:23.894 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:20:23.894 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:20:23.895 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:20:23.895 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:23.895 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:23.895 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:20:23.895 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:20:23.895 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:20:23.895 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:23.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 librados: starting msgr at 2024-09-16T05:20:23.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 librados: starting objecter 2024-09-16T05:20:23.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 Processor -- start 2024-09-16T05:20:23.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 -- start start 2024-09-16T05:20:23.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 librados: setting wanted keys 2024-09-16T05:20:23.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 librados: calling monclient init 2024-09-16T05:20:23.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient: init 2024-09-16T05:20:23.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:23.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:20:23.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3680a5d00 0x7fb3681cf110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:23.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient(hunting): picked mon.noname-a con 0x7fb3680a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:23.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb3681370a0 0x7fb368136090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:23.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient(hunting): picked mon.noname-b con 0x7fb3681370a0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:23.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 0x7fb36813be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:23.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient(hunting): picked mon.noname-c con 0x7fb3681376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:23.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:23.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb368141d80 con 0x7fb3680a5d00 2024-09-16T05:20:23.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:23.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb368141c00 con 0x7fb3681370a0 2024-09-16T05:20:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb368141f00 con 0x7fb3681376e0 2024-09-16T05:20:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient(hunting): authenticate will time out at 1708.517168s 2024-09-16T05:20:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3677fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 0x7fb36813be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3680a5d00 0x7fb3681cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3677fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 0x7fb36813be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:60182/0 (socket says 172.21.15.114:60182) 2024-09-16T05:20:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3677fe640 1 -- 172.21.15.114:0/4036959624 learned_addr learned my addr 172.21.15.114:0/4036959624 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3677fe640 10 monclient(hunting): get_auth_request con 0x7fb3681376e0 auth_method 0 2024-09-16T05:20:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3680a5d00 0x7fb3681cf110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34358/0 (socket says 172.21.15.114:34358) 2024-09-16T05:20:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 10 monclient(hunting): get_auth_request con 0x7fb3680a5d00 auth_method 0 2024-09-16T05:20:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3677fe640 1 -- 172.21.15.114:0/4036959624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 msgr2=0x7fb36813be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:20:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3677fe640 1 -- 172.21.15.114:0/4036959624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 msgr2=0x7fb36813be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:20:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3677fe640 1 --2- 172.21.15.114:0/4036959624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 0x7fb36813be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3677fe640 1 --2- 172.21.15.114:0/4036959624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 0x7fb36813be80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:20:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3667fc640 1 --2- 172.21.15.114:0/4036959624 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb3681370a0 0x7fb368136090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3667fc640 10 monclient(hunting): get_auth_request con 0x7fb3681370a0 auth_method 0 2024-09-16T05:20:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 10 monclient(hunting): handle_auth_done global_id 74514 payload 824 2024-09-16T05:20:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 1 -- 172.21.15.114:0/4036959624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 msgr2=0x7fb36813be80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:20:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 1 --2- 172.21.15.114:0/4036959624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 0x7fb36813be80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 1 -- 172.21.15.114:0/4036959624 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb3681370a0 msgr2=0x7fb368136090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 1 --2- 172.21.15.114:0/4036959624 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb3681370a0 0x7fb368136090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:20:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 1 monclient: found mon.noname-a 2024-09-16T05:20:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:23.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 1 -- 172.21.15.114:0/4036959624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36813db20 con 0x7fb3680a5d00 2024-09-16T05:20:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb366ffd640 1 --2- 172.21.15.114:0/4036959624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3680a5d00 0x7fb3681cf110 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb36000e810 tx=0x7fb36000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 10 client.?.objecter ms_handle_connect 0x7fb3680a5d00 2024-09-16T05:20:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:20:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 1 -- 172.21.15.114:0/4036959624 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb360017070 con 0x7fb3680a5d00 2024-09-16T05:20:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 10 monclient: dump: 2024-09-16T05:20:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:23.900 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:23.901 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:23.901 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:23.901 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:23.901 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:23.901 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:23.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:20:23.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:20:23.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:23.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 5 monclient: authenticate success, global_id 74514 2024-09-16T05:20:23.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 1 -- 172.21.15.114:0/4036959624 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb360004020 con 0x7fb3680a5d00 2024-09-16T05:20:23.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:23.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 1 -- 172.21.15.114:0/4036959624 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36000f040 con 0x7fb3680a5d00 2024-09-16T05:20:23.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:23.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:23.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 10 monclient: dump: 2024-09-16T05:20:23.902 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:23.902 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:23.902 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:23.902 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:23.902 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:23.902 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:23.902 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:23.902 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:23.902 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:23.902 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:23.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb3657fa640 4 set_mon_vals no callback set 2024-09-16T05:20:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient: _renew_subs 2024-09-16T05:20:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 -- 172.21.15.114:0/4036959624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb368120460 con 0x7fb3680a5d00 2024-09-16T05:20:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 client.74514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:20:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient: _renew_subs 2024-09-16T05:20:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 -- 172.21.15.114:0/4036959624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3680a8f50 con 0x7fb3680a5d00 2024-09-16T05:20:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 librados: init done 2024-09-16T05:20:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb36d29d640 1 -- 172.21.15.114:0/4036959624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb368137ac0 con 0x7fb3680a5d00 2024-09-16T05:20:23.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 1 -- 172.21.15.114:0/4036959624 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb36000c6d0 con 0x7fb3680a5d00 2024-09-16T05:20:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 1 --2- 172.21.15.114:0/4036959624 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb354037840 0x7fb354039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 1 -- 172.21.15.114:0/4036959624 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(514..514 src has 1..514) ==== 5853+0+0 (secure 0 0 0) 0x7fb360048a90 con 0x7fb3680a5d00 2024-09-16T05:20:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 10 client.74514.objecter ms_dispatch 0x7fb368000ba0 osd_map(514..514 src has 1..514) 2024-09-16T05:20:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 3 client.74514.objecter handle_osd_map got epochs [514,514] > 0 2024-09-16T05:20:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 3 client.74514.objecter handle_osd_map decoding full epoch 514 2024-09-16T05:20:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.887+0000 7fb34f7fe640 20 client.74514.objecter dump_active .. 0 homeless 2024-09-16T05:20:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.891+0000 7fb3667fc640 1 --2- 172.21.15.114:0/4036959624 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb354037840 0x7fb354039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.891+0000 7fb3667fc640 10 monclient: get_auth_request con 0x7fb354037840 auth_method 0 2024-09-16T05:20:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.891+0000 7fb34f7fe640 10 client.74514.objecter ms_handle_connect 0x7fb354037840 2024-09-16T05:20:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.891+0000 7fb3667fc640 1 --2- 172.21.15.114:0/4036959624 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb354037840 0x7fb354039d00 secure :-1 s=READY pgs=3558 cs=0 l=1 rev1=1 crypto rx=0x7fb3680a1e30 tx=0x7fb35c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:23.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.895+0000 7fb34f7fe640 1 -- 172.21.15.114:0/4036959624 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb360004460 con 0x7fb3680a5d00 2024-09-16T05:20:23.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.895+0000 7fb34f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:23.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:23.895+0000 7fb34f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:20:24.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.119+0000 7fb36d29d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:20:24.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.119+0000 7fb36d29d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:20:24.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.119+0000 7fb36d29d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:24.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.119+0000 7fb36d29d640 1 -- 172.21.15.114:0/4036959624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb36811fec0 con 0x7fb3680a5d00 2024-09-16T05:20:24.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb34f7fe640 1 -- 172.21.15.114:0/4036959624 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fb360012740 con 0x7fb3680a5d00 2024-09-16T05:20:24.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb34f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:20:24.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb34f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:20:24.120 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:20:24.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 10 librados: watch_flush enter 2024-09-16T05:20:24.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 10 librados: watch_flush exit 2024-09-16T05:20:24.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 20 client.74514.objecter shutdown clearing up homeless session... 2024-09-16T05:20:24.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 10 client.74514.objecter successfully canceled tick 2024-09-16T05:20:24.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 1 -- 172.21.15.114:0/4036959624 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb354037840 msgr2=0x7fb354039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:24.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 1 --2- 172.21.15.114:0/4036959624 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb354037840 0x7fb354039d00 secure :-1 s=READY pgs=3558 cs=0 l=1 rev1=1 crypto rx=0x7fb3680a1e30 tx=0x7fb35c009040 comp rx=0 tx=0).stop 2024-09-16T05:20:24.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 10 monclient: shutdown 2024-09-16T05:20:24.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:24.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 1 -- 172.21.15.114:0/4036959624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3680a5d00 msgr2=0x7fb3681cf110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:24.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 1 --2- 172.21.15.114:0/4036959624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3680a5d00 0x7fb3681cf110 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb36000e810 tx=0x7fb36000ecd0 comp rx=0 tx=0).stop 2024-09-16T05:20:24.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 1 -- 172.21.15.114:0/4036959624 shutdown_connections 2024-09-16T05:20:24.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 1 --2- 172.21.15.114:0/4036959624 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb354037840 0x7fb354039d00 secure :-1 s=CLOSED pgs=3558 cs=0 l=1 rev1=1 crypto rx=0x7fb3680a1e30 tx=0x7fb35c009040 comp rx=0 tx=0).stop 2024-09-16T05:20:24.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 1 --2- 172.21.15.114:0/4036959624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3681376e0 0x7fb36813be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:24.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 1 --2- 172.21.15.114:0/4036959624 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb3681370a0 0x7fb368136090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:24.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 1 --2- 172.21.15.114:0/4036959624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3680a5d00 0x7fb3681cf110 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:24.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 1 -- 172.21.15.114:0/4036959624 >> 172.21.15.114:0/4036959624 conn(0x7fb3680a9a40 msgr2=0x7fb3680a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:24.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 1 -- 172.21.15.114:0/4036959624 shutdown_connections 2024-09-16T05:20:24.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 1 -- 172.21.15.114:0/4036959624 wait complete. 2024-09-16T05:20:24.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:24.123+0000 7fb36d29d640 1 librados: shutdown 2024-09-16T05:20:27.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:20:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 10 monclient: get_monmap_and_config 2024-09-16T05:20:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 10 monclient: build_initial_monmap 2024-09-16T05:20:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 10 monclient: monmap: 2024-09-16T05:20:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:20:27.277558+0000 2024-09-16T05:20:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:20:27.277558+0000 2024-09-16T05:20:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:20:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:20:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:20:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:20:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:20:27.273 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:27.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 1 Processor -- start 2024-09-16T05:20:27.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 1 -- start start 2024-09-16T05:20:27.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 10 monclient: init 2024-09-16T05:20:27.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:27.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-16T05:20:27.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f01b409b6c0 addr v1:172.21.15.149:6789/0 2024-09-16T05:20:27.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a15d0 0x7f01b413a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:27.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 10 monclient(hunting): picked mon.noname-b con 0x7f01b40a15d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:27.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f01b40a19e0 0x7f01b413cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:27.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 10 monclient(hunting): picked mon.noname-a con 0x7f01b40a19e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:20:27.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01b409d4e0 con 0x7f01b409b6c0 2024-09-16T05:20:27.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:27.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f01b409e390 con 0x7f01b40a19e0 2024-09-16T05:20:27.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:27.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f01b409ece0 con 0x7f01b40a15d0 2024-09-16T05:20:27.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:27.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01bb4c4640 10 monclient(hunting): authenticate will time out at 1711.906812s 2024-09-16T05:20:27.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f01b40a19e0 0x7f01b413cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:27.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f01b40a19e0 0x7f01b413cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39584/0 (socket says 172.21.15.114:39584) 2024-09-16T05:20:27.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 1 -- 172.21.15.114:0/3442294878 learned_addr learned my addr 172.21.15.114:0/3442294878 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:27.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 10 monclient(hunting): get_auth_request con 0x7f01b40a19e0 auth_method 0 2024-09-16T05:20:27.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:27.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:27.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:27.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b8a5f640 1 --2- 172.21.15.114:0/3442294878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a15d0 0x7f01b413a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:27.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b8a5f640 10 monclient(hunting): get_auth_request con 0x7f01b40a15d0 auth_method 0 2024-09-16T05:20:27.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b8a5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:27.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b8a5f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:27.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b8a5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:27.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:27.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b8a5f640 1 -- 172.21.15.114:0/3442294878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a15d0 msgr2=0x7f01b413a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:20:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b8a5f640 1 -- 172.21.15.114:0/3442294878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a15d0 msgr2=0x7f01b413a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:20:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b8a5f640 1 --2- 172.21.15.114:0/3442294878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a15d0 0x7f01b413a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b8a5f640 1 --2- 172.21.15.114:0/3442294878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a15d0 0x7f01b413a650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:20:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 10 monclient(hunting): handle_auth_done global_id 74517 payload 274 2024-09-16T05:20:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 1 -- 172.21.15.114:0/3442294878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a15d0 msgr2=0x7f01b413a650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:20:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 1 --2- 172.21.15.114:0/3442294878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a15d0 0x7f01b413a650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 1 -- 172.21.15.114:0/3442294878 >> v1:172.21.15.149:6789/0 conn(0x7f01b409b6c0 legacy=0x7f01b40a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 10 monclient: _finish_hunting 0 2024-09-16T05:20:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 1 monclient: found mon.noname-a 2024-09-16T05:20:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.275+0000 7f01b9a61640 1 -- 172.21.15.114:0/3442294878 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01b409f430 con 0x7f01b40a19e0 2024-09-16T05:20:27.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01b9a61640 1 --2- 172.21.15.114:0/3442294878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01b40a19e0 0x7f01b413cba0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f01ac009e50 tx=0x7f01ac01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf48481b48045b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:27.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01abfff640 1 -- 172.21.15.114:0/3442294878 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ac0a5620 con 0x7f01b40a19e0 2024-09-16T05:20:27.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:27.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:27.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01abfff640 10 monclient: dump: 2024-09-16T05:20:27.284 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:27.284 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:27.284 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:27.284 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:27.285 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:27.285 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:27.285 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:27.285 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:27.285 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:27.285 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:27.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01abfff640 10 monclient: _finish_auth 0 2024-09-16T05:20:27.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01abfff640 10 monclient: _check_auth_tickets 2024-09-16T05:20:27.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:27.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01abfff640 1 -- 172.21.15.114:0/3442294878 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01ac0a57c0 con 0x7f01b40a19e0 2024-09-16T05:20:27.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 5 monclient: authenticate success, global_id 74517 2024-09-16T05:20:27.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:27.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01abfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:27.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 10 monclient: get_monmap_and_config success 2024-09-16T05:20:27.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01abfff640 1 -- 172.21.15.114:0/3442294878 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ac0a5b90 con 0x7f01b40a19e0 2024-09-16T05:20:27.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 4 set_mon_vals no callback set 2024-09-16T05:20:27.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:27.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:27.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01abfff640 10 monclient: dump: 2024-09-16T05:20:27.286 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:27.286 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:27.286 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:27.286 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:27.286 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:27.286 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:27.287 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:27.287 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:27.287 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:27.287 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:27.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 10 monclient: shutdown 2024-09-16T05:20:27.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:27.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 -- 172.21.15.114:0/3442294878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01b40a19e0 msgr2=0x7f01b413cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:27.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 --2- 172.21.15.114:0/3442294878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01b40a19e0 0x7f01b413cba0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f01ac009e50 tx=0x7f01ac01c8b0 comp rx=0 tx=0).stop 2024-09-16T05:20:27.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 -- 172.21.15.114:0/3442294878 shutdown_connections 2024-09-16T05:20:27.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 --2- 172.21.15.114:0/3442294878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01b40a19e0 0x7f01b413cba0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:27.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 --2- 172.21.15.114:0/3442294878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a15d0 0x7f01b413a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:27.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 -- 172.21.15.114:0/3442294878 >> 172.21.15.114:0/3442294878 conn(0x7f01b40a1fb0 msgr2=0x7f01b40a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:27.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 -- 172.21.15.114:0/3442294878 shutdown_connections 2024-09-16T05:20:27.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 -- 172.21.15.114:0/3442294878 wait complete. 2024-09-16T05:20:27.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 10 monclient: build_initial_monmap 2024-09-16T05:20:27.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 10 monclient: monmap: 2024-09-16T05:20:27.288 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:27.288 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:20:27.288 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:20:27.288 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:20:27.288 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:27.288 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:27.288 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:20:27.288 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:20:27.288 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:20:27.288 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:27.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 librados: starting msgr at 2024-09-16T05:20:27.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 librados: starting objecter 2024-09-16T05:20:27.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 Processor -- start 2024-09-16T05:20:27.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 -- start start 2024-09-16T05:20:27.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 librados: setting wanted keys 2024-09-16T05:20:27.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 librados: calling monclient init 2024-09-16T05:20:27.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 10 monclient: init 2024-09-16T05:20:27.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:27.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:20:27.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01b409b6c0 0x7f01b40969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:27.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 10 monclient(hunting): picked mon.noname-b con 0x7f01b409b6c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:27.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01b40a15d0 0x7f01b4099530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:27.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 10 monclient(hunting): picked mon.noname-a con 0x7f01b40a15d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:27.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a19e0 0x7f01b4099a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:27.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 10 monclient(hunting): picked mon.noname-c con 0x7f01b40a19e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:27.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:27.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f01b413f900 con 0x7f01b40a15d0 2024-09-16T05:20:27.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:27.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f01b409cad0 con 0x7f01b409b6c0 2024-09-16T05:20:27.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:27.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f01b413fa80 con 0x7f01b40a19e0 2024-09-16T05:20:27.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:27.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.279+0000 7f01bb4c4640 10 monclient(hunting): authenticate will time out at 1711.911905s 2024-09-16T05:20:27.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01b40a15d0 0x7f01b4099530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:27.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9a61640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a19e0 0x7f01b4099a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:27.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9a61640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a19e0 0x7f01b4099a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:42354/0 (socket says 172.21.15.114:42354) 2024-09-16T05:20:27.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9a61640 1 -- 172.21.15.114:0/101839620 learned_addr learned my addr 172.21.15.114:0/101839620 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:27.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9a61640 10 monclient(hunting): get_auth_request con 0x7f01b40a19e0 auth_method 0 2024-09-16T05:20:27.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9a61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:27.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9a61640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:27.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9a61640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:27.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 10 monclient(hunting): get_auth_request con 0x7f01b40a15d0 auth_method 0 2024-09-16T05:20:27.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:27.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:27.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:27.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9a61640 1 -- 172.21.15.114:0/101839620 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a19e0 msgr2=0x7f01b4099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T05:20:27.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9a61640 1 -- 172.21.15.114:0/101839620 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a19e0 msgr2=0x7f01b4099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:20:27.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9a61640 1 --2- 172.21.15.114:0/101839620 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a19e0 0x7f01b4099a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:27.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:27.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:27.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:27.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9a61640 1 --2- 172.21.15.114:0/101839620 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a19e0 0x7f01b4099a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:20:27.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9260640 1 --2- 172.21.15.114:0/101839620 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01b409b6c0 0x7f01b40969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:27.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9260640 10 monclient(hunting): get_auth_request con 0x7f01b409b6c0 auth_method 0 2024-09-16T05:20:27.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9260640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:27.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9260640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:27.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9260640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:27.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 10 monclient(hunting): handle_auth_done global_id 74520 payload 824 2024-09-16T05:20:27.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 1 -- 172.21.15.114:0/101839620 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a19e0 msgr2=0x7f01b4099a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:20:27.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 1 --2- 172.21.15.114:0/101839620 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a19e0 0x7f01b4099a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:27.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 1 -- 172.21.15.114:0/101839620 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01b409b6c0 msgr2=0x7f01b40969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:27.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 1 --2- 172.21.15.114:0/101839620 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01b409b6c0 0x7f01b40969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:27.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 10 monclient: _finish_hunting 0 2024-09-16T05:20:27.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 1 monclient: found mon.noname-a 2024-09-16T05:20:27.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:27.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:27.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 1 -- 172.21.15.114:0/101839620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01b4097b70 con 0x7f01b40a15d0 2024-09-16T05:20:27.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b8a5f640 1 --2- 172.21.15.114:0/101839620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01b40a15d0 0x7f01b4099530 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f01a400d5d0 tx=0x7f01a400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:27.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 10 client.?.objecter ms_handle_connect 0x7f01b40a15d0 2024-09-16T05:20:27.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:20:27.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 1 -- 172.21.15.114:0/101839620 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a4005bb0 con 0x7f01b40a15d0 2024-09-16T05:20:27.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:27.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:27.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 10 monclient: dump: 2024-09-16T05:20:27.293 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:27.293 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:27.294 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:27.294 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:27.294 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:27.294 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:27.294 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:27.294 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:27.294 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:27.294 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:27.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:20:27.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:20:27.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:27.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 1 -- 172.21.15.114:0/101839620 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01a4018070 con 0x7f01b40a15d0 2024-09-16T05:20:27.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01bb4c4640 5 monclient: authenticate success, global_id 74520 2024-09-16T05:20:27.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01bb4c4640 10 monclient: _renew_subs 2024-09-16T05:20:27.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01bb4c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:27.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01bb4c4640 1 -- 172.21.15.114:0/101839620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01b409b0e0 con 0x7f01b40a15d0 2024-09-16T05:20:27.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:27.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 1 -- 172.21.15.114:0/101839620 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a4006050 con 0x7f01b40a15d0 2024-09-16T05:20:27.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:27.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:27.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 10 monclient: dump: 2024-09-16T05:20:27.295 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:27.295 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:27.295 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:27.295 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:27.295 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:27.295 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:27.295 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:27.295 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:27.296 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:27.296 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:27.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01ab7fe640 4 set_mon_vals no callback set 2024-09-16T05:20:27.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01bb4c4640 10 client.74520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:20:27.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01bb4c4640 10 monclient: _renew_subs 2024-09-16T05:20:27.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01bb4c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:27.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01bb4c4640 1 -- 172.21.15.114:0/101839620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01b40963e0 con 0x7f01b40a15d0 2024-09-16T05:20:27.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01bb4c4640 1 librados: init done 2024-09-16T05:20:27.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01bb4c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:27.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01bb4c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:27.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01bb4c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:27.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01bb4c4640 1 -- 172.21.15.114:0/101839620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01b40a0ec0 con 0x7f01b40a15d0 2024-09-16T05:20:27.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 1 -- 172.21.15.114:0/101839620 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f01a40061f0 con 0x7f01b40a15d0 2024-09-16T05:20:27.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 1 --2- 172.21.15.114:0/101839620 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0198037810 0x7f0198039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:27.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 1 -- 172.21.15.114:0/101839620 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(518..518 src has 1..518) ==== 5861+0+0 (secure 0 0 0) 0x7f01a4049110 con 0x7f01b40a15d0 2024-09-16T05:20:27.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 10 client.74520.objecter ms_dispatch 0x7f01b4000ba0 osd_map(518..518 src has 1..518) 2024-09-16T05:20:27.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 3 client.74520.objecter handle_osd_map got epochs [518,518] > 0 2024-09-16T05:20:27.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 3 client.74520.objecter handle_osd_map decoding full epoch 518 2024-09-16T05:20:27.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 20 client.74520.objecter dump_active .. 0 homeless 2024-09-16T05:20:27.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9260640 1 --2- 172.21.15.114:0/101839620 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0198037810 0x7f0198039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:27.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9260640 10 monclient: get_auth_request con 0x7f0198037810 auth_method 0 2024-09-16T05:20:27.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01a9ffb640 10 client.74520.objecter ms_handle_connect 0x7f0198037810 2024-09-16T05:20:27.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.283+0000 7f01b9260640 1 --2- 172.21.15.114:0/101839620 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0198037810 0x7f0198039cd0 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 crypto rx=0x7f01b000b810 tx=0x7f01b0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:27.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.287+0000 7f01a9ffb640 1 -- 172.21.15.114:0/101839620 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f01a401be40 con 0x7f01b40a15d0 2024-09-16T05:20:27.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.287+0000 7f01a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:27.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.287+0000 7f01a9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:20:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:20:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:20:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 1 -- 172.21.15.114:0/101839620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f01b4097640 con 0x7f01b40a15d0 2024-09-16T05:20:27.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01a9ffb640 1 -- 172.21.15.114:0/101839620 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f01a40139c0 con 0x7f01b40a15d0 2024-09-16T05:20:27.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:20:27.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01a9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:20:27.516 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":94,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:20:27.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 10 librados: watch_flush enter 2024-09-16T05:20:27.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 10 librados: watch_flush exit 2024-09-16T05:20:27.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 20 client.74520.objecter shutdown clearing up homeless session... 2024-09-16T05:20:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 10 client.74520.objecter successfully canceled tick 2024-09-16T05:20:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 1 -- 172.21.15.114:0/101839620 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0198037810 msgr2=0x7f0198039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 1 --2- 172.21.15.114:0/101839620 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0198037810 0x7f0198039cd0 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 crypto rx=0x7f01b000b810 tx=0x7f01b0008040 comp rx=0 tx=0).stop 2024-09-16T05:20:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 10 monclient: shutdown 2024-09-16T05:20:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 1 -- 172.21.15.114:0/101839620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01b40a15d0 msgr2=0x7f01b4099530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 1 --2- 172.21.15.114:0/101839620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01b40a15d0 0x7f01b4099530 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f01a400d5d0 tx=0x7f01a400da90 comp rx=0 tx=0).stop 2024-09-16T05:20:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 1 -- 172.21.15.114:0/101839620 shutdown_connections 2024-09-16T05:20:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 1 --2- 172.21.15.114:0/101839620 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0198037810 0x7f0198039cd0 unknown :-1 s=CLOSED pgs=3559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 1 --2- 172.21.15.114:0/101839620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01b40a15d0 0x7f01b4099530 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 1 --2- 172.21.15.114:0/101839620 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f01b40a19e0 0x7f01b4099a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 1 --2- 172.21.15.114:0/101839620 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01b409b6c0 0x7f01b40969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:27.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 1 -- 172.21.15.114:0/101839620 >> 172.21.15.114:0/101839620 conn(0x7f01b40a1fb0 msgr2=0x7f01b40713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 1 -- 172.21.15.114:0/101839620 shutdown_connections 2024-09-16T05:20:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 1 -- 172.21.15.114:0/101839620 wait complete. 2024-09-16T05:20:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:27.519+0000 7f01bb4c4640 1 librados: shutdown 2024-09-16T05:20:30.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:20:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 10 monclient: get_monmap_and_config 2024-09-16T05:20:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 10 monclient: build_initial_monmap 2024-09-16T05:20:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 10 monclient: monmap: 2024-09-16T05:20:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:20:30.670047+0000 2024-09-16T05:20:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:20:30.670047+0000 2024-09-16T05:20:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:20:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:20:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:20:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:20:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:20:30.666 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:30.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 1 Processor -- start 2024-09-16T05:20:30.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 1 -- start start 2024-09-16T05:20:30.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 10 monclient: init 2024-09-16T05:20:30.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:30.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-16T05:20:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc55409b6c0 0x7fc5540a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 10 monclient(hunting): picked mon.noname-c con 0x7fc55409b6c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:20:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5540a1400 addr v1:172.21.15.149:6789/0 2024-09-16T05:20:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5540a1810 addr v1:172.21.15.114:6789/0 2024-09-16T05:20:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc55409d4e0 con 0x7fc5540a1810 2024-09-16T05:20:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc55409f430 con 0x7fc5540a1400 2024-09-16T05:20:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fc55409e390 con 0x7fc55409b6c0 2024-09-16T05:20:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55c174640 10 monclient(hunting): authenticate will time out at 1715.299218s 2024-09-16T05:20:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55a711640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc5540a1810 0x7fc55413cd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:41254/0 (socket says 172.21.15.114:41254) 2024-09-16T05:20:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc55a711640 1 -- 172.21.15.114:0/3565196522 learned_addr learned my addr 172.21.15.114:0/3565196522 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:20:30.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc558f0e640 1 -- 172.21.15.114:0/3565196522 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4175282525 0 0) 0x7fc55409d4e0 con 0x7fc5540a1810 2024-09-16T05:20:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc558f0e640 10 monclient(hunting): _negotiate 2024-09-16T05:20:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc558f0e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc558f0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc558f0e640 10 monclient(hunting): my global_id is 74523 2024-09-16T05:20:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc558f0e640 1 -- 172.21.15.114:0/3565196522 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc540003530 con 0x7fc5540a1810 2024-09-16T05:20:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc559f10640 1 --2- 172.21.15.114:0/3565196522 >> v2:172.21.15.149:3300/0 conn(0x7fc55409b6c0 0x7fc5540a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc559f10640 10 monclient(hunting): get_auth_request con 0x7fc55409b6c0 auth_method 0 2024-09-16T05:20:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc559f10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc559f10640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc559f10640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc559f10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc559f10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc559f10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.667+0000 7fc558f0e640 1 -- 172.21.15.114:0/3565196522 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2854477573 0 0) 0x7fc540003530 con 0x7fc5540a1810 2024-09-16T05:20:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 -- 172.21.15.114:0/3565196522 >> v2:172.21.15.149:3300/0 conn(0x7fc55409b6c0 msgr2=0x7fc5540a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 --2- 172.21.15.114:0/3565196522 >> v2:172.21.15.149:3300/0 conn(0x7fc55409b6c0 0x7fc5540a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 -- 172.21.15.114:0/3565196522 >> v1:172.21.15.149:6789/0 conn(0x7fc5540a1400 legacy=0x7fc554139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: _finish_hunting 0 2024-09-16T05:20:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:20:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:20:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 -- 172.21.15.114:0/3565196522 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5540a1cc0 con 0x7fc5540a1810 2024-09-16T05:20:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: _finish_auth 0 2024-09-16T05:20:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: _check_auth_tickets 2024-09-16T05:20:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55c174640 5 monclient: authenticate success, global_id 74523 2024-09-16T05:20:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 -- 172.21.15.114:0/3565196522 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7fc54c001a20 con 0x7fc5540a1810 2024-09-16T05:20:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55c174640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:20:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: dump: 2024-09-16T05:20:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:30.675 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:30.675 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:20:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:30.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 -- 172.21.15.114:0/3565196522 >> v1:172.21.15.114:6789/0 conn(0x7fc5540a1810 legacy=0x7fc55413cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:20:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 --2- 172.21.15.114:0/3565196522 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5400039c0 0x7fc540003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient(hunting): picked mon.b con 0x7fc5400039c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 --2- 172.21.15.114:0/3565196522 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc540004810 0x7fc540004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient(hunting): picked mon.c con 0x7fc540004810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 --2- 172.21.15.114:0/3565196522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5400052f0 0x7fc54000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient(hunting): picked mon.a con 0x7fc5400052f0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 -- 172.21.15.114:0/3565196522 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc55409e390 con 0x7fc5400052f0 2024-09-16T05:20:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:30.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 -- 172.21.15.114:0/3565196522 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fc54000c1c0 con 0x7fc5400039c0 2024-09-16T05:20:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 -- 172.21.15.114:0/3565196522 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc54000c3c0 con 0x7fc540004810 2024-09-16T05:20:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:20:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 1 --2- 172.21.15.114:0/3565196522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5400052f0 0x7fc54000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55c174640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 10 monclient(hunting): get_auth_request con 0x7fc5400052f0 auth_method 0 2024-09-16T05:20:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:30.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55970f640 1 --2- 172.21.15.114:0/3565196522 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc540004810 0x7fc540004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55970f640 10 monclient(hunting): get_auth_request con 0x7fc540004810 auth_method 0 2024-09-16T05:20:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55970f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55970f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55970f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:20:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc559f10640 1 --2- 172.21.15.114:0/3565196522 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5400039c0 0x7fc540003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55970f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55970f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55970f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:20:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc559f10640 10 monclient(hunting): get_auth_request con 0x7fc5400039c0 auth_method 0 2024-09-16T05:20:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc559f10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:30.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc559f10640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc559f10640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 10 monclient(hunting): handle_auth_done global_id 74523 payload 293 2024-09-16T05:20:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 1 -- 172.21.15.114:0/3565196522 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc540004810 msgr2=0x7fc540004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 1 --2- 172.21.15.114:0/3565196522 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc540004810 0x7fc540004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 1 -- 172.21.15.114:0/3565196522 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5400039c0 msgr2=0x7fc540003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 1 --2- 172.21.15.114:0/3565196522 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5400039c0 0x7fc540003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 10 monclient: _finish_hunting 0 2024-09-16T05:20:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 1 monclient: found mon.a 2024-09-16T05:20:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 1 -- 172.21.15.114:0/3565196522 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc540003530 con 0x7fc5400052f0 2024-09-16T05:20:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 10 monclient: _finish_auth 0 2024-09-16T05:20:30.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 10 monclient: _check_auth_tickets 2024-09-16T05:20:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 1 -- 172.21.15.114:0/3565196522 reap_dead start 2024-09-16T05:20:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55a711640 1 --2- 172.21.15.114:0/3565196522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5400052f0 0x7fc54000b9d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc55409d6b0 tx=0x7fc54c01d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5cb0ca8742ea9df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 -- 172.21.15.114:0/3565196522 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54c00a4c0 con 0x7fc5400052f0 2024-09-16T05:20:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: dump: 2024-09-16T05:20:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:30.681 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:30.682 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55c174640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 -- 172.21.15.114:0/3565196522 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc54c0ad650 con 0x7fc5400052f0 2024-09-16T05:20:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55c174640 10 monclient: get_monmap_and_config success 2024-09-16T05:20:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55c174640 4 set_mon_vals no callback set 2024-09-16T05:20:30.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 1 -- 172.21.15.114:0/3565196522 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54c0ada40 con 0x7fc5400052f0 2024-09-16T05:20:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc558f0e640 10 monclient: dump: 2024-09-16T05:20:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:30.683 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55c174640 10 monclient: shutdown 2024-09-16T05:20:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55c174640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55c174640 1 -- 172.21.15.114:0/3565196522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5400052f0 msgr2=0x7fc54000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55c174640 1 --2- 172.21.15.114:0/3565196522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5400052f0 0x7fc54000b9d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc55409d6b0 tx=0x7fc54c01d3c0 comp rx=0 tx=0).stop 2024-09-16T05:20:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55c174640 1 -- 172.21.15.114:0/3565196522 shutdown_connections 2024-09-16T05:20:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55c174640 1 --2- 172.21.15.114:0/3565196522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5400052f0 0x7fc54000b9d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55c174640 1 -- 172.21.15.114:0/3565196522 >> 172.21.15.114:0/3565196522 conn(0x7fc5540a1fb0 msgr2=0x7fc5540a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55c174640 1 -- 172.21.15.114:0/3565196522 shutdown_connections 2024-09-16T05:20:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.671+0000 7fc55c174640 1 -- 172.21.15.114:0/3565196522 wait complete. 2024-09-16T05:20:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient: build_initial_monmap 2024-09-16T05:20:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient: monmap: 2024-09-16T05:20:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:20:30.684 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:20:30.685 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:20:30.685 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:30.685 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:30.685 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:20:30.685 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:20:30.685 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:20:30.685 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:30.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 librados: starting msgr at 2024-09-16T05:20:30.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 librados: starting objecter 2024-09-16T05:20:30.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 Processor -- start 2024-09-16T05:20:30.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 -- start start 2024-09-16T05:20:30.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 librados: setting wanted keys 2024-09-16T05:20:30.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 librados: calling monclient init 2024-09-16T05:20:30.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient: init 2024-09-16T05:20:30.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:30.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:20:30.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5400052f0 0x7fc55413ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:30.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient(hunting): picked mon.noname-b con 0x7fc5400052f0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:30.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5540994e0 0x7fc554099900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:30.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient(hunting): picked mon.noname-a con 0x7fc5540994e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:30.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc554099e40 0x7fc5541d5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:30.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient(hunting): picked mon.noname-c con 0x7fc554099e40 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:30.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:30.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc5540a0ec0 con 0x7fc5540994e0 2024-09-16T05:20:30.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:30.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fc55409cad0 con 0x7fc5400052f0 2024-09-16T05:20:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc5540a1040 con 0x7fc554099e40 2024-09-16T05:20:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient(hunting): authenticate will time out at 1715.305123s 2024-09-16T05:20:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5540994e0 0x7fc554099900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55a711640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc554099e40 0x7fc5541d5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5540994e0 0x7fc554099900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39616/0 (socket says 172.21.15.114:39616) 2024-09-16T05:20:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55a711640 10 monclient(hunting): get_auth_request con 0x7fc554099e40 auth_method 0 2024-09-16T05:20:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 1 -- 172.21.15.114:0/3017610982 learned_addr learned my addr 172.21.15.114:0/3017610982 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55a711640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55a711640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55a711640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 10 monclient(hunting): get_auth_request con 0x7fc5540994e0 auth_method 0 2024-09-16T05:20:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:30.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:30.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:30.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:30.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:30.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc559f10640 1 --2- 172.21.15.114:0/3017610982 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5400052f0 0x7fc55413ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:30.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55a711640 1 -- 172.21.15.114:0/3017610982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc554099e40 msgr2=0x7fc5541d5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T05:20:30.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc559f10640 10 monclient(hunting): get_auth_request con 0x7fc5400052f0 auth_method 0 2024-09-16T05:20:30.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55a711640 1 -- 172.21.15.114:0/3017610982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc554099e40 msgr2=0x7fc5541d5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:20:30.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc559f10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:30.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc559f10640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:30.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc559f10640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:30.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55a711640 1 --2- 172.21.15.114:0/3017610982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc554099e40 0x7fc5541d5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:30.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55a711640 1 --2- 172.21.15.114:0/3017610982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc554099e40 0x7fc5541d5a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:20:30.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 10 monclient(hunting): handle_auth_done global_id 74526 payload 824 2024-09-16T05:20:30.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 1 -- 172.21.15.114:0/3017610982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc554099e40 msgr2=0x7fc5541d5a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:20:30.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 1 --2- 172.21.15.114:0/3017610982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc554099e40 0x7fc5541d5a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:30.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 1 -- 172.21.15.114:0/3017610982 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5400052f0 msgr2=0x7fc55413ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:30.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 1 --2- 172.21.15.114:0/3017610982 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5400052f0 0x7fc55413ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:30.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 10 monclient: _finish_hunting 0 2024-09-16T05:20:30.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 1 monclient: found mon.noname-a 2024-09-16T05:20:30.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:30.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:30.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 1 -- 172.21.15.114:0/3017610982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc55413b310 con 0x7fc5540994e0 2024-09-16T05:20:30.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc559f10640 1 --2- 172.21.15.114:0/3017610982 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5400052f0 0x7fc55413ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:20:30.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55970f640 1 --2- 172.21.15.114:0/3017610982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5540994e0 0x7fc554099900 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc54400f5f0 tx=0x7fc54400fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:30.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 10 client.?.objecter ms_handle_connect 0x7fc5540994e0 2024-09-16T05:20:30.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:20:30.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 1 -- 172.21.15.114:0/3017610982 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5440052e0 con 0x7fc5540994e0 2024-09-16T05:20:30.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:30.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:30.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 10 monclient: dump: 2024-09-16T05:20:30.694 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:30.694 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:30.694 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:30.694 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:30.694 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:30.694 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:30.694 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:30.694 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:30.695 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:30.695 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:30.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:20:30.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:20:30.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:30.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 1 -- 172.21.15.114:0/3017610982 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc544018070 con 0x7fc5540994e0 2024-09-16T05:20:30.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 5 monclient: authenticate success, global_id 74526 2024-09-16T05:20:30.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient: _renew_subs 2024-09-16T05:20:30.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:30.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 -- 172.21.15.114:0/3017610982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc55409af90 con 0x7fc5540994e0 2024-09-16T05:20:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 1 -- 172.21.15.114:0/3017610982 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc544003870 con 0x7fc5540994e0 2024-09-16T05:20:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 10 monclient: dump: 2024-09-16T05:20:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:30.697 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54bfff640 4 set_mon_vals no callback set 2024-09-16T05:20:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 client.74526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:20:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient: _renew_subs 2024-09-16T05:20:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 -- 172.21.15.114:0/3017610982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc55413b030 con 0x7fc5540994e0 2024-09-16T05:20:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 librados: init done 2024-09-16T05:20:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc55c174640 1 -- 172.21.15.114:0/3017610982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5541367a0 con 0x7fc5540994e0 2024-09-16T05:20:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 1 -- 172.21.15.114:0/3017610982 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc544003a10 con 0x7fc5540994e0 2024-09-16T05:20:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 1 --2- 172.21.15.114:0/3017610982 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc538037810 0x7fc538039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 1 -- 172.21.15.114:0/3017610982 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(521..521 src has 1..521) ==== 5861+0+0 (secure 0 0 0) 0x7fc544048f10 con 0x7fc5540994e0 2024-09-16T05:20:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 10 client.74526.objecter ms_dispatch 0x7fc554000ba0 osd_map(521..521 src has 1..521) 2024-09-16T05:20:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 3 client.74526.objecter handle_osd_map got epochs [521,521] > 0 2024-09-16T05:20:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 3 client.74526.objecter handle_osd_map decoding full epoch 521 2024-09-16T05:20:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc54a7fc640 20 client.74526.objecter dump_active .. 0 homeless 2024-09-16T05:20:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc559f10640 1 --2- 172.21.15.114:0/3017610982 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc538037810 0x7fc538039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.675+0000 7fc559f10640 10 monclient: get_auth_request con 0x7fc538037810 auth_method 0 2024-09-16T05:20:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.679+0000 7fc54a7fc640 10 client.74526.objecter ms_handle_connect 0x7fc538037810 2024-09-16T05:20:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.679+0000 7fc559f10640 1 --2- 172.21.15.114:0/3017610982 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc538037810 0x7fc538039cd0 secure :-1 s=READY pgs=3560 cs=0 l=1 rev1=1 crypto rx=0x7fc550009a70 tx=0x7fc550008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.679+0000 7fc54a7fc640 1 -- 172.21.15.114:0/3017610982 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc544013c70 con 0x7fc5540994e0 2024-09-16T05:20:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.679+0000 7fc54a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.679+0000 7fc54a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:20:30.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.903+0000 7fc55c174640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:20:30.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.903+0000 7fc55c174640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:20:30.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.903+0000 7fc55c174640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:30.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.903+0000 7fc55c174640 1 -- 172.21.15.114:0/3017610982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc55409a920 con 0x7fc5540994e0 2024-09-16T05:20:30.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.903+0000 7fc54a7fc640 1 -- 172.21.15.114:0/3017610982 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fc54404a030 con 0x7fc5540994e0 2024-09-16T05:20:30.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.903+0000 7fc54a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:20:30.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.903+0000 7fc54a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:20:30.902 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":97,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:20:30.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 10 librados: watch_flush enter 2024-09-16T05:20:30.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 10 librados: watch_flush exit 2024-09-16T05:20:30.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 20 client.74526.objecter shutdown clearing up homeless session... 2024-09-16T05:20:30.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 10 client.74526.objecter successfully canceled tick 2024-09-16T05:20:30.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 1 -- 172.21.15.114:0/3017610982 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc538037810 msgr2=0x7fc538039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:30.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 1 --2- 172.21.15.114:0/3017610982 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc538037810 0x7fc538039cd0 secure :-1 s=READY pgs=3560 cs=0 l=1 rev1=1 crypto rx=0x7fc550009a70 tx=0x7fc550008040 comp rx=0 tx=0).stop 2024-09-16T05:20:30.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 10 monclient: shutdown 2024-09-16T05:20:30.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:30.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 1 -- 172.21.15.114:0/3017610982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5540994e0 msgr2=0x7fc554099900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:30.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 1 --2- 172.21.15.114:0/3017610982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5540994e0 0x7fc554099900 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc54400f5f0 tx=0x7fc54400fab0 comp rx=0 tx=0).stop 2024-09-16T05:20:30.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 1 -- 172.21.15.114:0/3017610982 shutdown_connections 2024-09-16T05:20:30.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 1 --2- 172.21.15.114:0/3017610982 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc538037810 0x7fc538039cd0 unknown :-1 s=CLOSED pgs=3560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:30.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 1 --2- 172.21.15.114:0/3017610982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5540994e0 0x7fc554099900 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:30.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 1 --2- 172.21.15.114:0/3017610982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc554099e40 0x7fc5541d5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:30.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 1 --2- 172.21.15.114:0/3017610982 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5400052f0 0x7fc55413ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:30.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 1 -- 172.21.15.114:0/3017610982 >> 172.21.15.114:0/3017610982 conn(0x7fc5540a1fb0 msgr2=0x7fc554071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:30.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 1 -- 172.21.15.114:0/3017610982 shutdown_connections 2024-09-16T05:20:30.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 1 -- 172.21.15.114:0/3017610982 wait complete. 2024-09-16T05:20:30.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:30.907+0000 7fc55c174640 1 librados: shutdown 2024-09-16T05:20:33.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:20:34.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 10 monclient: get_monmap_and_config 2024-09-16T05:20:34.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 10 monclient: build_initial_monmap 2024-09-16T05:20:34.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 10 monclient: monmap: 2024-09-16T05:20:34.049 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:34.049 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:34.049 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:20:34.053739+0000 2024-09-16T05:20:34.049 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:20:34.053739+0000 2024-09-16T05:20:34.049 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:34.049 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:34.049 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:20:34.049 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:20:34.049 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:20:34.049 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:20:34.050 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:20:34.050 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:34.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 1 Processor -- start 2024-09-16T05:20:34.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 1 -- start start 2024-09-16T05:20:34.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 10 monclient: init 2024-09-16T05:20:34.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:34.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T05:20:34.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb67409b6c0 0x7fb6740a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:34.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 10 monclient(hunting): picked mon.noname-b con 0x7fb67409b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:34.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb6740a1400 0x7fb6740a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:34.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 10 monclient(hunting): picked mon.noname-c con 0x7fb6740a1400 addr v2:172.21.15.149:3300/0 2024-09-16T05:20:34.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb674139640 addr v1:172.21.15.114:6789/0 2024-09-16T05:20:34.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb67409d4e0 con 0x7fb674139640 2024-09-16T05:20:34.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:34.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fb67409e390 con 0x7fb6740a1400 2024-09-16T05:20:34.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:34.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb67409ece0 con 0x7fb67409b6c0 2024-09-16T05:20:34.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:34.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 10 monclient(hunting): authenticate will time out at 1718.682972s 2024-09-16T05:20:34.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6799d4640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb674139640 0x7fb67413ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:41260/0 (socket says 172.21.15.114:41260) 2024-09-16T05:20:34.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6799d4640 1 -- 172.21.15.114:0/2818182157 learned_addr learned my addr 172.21.15.114:0/2818182157 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:20:34.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6791d3640 1 --2- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb67409b6c0 0x7fb6740a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:34.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6791d3640 10 monclient(hunting): get_auth_request con 0x7fb67409b6c0 auth_method 0 2024-09-16T05:20:34.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6791d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:34.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6791d3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:34.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6791d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:34.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6791d3640 1 -- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb67409b6c0 msgr2=0x7fb6740a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:20:34.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6791d3640 1 -- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb67409b6c0 msgr2=0x7fb6740a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:20:34.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6791d3640 1 --2- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb67409b6c0 0x7fb6740a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:34.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6791d3640 1 --2- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb67409b6c0 0x7fb6740a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:20:34.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6789d2640 1 --2- 172.21.15.114:0/2818182157 >> v2:172.21.15.149:3300/0 conn(0x7fb6740a1400 0x7fb6740a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:34.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6789d2640 10 monclient(hunting): get_auth_request con 0x7fb6740a1400 auth_method 0 2024-09-16T05:20:34.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6789d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:34.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6789d2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:34.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6789d2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:34.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 1 -- 172.21.15.114:0/2818182157 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2253277137 0 0) 0x7fb67409d4e0 con 0x7fb674139640 2024-09-16T05:20:34.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 10 monclient(hunting): _negotiate 2024-09-16T05:20:34.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:34.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:34.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 10 monclient(hunting): my global_id is 74529 2024-09-16T05:20:34.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 1 -- 172.21.15.114:0/2818182157 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb65c003530 con 0x7fb674139640 2024-09-16T05:20:34.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6789d2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:34.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6789d2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:34.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb6789d2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:34.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 1 -- 172.21.15.114:0/2818182157 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 376924076 0 0) 0x7fb65c003530 con 0x7fb674139640 2024-09-16T05:20:34.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 1 -- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb67409b6c0 msgr2=0x7fb6740a0ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:20:34.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 1 --2- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb67409b6c0 0x7fb6740a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:34.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 1 -- 172.21.15.114:0/2818182157 >> v2:172.21.15.149:3300/0 conn(0x7fb6740a1400 msgr2=0x7fb6740a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:34.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 1 --2- 172.21.15.114:0/2818182157 >> v2:172.21.15.149:3300/0 conn(0x7fb6740a1400 0x7fb6740a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:34.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 10 monclient: _finish_hunting 0 2024-09-16T05:20:34.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:20:34.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:34.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:20:34.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 1 -- 172.21.15.114:0/2818182157 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67409f430 con 0x7fb674139640 2024-09-16T05:20:34.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 10 monclient: _finish_auth 0 2024-09-16T05:20:34.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 10 monclient: _check_auth_tickets 2024-09-16T05:20:34.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:34.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 1 -- 172.21.15.114:0/2818182157 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7fb6640016a0 con 0x7fb674139640 2024-09-16T05:20:34.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 5 monclient: authenticate success, global_id 74529 2024-09-16T05:20:34.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb67b437640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:34.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:34.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:20:34.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 10 monclient: dump: 2024-09-16T05:20:34.065 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:34.065 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:34.065 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:34.065 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:34.065 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:34.065 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:34.065 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:34.065 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:34.065 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:34.065 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:34.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:20:34.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:34.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 1 -- 172.21.15.114:0/2818182157 >> v1:172.21.15.114:6789/0 conn(0x7fb674139640 legacy=0x7fb67413ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:34.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:20:34.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 1 --2- 172.21.15.114:0/2818182157 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb65c0039c0 0x7fb65c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:34.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 10 monclient(hunting): picked mon.b con 0x7fb65c0039c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:34.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 1 --2- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb65c004810 0x7fb65c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:34.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.051+0000 7fb673fff640 10 monclient(hunting): picked mon.a con 0x7fb65c004810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:34.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 1 --2- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb67409b6c0 0x7fb65c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:34.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 10 monclient(hunting): picked mon.c con 0x7fb67409b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:34.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:34.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 1 -- 172.21.15.114:0/2818182157 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb67409e390 con 0x7fb65c004810 2024-09-16T05:20:34.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:34.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 1 -- 172.21.15.114:0/2818182157 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb67409ece0 con 0x7fb65c0039c0 2024-09-16T05:20:34.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:34.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 1 -- 172.21.15.114:0/2818182157 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb65c00bb00 con 0x7fb67409b6c0 2024-09-16T05:20:34.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:34.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:20:34.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb67b437640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:34.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 1 --2- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb65c004810 0x7fb65c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:34.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 10 monclient(hunting): get_auth_request con 0x7fb65c004810 auth_method 0 2024-09-16T05:20:34.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:34.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6799d4640 1 --2- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb67409b6c0 0x7fb65c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:34.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:34.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:34.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6799d4640 10 monclient(hunting): get_auth_request con 0x7fb67409b6c0 auth_method 0 2024-09-16T05:20:34.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6799d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:34.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6799d4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:34.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6799d4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:34.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:34.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:34.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:20:34.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6799d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:34.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6799d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:34.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6799d4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:20:34.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6791d3640 1 --2- 172.21.15.114:0/2818182157 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb65c0039c0 0x7fb65c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:34.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 10 monclient(hunting): handle_auth_done global_id 74529 payload 293 2024-09-16T05:20:34.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 1 -- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb67409b6c0 msgr2=0x7fb65c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:34.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 1 --2- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb67409b6c0 0x7fb65c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:34.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 1 -- 172.21.15.114:0/2818182157 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb65c0039c0 msgr2=0x7fb65c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:34.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 1 --2- 172.21.15.114:0/2818182157 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb65c0039c0 0x7fb65c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:34.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 10 monclient: _finish_hunting 0 2024-09-16T05:20:34.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 1 monclient: found mon.a 2024-09-16T05:20:34.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:34.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:34.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 1 -- 172.21.15.114:0/2818182157 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65c003530 con 0x7fb65c004810 2024-09-16T05:20:34.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 10 monclient: _finish_auth 0 2024-09-16T05:20:34.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 10 monclient: _check_auth_tickets 2024-09-16T05:20:34.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:34.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6791d3640 10 monclient: get_auth_request con 0x7fb65c0039c0 auth_method 0 2024-09-16T05:20:34.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6791d3640 1 --2- 172.21.15.114:0/2818182157 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb65c0039c0 0x7fb65c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:20:34.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6789d2640 1 --2- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb65c004810 0x7fb65c004c90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb668009ac0 tx=0x7fb66801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c075c556660962c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:34.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 1 -- 172.21.15.114:0/2818182157 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6680ac070 con 0x7fb65c004810 2024-09-16T05:20:34.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:34.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:34.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 10 monclient: dump: 2024-09-16T05:20:34.070 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:34.070 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:34.070 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:34.070 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:34.070 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:34.070 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:34.070 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:34.070 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:34.070 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:34.070 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:34.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 1 -- 172.21.15.114:0/2818182157 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6680059a0 con 0x7fb65c004810 2024-09-16T05:20:34.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb67b437640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:34.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:34.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 1 -- 172.21.15.114:0/2818182157 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668009690 con 0x7fb65c004810 2024-09-16T05:20:34.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb67b437640 10 monclient: get_monmap_and_config success 2024-09-16T05:20:34.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb67b437640 4 set_mon_vals no callback set 2024-09-16T05:20:34.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:34.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:34.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb673fff640 10 monclient: dump: 2024-09-16T05:20:34.071 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:34.071 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:34.071 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:34.071 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:34.071 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:34.072 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:34.072 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:34.072 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:34.072 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:34.072 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:34.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb67b437640 10 monclient: shutdown 2024-09-16T05:20:34.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb67b437640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:34.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb67b437640 1 -- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb65c004810 msgr2=0x7fb65c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:34.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb67b437640 1 --2- 172.21.15.114:0/2818182157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb65c004810 0x7fb65c004c90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb668009ac0 tx=0x7fb66801ca70 comp rx=0 tx=0).stop 2024-09-16T05:20:34.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb6799d4640 1 -- 172.21.15.114:0/2818182157 reap_dead start 2024-09-16T05:20:34.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb67b437640 1 -- 172.21.15.114:0/2818182157 shutdown_connections 2024-09-16T05:20:34.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb67b437640 1 -- 172.21.15.114:0/2818182157 >> 172.21.15.114:0/2818182157 conn(0x7fb6740a1fb0 msgr2=0x7fb6740a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:34.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb67b437640 1 -- 172.21.15.114:0/2818182157 shutdown_connections 2024-09-16T05:20:34.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.055+0000 7fb67b437640 1 -- 172.21.15.114:0/2818182157 wait complete. 2024-09-16T05:20:34.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient: build_initial_monmap 2024-09-16T05:20:34.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient: monmap: 2024-09-16T05:20:34.073 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:34.073 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:20:34.073 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:20:34.073 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:20:34.073 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:34.073 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:34.073 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:20:34.073 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:20:34.073 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:20:34.073 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:34.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 librados: starting msgr at 2024-09-16T05:20:34.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 librados: starting objecter 2024-09-16T05:20:34.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 Processor -- start 2024-09-16T05:20:34.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 -- start start 2024-09-16T05:20:34.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 librados: setting wanted keys 2024-09-16T05:20:34.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 librados: calling monclient init 2024-09-16T05:20:34.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient: init 2024-09-16T05:20:34.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:34.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:20:34.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c004810 0x7fb674073020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:34.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient(hunting): picked mon.noname-c con 0x7fb65c004810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:34.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6740994b0 0x7fb6740998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:34.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient(hunting): picked mon.noname-b con 0x7fb6740994b0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:34.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb674099df0 0x7fb6741d5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:34.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient(hunting): picked mon.noname-a con 0x7fb674099df0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:34.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:34.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb67409cad0 con 0x7fb674099df0 2024-09-16T05:20:34.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:34.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb67409baa0 con 0x7fb6740994b0 2024-09-16T05:20:34.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:34.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb67413d960 con 0x7fb65c004810 2024-09-16T05:20:34.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:34.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient(hunting): authenticate will time out at 1718.688971s 2024-09-16T05:20:34.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6791d3640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c004810 0x7fb674073020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:34.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb674099df0 0x7fb6741d5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:34.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6791d3640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c004810 0x7fb674073020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:42402/0 (socket says 172.21.15.114:42402) 2024-09-16T05:20:34.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6791d3640 1 -- 172.21.15.114:0/880895404 learned_addr learned my addr 172.21.15.114:0/880895404 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:34.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6791d3640 10 monclient(hunting): get_auth_request con 0x7fb65c004810 auth_method 0 2024-09-16T05:20:34.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb674099df0 0x7fb6741d5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39628/0 (socket says 172.21.15.114:39628) 2024-09-16T05:20:34.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6791d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:34.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6791d3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:34.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6791d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:34.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 10 monclient(hunting): get_auth_request con 0x7fb674099df0 auth_method 0 2024-09-16T05:20:34.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:34.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:34.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:34.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:34.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:34.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6791d3640 1 -- 172.21.15.114:0/880895404 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c004810 msgr2=0x7fb674073020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:20:34.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6791d3640 1 -- 172.21.15.114:0/880895404 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c004810 msgr2=0x7fb674073020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:20:34.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6791d3640 1 --2- 172.21.15.114:0/880895404 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c004810 0x7fb674073020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:34.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:34.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6791d3640 1 --2- 172.21.15.114:0/880895404 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c004810 0x7fb674073020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:20:34.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6789d2640 1 --2- 172.21.15.114:0/880895404 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6740994b0 0x7fb6740998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:34.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6789d2640 10 monclient(hunting): get_auth_request con 0x7fb6740994b0 auth_method 0 2024-09-16T05:20:34.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6789d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:34.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6789d2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:34.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6789d2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:34.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 10 monclient(hunting): handle_auth_done global_id 74532 payload 824 2024-09-16T05:20:34.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 1 -- 172.21.15.114:0/880895404 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c004810 msgr2=0x7fb674073020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:20:34.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 1 --2- 172.21.15.114:0/880895404 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c004810 0x7fb674073020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:34.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 1 -- 172.21.15.114:0/880895404 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6740994b0 msgr2=0x7fb6740998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:34.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 1 --2- 172.21.15.114:0/880895404 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6740994b0 0x7fb6740998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:34.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 10 monclient: _finish_hunting 0 2024-09-16T05:20:34.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 1 monclient: found mon.noname-a 2024-09-16T05:20:34.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:34.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:34.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 1 -- 172.21.15.114:0/880895404 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb674072640 con 0x7fb674099df0 2024-09-16T05:20:34.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6799d4640 1 --2- 172.21.15.114:0/880895404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb674099df0 0x7fb6741d5aa0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb67409d780 tx=0x7fb664007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:34.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 10 client.?.objecter ms_handle_connect 0x7fb674099df0 2024-09-16T05:20:34.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:20:34.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 1 -- 172.21.15.114:0/880895404 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664019070 con 0x7fb674099df0 2024-09-16T05:20:34.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:34.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:34.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 10 monclient: dump: 2024-09-16T05:20:34.079 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:34.080 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 10 monclient: _finish_auth 0 2024-09-16T05:20:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:20:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 1 -- 172.21.15.114:0/880895404 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb664011580 con 0x7fb674099df0 2024-09-16T05:20:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 5 monclient: authenticate success, global_id 74532 2024-09-16T05:20:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient: _renew_subs 2024-09-16T05:20:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 -- 172.21.15.114:0/880895404 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb674072c00 con 0x7fb674099df0 2024-09-16T05:20:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 1 -- 172.21.15.114:0/880895404 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664011970 con 0x7fb674099df0 2024-09-16T05:20:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 10 monclient: dump: 2024-09-16T05:20:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:34.082 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:34.082 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:34.082 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:34.082 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:34.082 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:34.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6737fe640 4 set_mon_vals no callback set 2024-09-16T05:20:34.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 client.74532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:20:34.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient: _renew_subs 2024-09-16T05:20:34.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:34.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 -- 172.21.15.114:0/880895404 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb67409a230 con 0x7fb674099df0 2024-09-16T05:20:34.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 librados: init done 2024-09-16T05:20:34.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:34.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:34.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:34.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb67b437640 1 -- 172.21.15.114:0/880895404 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6740a17e0 con 0x7fb674099df0 2024-09-16T05:20:34.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 1 -- 172.21.15.114:0/880895404 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb664011b10 con 0x7fb674099df0 2024-09-16T05:20:34.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 1 --2- 172.21.15.114:0/880895404 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb660037810 0x7fb660039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:34.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 1 -- 172.21.15.114:0/880895404 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(524..524 src has 1..524) ==== 5853+0+0 (secure 0 0 0) 0x7fb6640485b0 con 0x7fb674099df0 2024-09-16T05:20:34.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 10 client.74532.objecter ms_dispatch 0x7fb674000ba0 osd_map(524..524 src has 1..524) 2024-09-16T05:20:34.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 3 client.74532.objecter handle_osd_map got epochs [524,524] > 0 2024-09-16T05:20:34.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 3 client.74532.objecter handle_osd_map decoding full epoch 524 2024-09-16T05:20:34.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb671ffb640 20 client.74532.objecter dump_active .. 0 homeless 2024-09-16T05:20:34.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6791d3640 1 --2- 172.21.15.114:0/880895404 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb660037810 0x7fb660039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:34.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.059+0000 7fb6791d3640 10 monclient: get_auth_request con 0x7fb660037810 auth_method 0 2024-09-16T05:20:34.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.063+0000 7fb671ffb640 10 client.74532.objecter ms_handle_connect 0x7fb660037810 2024-09-16T05:20:34.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.063+0000 7fb6791d3640 1 --2- 172.21.15.114:0/880895404 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb660037810 0x7fb660039cd0 secure :-1 s=READY pgs=3561 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0097e0 tx=0x7fb66c00b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:34.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.063+0000 7fb671ffb640 1 -- 172.21.15.114:0/880895404 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb66401b310 con 0x7fb674099df0 2024-09-16T05:20:34.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.063+0000 7fb671ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:34.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.063+0000 7fb671ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:20:34.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.303+0000 7fb67b437640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:20:34.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.303+0000 7fb67b437640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:20:34.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.303+0000 7fb67b437640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:34.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.303+0000 7fb67b437640 1 -- 172.21.15.114:0/880895404 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb67409c6a0 con 0x7fb674099df0 2024-09-16T05:20:34.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.303+0000 7fb671ffb640 1 -- 172.21.15.114:0/880895404 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7fb66401be30 con 0x7fb674099df0 2024-09-16T05:20:34.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.303+0000 7fb671ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:20:34.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.303+0000 7fb671ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:20:34.303 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":101,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:20:34.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 10 librados: watch_flush enter 2024-09-16T05:20:34.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 10 librados: watch_flush exit 2024-09-16T05:20:34.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 20 client.74532.objecter shutdown clearing up homeless session... 2024-09-16T05:20:34.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 10 client.74532.objecter successfully canceled tick 2024-09-16T05:20:34.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 1 -- 172.21.15.114:0/880895404 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb660037810 msgr2=0x7fb660039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:34.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 1 --2- 172.21.15.114:0/880895404 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb660037810 0x7fb660039cd0 secure :-1 s=READY pgs=3561 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0097e0 tx=0x7fb66c00b040 comp rx=0 tx=0).stop 2024-09-16T05:20:34.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 10 monclient: shutdown 2024-09-16T05:20:34.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:34.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 1 -- 172.21.15.114:0/880895404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb674099df0 msgr2=0x7fb6741d5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:34.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 1 --2- 172.21.15.114:0/880895404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb674099df0 0x7fb6741d5aa0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb67409d780 tx=0x7fb664007430 comp rx=0 tx=0).stop 2024-09-16T05:20:34.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 1 -- 172.21.15.114:0/880895404 shutdown_connections 2024-09-16T05:20:34.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 1 --2- 172.21.15.114:0/880895404 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb660037810 0x7fb660039cd0 unknown :-1 s=CLOSED pgs=3561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:34.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 1 --2- 172.21.15.114:0/880895404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb674099df0 0x7fb6741d5aa0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:34.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 1 --2- 172.21.15.114:0/880895404 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6740994b0 0x7fb6740998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:34.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 1 --2- 172.21.15.114:0/880895404 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb65c004810 0x7fb674073020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:34.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 1 -- 172.21.15.114:0/880895404 >> 172.21.15.114:0/880895404 conn(0x7fb6740a1fb0 msgr2=0x7fb674098560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:34.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 1 -- 172.21.15.114:0/880895404 shutdown_connections 2024-09-16T05:20:34.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 1 -- 172.21.15.114:0/880895404 wait complete. 2024-09-16T05:20:34.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:34.307+0000 7fb67b437640 1 librados: shutdown 2024-09-16T05:20:37.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:20:37.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 10 monclient: get_monmap_and_config 2024-09-16T05:20:37.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 10 monclient: build_initial_monmap 2024-09-16T05:20:37.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 10 monclient: monmap: 2024-09-16T05:20:37.452 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:37.452 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:37.452 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:20:37.456864+0000 2024-09-16T05:20:37.452 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:20:37.456864+0000 2024-09-16T05:20:37.452 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:37.452 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:37.452 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:20:37.452 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:20:37.452 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:20:37.452 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:20:37.452 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:20:37.453 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:37.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 1 Processor -- start 2024-09-16T05:20:37.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 1 -- start start 2024-09-16T05:20:37.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 10 monclient: init 2024-09-16T05:20:37.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:37.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T05:20:37.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f96b0137480 0x7f96b0137860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 10 monclient(hunting): picked mon.noname-c con 0x7f96b0137480 addr v2:172.21.15.149:3300/0 2024-09-16T05:20:37.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96b00a57b0 addr v1:172.21.15.114:6789/0 2024-09-16T05:20:37.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f96b00a4980 0x7f96b013f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 10 monclient(hunting): picked mon.noname-a con 0x7f96b00a4980 addr v2:172.21.15.114:3300/0 2024-09-16T05:20:37.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96b009cfa0 con 0x7f96b00a57b0 2024-09-16T05:20:37.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:37.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f96b00a52f0 con 0x7f96b00a4980 2024-09-16T05:20:37.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:37.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f96b00a5570 con 0x7f96b0137480 2024-09-16T05:20:37.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:37.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b7e0a640 10 monclient(hunting): authenticate will time out at 1722.086563s 2024-09-16T05:20:37.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f96b00a4980 0x7f96b013f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f96b00a4980 0x7f96b013f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53420/0 (socket says 172.21.15.114:53420) 2024-09-16T05:20:37.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b53a5640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f96b00a57b0 0x7f96b00a4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36722/0 (socket says 172.21.15.114:36722) 2024-09-16T05:20:37.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 1 -- 172.21.15.114:0/726378735 learned_addr learned my addr 172.21.15.114:0/726378735 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:37.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 10 monclient(hunting): get_auth_request con 0x7f96b00a4980 auth_method 0 2024-09-16T05:20:37.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:37.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:37.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:37.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:37.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:37.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:37.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b5ba6640 1 --2- 172.21.15.114:0/726378735 >> v2:172.21.15.149:3300/0 conn(0x7f96b0137480 0x7f96b0137860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 10 monclient(hunting): handle_auth_done global_id 74535 payload 274 2024-09-16T05:20:37.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 1 -- 172.21.15.114:0/726378735 >> v2:172.21.15.149:3300/0 conn(0x7f96b0137480 msgr2=0x7f96b0137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:37.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 1 --2- 172.21.15.114:0/726378735 >> v2:172.21.15.149:3300/0 conn(0x7f96b0137480 0x7f96b0137860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 1 -- 172.21.15.114:0/726378735 >> v1:172.21.15.114:6789/0 conn(0x7f96b00a57b0 legacy=0x7f96b00a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:37.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 10 monclient: _finish_hunting 0 2024-09-16T05:20:37.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 1 monclient: found mon.noname-a 2024-09-16T05:20:37.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:37.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:37.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b63a7640 1 -- 172.21.15.114:0/726378735 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b009bd20 con 0x7f96b00a4980 2024-09-16T05:20:37.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b4ba4640 1 -- 172.21.15.114:0/726378735 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 358317507 0 0) 0x7f96ac0014a0 con 0x7f96b00a57b0 2024-09-16T05:20:37.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b5ba6640 10 monclient: get_auth_request con 0x7f96b0137480 auth_method 0 2024-09-16T05:20:37.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b4ba4640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:20:37.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.455+0000 7f96b5ba6640 1 --2- 172.21.15.114:0/726378735 >> v2:172.21.15.149:3300/0 conn(0x7f96b0137480 0x7f96b0137860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:20:37.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b63a7640 1 --2- 172.21.15.114:0/726378735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96b00a4980 0x7f96b013f370 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f96b00a7430 tx=0x7f96a801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a429ef98e9df9fc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:37.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b4ba4640 1 -- 172.21.15.114:0/726378735 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a80b1070 con 0x7f96b00a4980 2024-09-16T05:20:37.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b4ba4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:37.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b4ba4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:37.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b4ba4640 10 monclient: dump: 2024-09-16T05:20:37.463 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:37.463 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:37.463 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.463 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.464 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:37.464 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:37.464 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:37.464 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:37.464 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:37.464 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:37.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b4ba4640 10 monclient: _finish_auth 0 2024-09-16T05:20:37.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b4ba4640 10 monclient: _check_auth_tickets 2024-09-16T05:20:37.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b4ba4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:37.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 5 monclient: authenticate success, global_id 74535 2024-09-16T05:20:37.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:37.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b4ba4640 1 -- 172.21.15.114:0/726378735 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96a80ac640 con 0x7f96b00a4980 2024-09-16T05:20:37.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b4ba4640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:37.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 10 monclient: get_monmap_and_config success 2024-09-16T05:20:37.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 4 set_mon_vals no callback set 2024-09-16T05:20:37.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b4ba4640 1 -- 172.21.15.114:0/726378735 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a80aca10 con 0x7f96b00a4980 2024-09-16T05:20:37.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b4ba4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:37.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b4ba4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:37.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b4ba4640 10 monclient: dump: 2024-09-16T05:20:37.465 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:37.465 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:37.465 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.465 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.465 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:37.465 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:37.465 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:37.465 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:37.465 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:37.466 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:37.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 10 monclient: shutdown 2024-09-16T05:20:37.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:37.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 -- 172.21.15.114:0/726378735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96b00a4980 msgr2=0x7f96b013f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:37.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 --2- 172.21.15.114:0/726378735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96b00a4980 0x7f96b013f370 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f96b00a7430 tx=0x7f96a801c3b0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 -- 172.21.15.114:0/726378735 shutdown_connections 2024-09-16T05:20:37.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 --2- 172.21.15.114:0/726378735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96b00a4980 0x7f96b013f370 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 --2- 172.21.15.114:0/726378735 >> v2:172.21.15.149:3300/0 conn(0x7f96b0137480 0x7f96b0137860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 -- 172.21.15.114:0/726378735 >> 172.21.15.114:0/726378735 conn(0x7f96b00a9c40 msgr2=0x7f96b00aa050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:37.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 -- 172.21.15.114:0/726378735 shutdown_connections 2024-09-16T05:20:37.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 -- 172.21.15.114:0/726378735 wait complete. 2024-09-16T05:20:37.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 10 monclient: build_initial_monmap 2024-09-16T05:20:37.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 10 monclient: monmap: 2024-09-16T05:20:37.467 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:37.467 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:20:37.467 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:20:37.467 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:20:37.467 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:37.467 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:37.467 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:20:37.468 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:20:37.468 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:20:37.468 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:37.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 librados: starting msgr at 2024-09-16T05:20:37.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 librados: starting objecter 2024-09-16T05:20:37.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 Processor -- start 2024-09-16T05:20:37.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 -- start start 2024-09-16T05:20:37.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 librados: setting wanted keys 2024-09-16T05:20:37.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 librados: calling monclient init 2024-09-16T05:20:37.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 10 monclient: init 2024-09-16T05:20:37.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:37.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:20:37.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b00a4980 0x7f96b0135b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 10 monclient(hunting): picked mon.noname-c con 0x7f96b00a4980 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:37.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96b00a57b0 0x7f96b0136070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 10 monclient(hunting): picked mon.noname-a con 0x7f96b00a57b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:37.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f96b0137480 0x7f96b013be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 10 monclient(hunting): picked mon.noname-b con 0x7f96b0137480 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:37.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:37.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f96b0072a90 con 0x7f96b00a57b0 2024-09-16T05:20:37.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:37.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f96b0072910 con 0x7f96b0137480 2024-09-16T05:20:37.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:37.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f96b01420d0 con 0x7f96b00a4980 2024-09-16T05:20:37.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:37.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b7e0a640 10 monclient(hunting): authenticate will time out at 1722.091533s 2024-09-16T05:20:37.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96b00a57b0 0x7f96b0136070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b5ba6640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b00a4980 0x7f96b0135b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96b00a57b0 0x7f96b0136070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53430/0 (socket says 172.21.15.114:53430) 2024-09-16T05:20:37.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 1 -- 172.21.15.114:0/2828813755 learned_addr learned my addr 172.21.15.114:0/2828813755 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:37.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b5ba6640 10 monclient(hunting): get_auth_request con 0x7f96b00a4980 auth_method 0 2024-09-16T05:20:37.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b5ba6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:37.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b5ba6640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:37.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b5ba6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:37.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 10 monclient(hunting): get_auth_request con 0x7f96b00a57b0 auth_method 0 2024-09-16T05:20:37.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:37.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:37.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:37.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:37.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:37.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:37.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b5ba6640 1 -- 172.21.15.114:0/2828813755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b00a4980 msgr2=0x7f96b0135b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:20:37.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b5ba6640 1 -- 172.21.15.114:0/2828813755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b00a4980 msgr2=0x7f96b0135b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:20:37.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b5ba6640 1 --2- 172.21.15.114:0/2828813755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b00a4980 0x7f96b0135b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:37.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b5ba6640 1 --2- 172.21.15.114:0/2828813755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b00a4980 0x7f96b0135b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:20:37.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b63a7640 1 --2- 172.21.15.114:0/2828813755 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f96b0137480 0x7f96b013be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 10 monclient(hunting): handle_auth_done global_id 74541 payload 824 2024-09-16T05:20:37.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 1 -- 172.21.15.114:0/2828813755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b00a4980 msgr2=0x7f96b0135b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:20:37.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 1 --2- 172.21.15.114:0/2828813755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b00a4980 0x7f96b0135b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 1 -- 172.21.15.114:0/2828813755 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f96b0137480 msgr2=0x7f96b013be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:37.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 1 --2- 172.21.15.114:0/2828813755 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f96b0137480 0x7f96b013be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 10 monclient: _finish_hunting 0 2024-09-16T05:20:37.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 1 monclient: found mon.noname-a 2024-09-16T05:20:37.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:37.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:37.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b53a5640 1 -- 172.21.15.114:0/2828813755 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b0120860 con 0x7f96b00a57b0 2024-09-16T05:20:37.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b63a7640 10 monclient: get_auth_request con 0x7f96b0137480 auth_method 0 2024-09-16T05:20:37.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.459+0000 7f96b63a7640 1 --2- 172.21.15.114:0/2828813755 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f96b0137480 0x7f96b013be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:20:37.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b53a5640 1 --2- 172.21.15.114:0/2828813755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96b00a57b0 0x7f96b0136070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f96ac00c8c0 tx=0x7f96ac00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:37.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 10 client.?.objecter ms_handle_connect 0x7f96b00a57b0 2024-09-16T05:20:37.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:20:37.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 1 -- 172.21.15.114:0/2828813755 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac018070 con 0x7f96b00a57b0 2024-09-16T05:20:37.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:37.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:37.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 10 monclient: dump: 2024-09-16T05:20:37.473 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:37.473 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:37.473 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:37.474 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 10 monclient: _finish_auth 0 2024-09-16T05:20:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:20:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 1 -- 172.21.15.114:0/2828813755 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96ac00a5e0 con 0x7f96b00a57b0 2024-09-16T05:20:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b7e0a640 5 monclient: authenticate success, global_id 74541 2024-09-16T05:20:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b7e0a640 10 monclient: _renew_subs 2024-09-16T05:20:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b7e0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b7e0a640 1 -- 172.21.15.114:0/2828813755 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b0120b40 con 0x7f96b00a57b0 2024-09-16T05:20:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 1 -- 172.21.15.114:0/2828813755 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac00aa50 con 0x7f96b00a57b0 2024-09-16T05:20:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 10 monclient: dump: 2024-09-16T05:20:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:37.476 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a3fff640 4 set_mon_vals no callback set 2024-09-16T05:20:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b7e0a640 10 client.74541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:20:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b7e0a640 10 monclient: _renew_subs 2024-09-16T05:20:37.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b7e0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:37.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b7e0a640 1 -- 172.21.15.114:0/2828813755 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b0120580 con 0x7f96b00a57b0 2024-09-16T05:20:37.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b7e0a640 1 librados: init done 2024-09-16T05:20:37.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b7e0a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:37.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b7e0a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:37.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b7e0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:37.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b7e0a640 1 -- 172.21.15.114:0/2828813755 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b0137860 con 0x7f96b00a57b0 2024-09-16T05:20:37.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 1 -- 172.21.15.114:0/2828813755 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f96ac01b070 con 0x7f96b00a57b0 2024-09-16T05:20:37.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 1 --2- 172.21.15.114:0/2828813755 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9694037810 0x7f9694039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 1 -- 172.21.15.114:0/2828813755 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(528..528 src has 1..528) ==== 5861+0+0 (secure 0 0 0) 0x7f96ac048650 con 0x7f96b00a57b0 2024-09-16T05:20:37.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 10 client.74541.objecter ms_dispatch 0x7f96b0000ba0 osd_map(528..528 src has 1..528) 2024-09-16T05:20:37.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 3 client.74541.objecter handle_osd_map got epochs [528,528] > 0 2024-09-16T05:20:37.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 3 client.74541.objecter handle_osd_map decoding full epoch 528 2024-09-16T05:20:37.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 20 client.74541.objecter dump_active .. 0 homeless 2024-09-16T05:20:37.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b5ba6640 1 --2- 172.21.15.114:0/2828813755 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9694037810 0x7f9694039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b5ba6640 10 monclient: get_auth_request con 0x7f9694037810 auth_method 0 2024-09-16T05:20:37.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96a27fc640 10 client.74541.objecter ms_handle_connect 0x7f9694037810 2024-09-16T05:20:37.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.463+0000 7f96b5ba6640 1 --2- 172.21.15.114:0/2828813755 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9694037810 0x7f9694039cd0 secure :-1 s=READY pgs=3562 cs=0 l=1 rev1=1 crypto rx=0x7f96a4006fd0 tx=0x7f96a4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:37.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.467+0000 7f96a27fc640 1 -- 172.21.15.114:0/2828813755 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f96b009cfa0 con 0x7f96b00a57b0 2024-09-16T05:20:37.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.467+0000 7f96a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:37.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.467+0000 7f96a27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:20:37.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.695+0000 7f96b7e0a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:20:37.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.695+0000 7f96b7e0a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:20:37.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.695+0000 7f96b7e0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:37.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.695+0000 7f96b7e0a640 1 -- 172.21.15.114:0/2828813755 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f96b013e410 con 0x7f96b00a57b0 2024-09-16T05:20:37.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.695+0000 7f96a27fc640 1 -- 172.21.15.114:0/2828813755 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f96ac010300 con 0x7f96b00a57b0 2024-09-16T05:20:37.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.695+0000 7f96a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:20:37.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.695+0000 7f96a27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:20:37.693 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:20:37.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.695+0000 7f96b7e0a640 10 librados: watch_flush enter 2024-09-16T05:20:37.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.695+0000 7f96b7e0a640 10 librados: watch_flush exit 2024-09-16T05:20:37.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.695+0000 7f96b7e0a640 20 client.74541.objecter shutdown clearing up homeless session... 2024-09-16T05:20:37.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.695+0000 7f96b7e0a640 10 client.74541.objecter successfully canceled tick 2024-09-16T05:20:37.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.695+0000 7f96b7e0a640 1 -- 172.21.15.114:0/2828813755 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9694037810 msgr2=0x7f9694039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:37.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.695+0000 7f96b7e0a640 1 --2- 172.21.15.114:0/2828813755 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9694037810 0x7f9694039cd0 secure :-1 s=READY pgs=3562 cs=0 l=1 rev1=1 crypto rx=0x7f96a4006fd0 tx=0x7f96a4008040 comp rx=0 tx=0).stop 2024-09-16T05:20:37.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.699+0000 7f96b7e0a640 10 monclient: shutdown 2024-09-16T05:20:37.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.699+0000 7f96b7e0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:37.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.699+0000 7f96b7e0a640 1 -- 172.21.15.114:0/2828813755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96b00a57b0 msgr2=0x7f96b0136070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:37.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.699+0000 7f96b7e0a640 1 --2- 172.21.15.114:0/2828813755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96b00a57b0 0x7f96b0136070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f96ac00c8c0 tx=0x7f96ac00cd80 comp rx=0 tx=0).stop 2024-09-16T05:20:37.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.699+0000 7f96b7e0a640 1 -- 172.21.15.114:0/2828813755 shutdown_connections 2024-09-16T05:20:37.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.699+0000 7f96b7e0a640 1 --2- 172.21.15.114:0/2828813755 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9694037810 0x7f9694039cd0 unknown :-1 s=CLOSED pgs=3562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.699+0000 7f96b7e0a640 1 --2- 172.21.15.114:0/2828813755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96b00a57b0 0x7f96b0136070 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.699+0000 7f96b7e0a640 1 --2- 172.21.15.114:0/2828813755 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f96b0137480 0x7f96b013be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.699+0000 7f96b7e0a640 1 --2- 172.21.15.114:0/2828813755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b00a4980 0x7f96b0135b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.699+0000 7f96b7e0a640 1 -- 172.21.15.114:0/2828813755 >> 172.21.15.114:0/2828813755 conn(0x7f96b00a9c40 msgr2=0x7f96b00a8600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:37.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.699+0000 7f96b7e0a640 1 -- 172.21.15.114:0/2828813755 shutdown_connections 2024-09-16T05:20:37.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.699+0000 7f96b7e0a640 1 -- 172.21.15.114:0/2828813755 wait complete. 2024-09-16T05:20:37.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.699+0000 7f96b7e0a640 1 librados: shutdown 2024-09-16T05:20:37.711 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:20:37.711 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:20:37.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:20:37.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.835+0000 7fc564adb640 10 monclient: get_monmap_and_config 2024-09-16T05:20:37.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.835+0000 7fc564adb640 10 monclient: build_initial_monmap 2024-09-16T05:20:37.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 10 monclient: monmap: 2024-09-16T05:20:37.836 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:37.836 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:37.836 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:20:37.840307+0000 2024-09-16T05:20:37.836 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:20:37.840307+0000 2024-09-16T05:20:37.836 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:37.836 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:37.836 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:20:37.836 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:20:37.837 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:20:37.837 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:20:37.837 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:20:37.837 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:37.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 1 Processor -- start 2024-09-16T05:20:37.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 1 -- start start 2024-09-16T05:20:37.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 10 monclient: init 2024-09-16T05:20:37.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:37.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-16T05:20:37.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc5600a0ec0 0x7fc56009dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 10 monclient(hunting): picked mon.noname-c con 0x7fc5600a0ec0 addr v2:172.21.15.149:3300/0 2024-09-16T05:20:37.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc56009e000 addr v1:172.21.15.114:6789/0 2024-09-16T05:20:37.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc560139640 0x7fc56013ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 10 monclient(hunting): picked mon.noname-b con 0x7fc560139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:37.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc56009a760 con 0x7fc56009e000 2024-09-16T05:20:37.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:37.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fc5600a12a0 con 0x7fc5600a0ec0 2024-09-16T05:20:37.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:37.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc5600994b0 con 0x7fc560139640 2024-09-16T05:20:37.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:37.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 10 monclient(hunting): authenticate will time out at 1722.469493s 2024-09-16T05:20:37.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55dd9b640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc56009e000 0x7fc56009e410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36726/0 (socket says 172.21.15.114:36726) 2024-09-16T05:20:37.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55ed9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc560139640 0x7fc56013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55dd9b640 1 -- 172.21.15.114:0/282458475 learned_addr learned my addr 172.21.15.114:0/282458475 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:20:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55ed9d640 10 monclient(hunting): get_auth_request con 0x7fc560139640 auth_method 0 2024-09-16T05:20:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 1 --2- 172.21.15.114:0/282458475 >> v2:172.21.15.149:3300/0 conn(0x7fc5600a0ec0 0x7fc56009dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55ed9d640 1 -- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc560139640 msgr2=0x7fc56013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:20:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55ed9d640 1 -- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc560139640 msgr2=0x7fc56013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:20:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55ed9d640 1 --2- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc560139640 0x7fc56013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient(hunting): get_auth_request con 0x7fc5600a0ec0 auth_method 0 2024-09-16T05:20:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55ed9d640 1 --2- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc560139640 0x7fc56013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:20:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 -- 172.21.15.114:0/282458475 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1186825219 0 0) 0x7fc56009a760 con 0x7fc56009e000 2024-09-16T05:20:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:20:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient(hunting): my global_id is 74544 2024-09-16T05:20:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 -- 172.21.15.114:0/282458475 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc548003530 con 0x7fc56009e000 2024-09-16T05:20:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 -- 172.21.15.114:0/282458475 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1996301398 0 0) 0x7fc548003530 con 0x7fc56009e000 2024-09-16T05:20:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 -- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc560139640 msgr2=0x7fc56013ba30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:20:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 --2- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc560139640 0x7fc56013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 -- 172.21.15.114:0/282458475 >> v2:172.21.15.149:3300/0 conn(0x7fc5600a0ec0 msgr2=0x7fc56009dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:37.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 --2- 172.21.15.114:0/282458475 >> v2:172.21.15.149:3300/0 conn(0x7fc5600a0ec0 0x7fc56009dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient: _finish_hunting 0 2024-09-16T05:20:37.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:20:37.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:37.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:20:37.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 -- 172.21.15.114:0/282458475 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5600a1870 con 0x7fc56009e000 2024-09-16T05:20:37.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient: _finish_auth 0 2024-09-16T05:20:37.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient: _check_auth_tickets 2024-09-16T05:20:37.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:37.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 -- 172.21.15.114:0/282458475 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7fc554001800 con 0x7fc56009e000 2024-09-16T05:20:37.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 5 monclient: authenticate success, global_id 74544 2024-09-16T05:20:37.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:37.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:37.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:20:37.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient: dump: 2024-09-16T05:20:37.855 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:37.855 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:37.855 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.855 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.855 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:37.855 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:37.855 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:37.855 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:37.856 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:37.856 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:37.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:20:37.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:37.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 -- 172.21.15.114:0/282458475 >> v1:172.21.15.114:6789/0 conn(0x7fc56009e000 legacy=0x7fc56009e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:37.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:20:37.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 --2- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc560139640 0x7fc5480039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient(hunting): picked mon.c con 0x7fc560139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:37.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 --2- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc548003f40 0x7fc548004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient(hunting): picked mon.a con 0x7fc548003f40 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:37.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 --2- 172.21.15.114:0/282458475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc548004d50 0x7fc5480051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient(hunting): picked mon.b con 0x7fc548004d50 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:37.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:37.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 -- 172.21.15.114:0/282458475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc5600a12a0 con 0x7fc548003f40 2024-09-16T05:20:37.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:37.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 -- 172.21.15.114:0/282458475 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fc5600994b0 con 0x7fc548004d50 2024-09-16T05:20:37.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:37.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 1 -- 172.21.15.114:0/282458475 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc54800bad0 con 0x7fc560139640 2024-09-16T05:20:37.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:37.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55d59a640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:20:37.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 1 --2- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc560139640 0x7fc5480039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55dd9b640 1 --2- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc548003f40 0x7fc548004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55dd9b640 10 monclient(hunting): get_auth_request con 0x7fc548003f40 auth_method 0 2024-09-16T05:20:37.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:37.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:37.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:37.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc564adb640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:37.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient(hunting): get_auth_request con 0x7fc560139640 auth_method 0 2024-09-16T05:20:37.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:37.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:37.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:37.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:37.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:37.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:20:37.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:37.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:37.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:20:37.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55ed9d640 1 --2- 172.21.15.114:0/282458475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc548004d50 0x7fc5480051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient(hunting): handle_auth_done global_id 74544 payload 293 2024-09-16T05:20:37.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 1 -- 172.21.15.114:0/282458475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc548004d50 msgr2=0x7fc5480051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:37.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 1 --2- 172.21.15.114:0/282458475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc548004d50 0x7fc5480051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 1 -- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc548003f40 msgr2=0x7fc548004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:37.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 1 --2- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc548003f40 0x7fc548004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient: _finish_hunting 0 2024-09-16T05:20:37.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 1 monclient: found mon.c 2024-09-16T05:20:37.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:37.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:20:37.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 1 -- 172.21.15.114:0/282458475 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc548003530 con 0x7fc560139640 2024-09-16T05:20:37.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient: _finish_auth 0 2024-09-16T05:20:37.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 10 monclient: _check_auth_tickets 2024-09-16T05:20:37.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:37.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55ed9d640 10 monclient: get_auth_request con 0x7fc548004d50 auth_method 0 2024-09-16T05:20:37.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55dd9b640 1 --2- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc548003f40 0x7fc548004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:20:37.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.839+0000 7fc55ed9d640 1 --2- 172.21.15.114:0/282458475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc548004d50 0x7fc5480051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:20:37.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55e59c640 1 --2- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc560139640 0x7fc5480039c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc550009b10 tx=0x7fc55001ca90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb595b11617114ec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:37.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55d59a640 1 -- 172.21.15.114:0/282458475 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5500096b0 con 0x7fc560139640 2024-09-16T05:20:37.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:37.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:20:37.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55d59a640 10 monclient: dump: 2024-09-16T05:20:37.871 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:37.871 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:37.873 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55d59a640 1 -- 172.21.15.114:0/282458475 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5500a6480 con 0x7fc560139640 2024-09-16T05:20:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 10 monclient: get_monmap_and_config success 2024-09-16T05:20:37.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 4 set_mon_vals no callback set 2024-09-16T05:20:37.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55d59a640 1 -- 172.21.15.114:0/282458475 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5500a6870 con 0x7fc560139640 2024-09-16T05:20:37.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:37.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:20:37.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55d59a640 10 monclient: dump: 2024-09-16T05:20:37.874 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:37.875 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:37.875 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.875 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.875 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:37.875 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:37.875 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:37.876 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:37.876 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:37.876 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:37.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 10 monclient: shutdown 2024-09-16T05:20:37.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:37.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 -- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc560139640 msgr2=0x7fc5480039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:37.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 --2- 172.21.15.114:0/282458475 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc560139640 0x7fc5480039c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc550009b10 tx=0x7fc55001ca90 comp rx=0 tx=0).stop 2024-09-16T05:20:37.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55ed9d640 1 -- 172.21.15.114:0/282458475 reap_dead start 2024-09-16T05:20:37.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 -- 172.21.15.114:0/282458475 shutdown_connections 2024-09-16T05:20:37.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 -- 172.21.15.114:0/282458475 >> 172.21.15.114:0/282458475 conn(0x7fc5600a1fb0 msgr2=0x7fc5600a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:37.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 -- 172.21.15.114:0/282458475 shutdown_connections 2024-09-16T05:20:37.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 -- 172.21.15.114:0/282458475 wait complete. 2024-09-16T05:20:37.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 10 monclient: build_initial_monmap 2024-09-16T05:20:37.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 10 monclient: monmap: 2024-09-16T05:20:37.878 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:37.878 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:20:37.878 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:20:37.878 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:20:37.878 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:37.878 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:37.879 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:20:37.879 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:20:37.879 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:20:37.879 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:37.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 librados: starting msgr at 2024-09-16T05:20:37.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 librados: starting objecter 2024-09-16T05:20:37.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 Processor -- start 2024-09-16T05:20:37.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 -- start start 2024-09-16T05:20:37.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 librados: setting wanted keys 2024-09-16T05:20:37.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 librados: calling monclient init 2024-09-16T05:20:37.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 10 monclient: init 2024-09-16T05:20:37.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:37.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:20:37.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc560139640 0x7fc5600987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 10 monclient(hunting): picked mon.noname-b con 0x7fc560139640 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:37.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56013cf80 0x7fc56013d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 10 monclient(hunting): picked mon.noname-a con 0x7fc56013cf80 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:37.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc56013d920 0x7fc5601d5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 10 monclient(hunting): picked mon.noname-c con 0x7fc56013d920 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:37.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:37.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc56009d340 con 0x7fc56013cf80 2024-09-16T05:20:37.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:37.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fc56009d1c0 con 0x7fc560139640 2024-09-16T05:20:37.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:37.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc56009d4c0 con 0x7fc56013d920 2024-09-16T05:20:37.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:37.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc564adb640 10 monclient(hunting): authenticate will time out at 1722.475459s 2024-09-16T05:20:37.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56013cf80 0x7fc56013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56013cf80 0x7fc56013d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53438/0 (socket says 172.21.15.114:53438) 2024-09-16T05:20:37.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55ed9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc56013d920 0x7fc5601d5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 1 -- 172.21.15.114:0/544360081 learned_addr learned my addr 172.21.15.114:0/544360081 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:37.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 10 monclient(hunting): get_auth_request con 0x7fc56013cf80 auth_method 0 2024-09-16T05:20:37.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:37.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:37.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:37.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55ed9d640 10 monclient(hunting): get_auth_request con 0x7fc56013d920 auth_method 0 2024-09-16T05:20:37.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:37.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:37.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:37.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:37.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:37.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:37.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55ed9d640 1 -- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc56013d920 msgr2=0x7fc5601d5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T05:20:37.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55ed9d640 1 -- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc56013d920 msgr2=0x7fc5601d5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:20:37.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55ed9d640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc56013d920 0x7fc5601d5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:37.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55ed9d640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc56013d920 0x7fc5601d5f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:20:37.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55e59c640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc560139640 0x7fc5600987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 10 monclient(hunting): handle_auth_done global_id 74547 payload 824 2024-09-16T05:20:37.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 1 -- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc56013d920 msgr2=0x7fc5601d5f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:20:37.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc56013d920 0x7fc5601d5f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 1 -- 172.21.15.114:0/544360081 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc560139640 msgr2=0x7fc5600987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:37.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc560139640 0x7fc5600987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 10 monclient: _finish_hunting 0 2024-09-16T05:20:37.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 1 monclient: found mon.noname-a 2024-09-16T05:20:37.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:37.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:37.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.843+0000 7fc55dd9b640 1 -- 172.21.15.114:0/544360081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc560094e00 con 0x7fc56013cf80 2024-09-16T05:20:37.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55dd9b640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56013cf80 0x7fc56013d3e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc55400fb60 tx=0x7fc55400d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:37.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 client.?.objecter ms_handle_connect 0x7fc56013cf80 2024-09-16T05:20:37.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:20:37.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 1 -- 172.21.15.114:0/544360081 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55400dc30 con 0x7fc56013cf80 2024-09-16T05:20:37.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:37.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:37.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 monclient: dump: 2024-09-16T05:20:37.888 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:37.888 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:37.889 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.889 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.889 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:37.889 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:37.889 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:37.889 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:37.890 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:37.890 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:37.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 monclient: _finish_auth 0 2024-09-16T05:20:37.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 monclient: _check_auth_tickets 2024-09-16T05:20:37.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:37.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 1 -- 172.21.15.114:0/544360081 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc554018070 con 0x7fc56013cf80 2024-09-16T05:20:37.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc564adb640 5 monclient: authenticate success, global_id 74547 2024-09-16T05:20:37.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc564adb640 10 monclient: _renew_subs 2024-09-16T05:20:37.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc564adb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:37.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc564adb640 1 -- 172.21.15.114:0/544360081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc560098ce0 con 0x7fc56013cf80 2024-09-16T05:20:37.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:37.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 1 -- 172.21.15.114:0/544360081 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5540070d0 con 0x7fc56013cf80 2024-09-16T05:20:37.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:37.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:37.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 monclient: dump: 2024-09-16T05:20:37.891 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:37.892 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:37.892 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.892 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.892 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:37.892 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:37.892 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:37.892 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:37.892 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:37.892 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:37.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55cd99640 4 set_mon_vals no callback set 2024-09-16T05:20:37.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc564adb640 10 client.74547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:20:37.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc564adb640 10 monclient: _renew_subs 2024-09-16T05:20:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc564adb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc564adb640 1 -- 172.21.15.114:0/544360081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5600a4930 con 0x7fc56013cf80 2024-09-16T05:20:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc564adb640 1 librados: init done 2024-09-16T05:20:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc564adb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc564adb640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56002d7d0 0x7fc560062ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc564adb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc564adb640 1 -- 172.21.15.114:0/544360081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc560069390 con 0x7fc56002d7d0 2024-09-16T05:20:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 1 -- 172.21.15.114:0/544360081 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc554007270 con 0x7fc56013cf80 2024-09-16T05:20:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55e59c640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56002d7d0 0x7fc560062ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55e59c640 10 monclient: get_auth_request con 0x7fc56002d7d0 auth_method 0 2024-09-16T05:20:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 1 --2- 172.21.15.114:0/544360081 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc540037810 0x7fc540039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 1 -- 172.21.15.114:0/544360081 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(528..528 src has 1..528) ==== 5861+0+0 (secure 0 0 0) 0x7fc554049300 con 0x7fc56013cf80 2024-09-16T05:20:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 client.74547.objecter ms_dispatch 0x7fc560000ba0 osd_map(528..528 src has 1..528) 2024-09-16T05:20:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 3 client.74547.objecter handle_osd_map got epochs [528,528] > 0 2024-09-16T05:20:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 3 client.74547.objecter handle_osd_map decoding full epoch 528 2024-09-16T05:20:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55e59c640 10 monclient(hunting): handle_auth_done global_id 74550 payload 824 2024-09-16T05:20:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55e59c640 1 -- 172.21.15.114:0/544360081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc56002dbb0 con 0x7fc56002d7d0 2024-09-16T05:20:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 20 client.74547.objecter dump_active .. 0 homeless 2024-09-16T05:20:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 client.74547.objecter ms_handle_connect 0x7fc56002d7d0 2024-09-16T05:20:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 client.74547.objecter resend_mon_ops 2024-09-16T05:20:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55e59c640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56002d7d0 0x7fc560062ea0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc550009ae0 tx=0x7fc548005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55ed9d640 1 --2- 172.21.15.114:0/544360081 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc540037810 0x7fc540039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55ed9d640 10 monclient: get_auth_request con 0x7fc540037810 auth_method 0 2024-09-16T05:20:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 1 -- 172.21.15.114:0/544360081 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5500c0070 con 0x7fc56002d7d0 2024-09-16T05:20:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 monclient: dump: 2024-09-16T05:20:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:37.896 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 1 -- 172.21.15.114:0/544360081 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc55000f000 con 0x7fc56002d7d0 2024-09-16T05:20:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:20:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 1 -- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56002d7d0 msgr2=0x7fc560062ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56002d7d0 0x7fc560062ea0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc550009ae0 tx=0x7fc548005840 comp rx=0 tx=0).stop 2024-09-16T05:20:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 1 -- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56002d7d0 msgr2=0x7fc560062ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:20:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56002d7d0 0x7fc560062ea0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc54effd640 10 client.74547.objecter ms_handle_connect 0x7fc540037810 2024-09-16T05:20:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.847+0000 7fc55ed9d640 1 --2- 172.21.15.114:0/544360081 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc540037810 0x7fc540039cd0 secure :-1 s=READY pgs=3563 cs=0 l=1 rev1=1 crypto rx=0x7fc5600a3050 tx=0x7fc55800b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:37.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:20:37.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5600633e0 0x7fc5601da4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:37.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:37.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 1 -- 172.21.15.114:0/544360081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc56013ea50 con 0x7fc5600633e0 2024-09-16T05:20:37.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc55e59c640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5600633e0 0x7fc5601da4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:37.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc55e59c640 10 monclient: get_auth_request con 0x7fc5600633e0 auth_method 0 2024-09-16T05:20:37.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc55e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:37.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc55e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:37.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc55e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:37.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc55e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:37.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc55e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:37.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc55e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:37.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc55e59c640 10 monclient(hunting): handle_auth_done global_id 74553 payload 824 2024-09-16T05:20:37.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc55e59c640 1 -- 172.21.15.114:0/544360081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc5600727d0 con 0x7fc5600633e0 2024-09-16T05:20:37.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc54effd640 10 client.74547.objecter ms_handle_connect 0x7fc5600633e0 2024-09-16T05:20:37.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc54effd640 10 client.74547.objecter resend_mon_ops 2024-09-16T05:20:37.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc55e59c640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5600633e0 0x7fc5601da4f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc550009a30 tx=0x7fc5500a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:37.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc54effd640 1 -- 172.21.15.114:0/544360081 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5500c0040 con 0x7fc5600633e0 2024-09-16T05:20:37.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc54effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:37.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc54effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:37.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc54effd640 10 monclient: dump: 2024-09-16T05:20:37.901 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:37.901 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:37.901 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.901 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:37.901 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:37.901 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:37.902 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:37.902 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:37.902 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:37.902 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:37.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc54effd640 1 -- 172.21.15.114:0/544360081 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7fc5500a6480 con 0x7fc5600633e0 2024-09-16T05:20:37.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc54effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:20:37.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc54effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:20:37.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc54effd640 1 -- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5600633e0 msgr2=0x7fc5601da4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:37.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc54effd640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5600633e0 0x7fc5601da4f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc550009a30 tx=0x7fc5500a7000 comp rx=0 tx=0).stop 2024-09-16T05:20:37.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc54effd640 1 -- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5600633e0 msgr2=0x7fc5601da4f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:20:37.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc54effd640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5600633e0 0x7fc5601da4f0 secure :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc550009a30 tx=0x7fc5500a7000 comp rx=0 tx=0).stop 2024-09-16T05:20:37.903 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:20:37.903 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:20:37.903 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:20:37.903 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:20:37.903 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 56, 2024-09-16T05:20:37.903 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 396910, 2024-09-16T05:20:37.903 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:20:37.903 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:20:37.903 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:20:37.903 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:20:37.903 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:37.903 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 3, 2024-09-16T05:20:37.903 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:20:37.904 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:20:37.904 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:20:37.904 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:37.904 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:37.904 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:37.904 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:37.904 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:37.904 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:37.904 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:37.904 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:37.904 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:37.904 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:37.904 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:37.905 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:37.905 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:20:37.905 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:20:37.905 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:37.905 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:37.905 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:37.905 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:37.905 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:37.905 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:37.905 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:37.905 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:37.905 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:37.905 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:37.906 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:37.906 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:37.906 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:37.906 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:20:37.906 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:20:37.906 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:20:37.906 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:20:37.906 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:20:37.906 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:20:37.906 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:20:37.906 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:20:37.906 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:20:37.906 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:20:37.906 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:20:37.907 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:20:37.907 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:20:37.907 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:20:37.907 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:20:37.907 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:20:37.907 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:20:37.907 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:37.907 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:37.907 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:37.907 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:37.907 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:37.907 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:37.907 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:37.907 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:37.908 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:37.908 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:37.908 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:37.908 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:37.908 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:20:37.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:37.908 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:20:37.908 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:37.908 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:20:37.908 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:20:37.908 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:37.908 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:37.908 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:37.909 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:37.909 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:20:37.909 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:37.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:37.909 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:37.909 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:37.909 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:20:37.909 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:37.909 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:37.909 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:37.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:37.909 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:20:37.909 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:20:37.910 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:37.910 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:37.910 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:37.910 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:37.910 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:37.910 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:20:37.910 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:20:37.910 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:37.910 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:37.910 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:37.910 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:37.910 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:20:37.910 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:37.911 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:37.911 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:37.911 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:37.911 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:20:37.911 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:37.911 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:37.911 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:37.912 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:37.912 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:20:37.912 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:20:37.912 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:37.912 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:37.912 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:37.912 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:37.912 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:37.912 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:20:37.912 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:20:37.913 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:37.913 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:37.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:37.913 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:37.913 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:20:37.913 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:37.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:37.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:37.914 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:37.914 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:20:37.914 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:37.914 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:37.914 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:37.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:37.915 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:20:37.915 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:20:37.915 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:37.915 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:37.915 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:37.915 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:37.915 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:37.915 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:37.915 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:20:37.915 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:20:37.915 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:37.915 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:37.915 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:37.916 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:20:37.916 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:37.916 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:37.916 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:20:37.916 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:37.916 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:37.916 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:37.916 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:20:37.916 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:37.916 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:37.916 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:20:37.916 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:37.916 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:37.917 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:37.917 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:20:37.917 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:37.917 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:37.917 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:37.917 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:20:37.917 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:20:37.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 10 librados: watch_flush enter 2024-09-16T05:20:37.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 10 librados: watch_flush exit 2024-09-16T05:20:37.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 20 client.74547.objecter shutdown clearing up homeless session... 2024-09-16T05:20:37.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 10 client.74547.objecter successfully canceled tick 2024-09-16T05:20:37.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 1 -- 172.21.15.114:0/544360081 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc540037810 msgr2=0x7fc540039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:37.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 1 --2- 172.21.15.114:0/544360081 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc540037810 0x7fc540039cd0 secure :-1 s=READY pgs=3563 cs=0 l=1 rev1=1 crypto rx=0x7fc5600a3050 tx=0x7fc55800b040 comp rx=0 tx=0).stop 2024-09-16T05:20:37.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 10 monclient: shutdown 2024-09-16T05:20:37.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:37.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 1 -- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56013cf80 msgr2=0x7fc56013d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:37.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 1 --2- 172.21.15.114:0/544360081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56013cf80 0x7fc56013d3e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc55400fb60 tx=0x7fc55400d520 comp rx=0 tx=0).stop 2024-09-16T05:20:37.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc55ed9d640 1 -- 172.21.15.114:0/544360081 reap_dead start 2024-09-16T05:20:37.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc55ed9d640 1 -- 172.21.15.114:0/544360081 reap_dead start 2024-09-16T05:20:37.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 1 -- 172.21.15.114:0/544360081 shutdown_connections 2024-09-16T05:20:37.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 1 -- 172.21.15.114:0/544360081 >> 172.21.15.114:0/544360081 conn(0x7fc5600a1fb0 msgr2=0x7fc56006f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:37.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 1 -- 172.21.15.114:0/544360081 shutdown_connections 2024-09-16T05:20:37.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 1 -- 172.21.15.114:0/544360081 wait complete. 2024-09-16T05:20:37.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:37.855+0000 7fc564adb640 1 librados: shutdown 2024-09-16T05:20:37.920 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:20:37.921 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:20:37.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:20:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 10 monclient: get_monmap_and_config 2024-09-16T05:20:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 10 monclient: build_initial_monmap 2024-09-16T05:20:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 10 monclient: monmap: 2024-09-16T05:20:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:20:38.048715+0000 2024-09-16T05:20:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:20:38.048715+0000 2024-09-16T05:20:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:20:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:20:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:20:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:20:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:20:38.044 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:38.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 1 Processor -- start 2024-09-16T05:20:38.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 1 -- start start 2024-09-16T05:20:38.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 10 monclient: init 2024-09-16T05:20:38.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:38.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-16T05:20:38.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f436c0a0ec0 0x7f436c09dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:38.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 10 monclient(hunting): picked mon.noname-c con 0x7f436c0a0ec0 addr v2:172.21.15.149:3300/0 2024-09-16T05:20:38.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f436c09e000 addr v1:172.21.15.114:6789/0 2024-09-16T05:20:38.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f436c139640 addr v1:172.21.15.149:6789/0 2024-09-16T05:20:38.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f436c09a760 con 0x7f436c09e000 2024-09-16T05:20:38.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f436c0a1870 con 0x7f436c139640 2024-09-16T05:20:38.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:38.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f436c0994b0 con 0x7f436c0a0ec0 2024-09-16T05:20:38.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:38.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 10 monclient(hunting): authenticate will time out at 1722.677966s 2024-09-16T05:20:38.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436a7fc640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f436c09e000 0x7f436c09e410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36740/0 (socket says 172.21.15.114:36740) 2024-09-16T05:20:38.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436a7fc640 1 -- 172.21.15.114:0/1149671788 learned_addr learned my addr 172.21.15.114:0/1149671788 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:20:38.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 -- 172.21.15.114:0/1149671788 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 65157601 0 0) 0x7f436c09a760 con 0x7f436c09e000 2024-09-16T05:20:38.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:20:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient(hunting): my global_id is 74556 2024-09-16T05:20:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436affd640 1 --2- 172.21.15.114:0/1149671788 >> v2:172.21.15.149:3300/0 conn(0x7f436c0a0ec0 0x7f436c09dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 -- 172.21.15.114:0/1149671788 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4358003530 con 0x7f436c09e000 2024-09-16T05:20:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436affd640 10 monclient(hunting): get_auth_request con 0x7f436c0a0ec0 auth_method 0 2024-09-16T05:20:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 -- 172.21.15.114:0/1149671788 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2909156430 0 0) 0x7f4358003530 con 0x7f436c09e000 2024-09-16T05:20:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 -- 172.21.15.114:0/1149671788 >> v2:172.21.15.149:3300/0 conn(0x7f436c0a0ec0 msgr2=0x7f436c09dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 --2- 172.21.15.114:0/1149671788 >> v2:172.21.15.149:3300/0 conn(0x7f436c0a0ec0 0x7f436c09dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 -- 172.21.15.114:0/1149671788 >> v1:172.21.15.149:6789/0 conn(0x7f436c139640 legacy=0x7f436c13ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:20:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:20:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:20:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 -- 172.21.15.114:0/1149671788 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436c099790 con 0x7f436c09e000 2024-09-16T05:20:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient: _finish_auth 0 2024-09-16T05:20:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:20:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 5 monclient: authenticate success, global_id 74556 2024-09-16T05:20:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 -- 172.21.15.114:0/1149671788 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f435c001820 con 0x7f436c09e000 2024-09-16T05:20:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:38.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:38.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:20:38.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient: dump: 2024-09-16T05:20:38.058 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:38.058 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:38.058 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:38.058 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:38.058 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:38.058 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:38.058 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:38.058 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:38.058 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:38.058 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:38.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:20:38.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:38.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 -- 172.21.15.114:0/1149671788 >> v1:172.21.15.114:6789/0 conn(0x7f436c09e000 legacy=0x7f436c09e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:38.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:20:38.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 --2- 172.21.15.114:0/1149671788 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f43580039c0 0x7f4358003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:38.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient(hunting): picked mon.b con 0x7f43580039c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:38.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 --2- 172.21.15.114:0/1149671788 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4358004810 0x7f4358004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:38.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient(hunting): picked mon.c con 0x7f4358004810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:38.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 --2- 172.21.15.114:0/1149671788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43580051d0 0x7f435800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:38.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient(hunting): picked mon.a con 0x7f43580051d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:38.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:38.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 -- 172.21.15.114:0/1149671788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f436c0994b0 con 0x7f43580051d0 2024-09-16T05:20:38.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:38.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 -- 172.21.15.114:0/1149671788 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f435800c170 con 0x7f43580039c0 2024-09-16T05:20:38.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:38.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 1 -- 172.21.15.114:0/1149671788 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f435800c370 con 0x7f4358004810 2024-09-16T05:20:38.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:38.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f4369ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:20:38.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f43714c3640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:38.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 1 --2- 172.21.15.114:0/1149671788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43580051d0 0x7f435800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:38.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 10 monclient(hunting): get_auth_request con 0x7f43580051d0 auth_method 0 2024-09-16T05:20:38.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:38.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:38.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:38.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436a7fc640 1 --2- 172.21.15.114:0/1149671788 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4358004810 0x7f4358004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:38.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436a7fc640 10 monclient(hunting): get_auth_request con 0x7f4358004810 auth_method 0 2024-09-16T05:20:38.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:38.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:38.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:38.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:38.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:38.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:20:38.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436affd640 1 --2- 172.21.15.114:0/1149671788 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f43580039c0 0x7f4358003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:38.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436affd640 10 monclient(hunting): get_auth_request con 0x7f43580039c0 auth_method 0 2024-09-16T05:20:38.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:38.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:38.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:38.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:38.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:38.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:20:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 10 monclient(hunting): handle_auth_done global_id 74556 payload 293 2024-09-16T05:20:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 1 -- 172.21.15.114:0/1149671788 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4358004810 msgr2=0x7f4358004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 1 --2- 172.21.15.114:0/1149671788 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4358004810 0x7f4358004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 1 -- 172.21.15.114:0/1149671788 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f43580039c0 msgr2=0x7f4358003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 1 --2- 172.21.15.114:0/1149671788 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f43580039c0 0x7f4358003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:20:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 1 monclient: found mon.a 2024-09-16T05:20:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 1 -- 172.21.15.114:0/1149671788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4358003530 con 0x7f43580051d0 2024-09-16T05:20:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:20:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:20:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.047+0000 7f436b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436b7fe640 1 -- 172.21.15.114:0/1149671788 reap_dead start 2024-09-16T05:20:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436b7fe640 1 --2- 172.21.15.114:0/1149671788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43580051d0 0x7f435800b9d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f436c0a1a40 tx=0x7f435401c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da14b30021986fd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f4369ffb640 1 -- 172.21.15.114:0/1149671788 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354001aa0 con 0x7f43580051d0 2024-09-16T05:20:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f4369ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f4369ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f4369ffb640 10 monclient: dump: 2024-09-16T05:20:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:38.064 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f4369ffb640 1 -- 172.21.15.114:0/1149671788 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4354003b80 con 0x7f43580051d0 2024-09-16T05:20:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f4369ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f4369ffb640 1 -- 172.21.15.114:0/1149671788 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43540050d0 con 0x7f43580051d0 2024-09-16T05:20:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f4369ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f4369ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f4369ffb640 10 monclient: dump: 2024-09-16T05:20:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:38.065 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 10 monclient: get_monmap_and_config success 2024-09-16T05:20:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 4 set_mon_vals no callback set 2024-09-16T05:20:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 10 monclient: shutdown 2024-09-16T05:20:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 -- 172.21.15.114:0/1149671788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43580051d0 msgr2=0x7f435800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 --2- 172.21.15.114:0/1149671788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43580051d0 0x7f435800b9d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f436c0a1a40 tx=0x7f435401c660 comp rx=0 tx=0).stop 2024-09-16T05:20:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 -- 172.21.15.114:0/1149671788 shutdown_connections 2024-09-16T05:20:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 --2- 172.21.15.114:0/1149671788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43580051d0 0x7f435800b9d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 -- 172.21.15.114:0/1149671788 >> 172.21.15.114:0/1149671788 conn(0x7f436c0a1fb0 msgr2=0x7f436c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 -- 172.21.15.114:0/1149671788 shutdown_connections 2024-09-16T05:20:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 -- 172.21.15.114:0/1149671788 wait complete. 2024-09-16T05:20:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 10 monclient: build_initial_monmap 2024-09-16T05:20:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 10 monclient: monmap: 2024-09-16T05:20:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:20:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:20:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:20:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:20:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:20:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:20:38.067 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 librados: starting msgr at 2024-09-16T05:20:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 librados: starting objecter 2024-09-16T05:20:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 Processor -- start 2024-09-16T05:20:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 -- start start 2024-09-16T05:20:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 librados: setting wanted keys 2024-09-16T05:20:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 librados: calling monclient init 2024-09-16T05:20:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 10 monclient: init 2024-09-16T05:20:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:20:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43580051d0 0x7f436c1389a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 10 monclient(hunting): picked mon.noname-c con 0x7f43580051d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f436c0980f0 0x7f436c138ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:38.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 10 monclient(hunting): picked mon.noname-b con 0x7f436c0980f0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f436c06d990 0x7f436c06de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 10 monclient(hunting): picked mon.noname-a con 0x7f436c06d990 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f436c0a1df0 con 0x7f436c06d990 2024-09-16T05:20:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f436c0a1c70 con 0x7f436c0980f0 2024-09-16T05:20:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f436c09c230 con 0x7f43580051d0 2024-09-16T05:20:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f43714c3640 10 monclient(hunting): authenticate will time out at 1722.683728s 2024-09-16T05:20:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436affd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43580051d0 0x7f436c1389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f436c06d990 0x7f436c06de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:38.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436affd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43580051d0 0x7f436c1389a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:35530/0 (socket says 172.21.15.114:35530) 2024-09-16T05:20:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436affd640 1 -- 172.21.15.114:0/1124130313 learned_addr learned my addr 172.21.15.114:0/1124130313 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436affd640 10 monclient(hunting): get_auth_request con 0x7f43580051d0 auth_method 0 2024-09-16T05:20:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436b7fe640 10 monclient(hunting): get_auth_request con 0x7f436c06d990 auth_method 0 2024-09-16T05:20:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436affd640 1 -- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43580051d0 msgr2=0x7f436c1389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:20:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436affd640 1 -- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43580051d0 msgr2=0x7f436c1389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:20:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.051+0000 7f436affd640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43580051d0 0x7f436c1389a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436affd640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43580051d0 0x7f436c1389a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:20:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436a7fc640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f436c0980f0 0x7f436c138ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436b7fe640 10 monclient(hunting): handle_auth_done global_id 74559 payload 824 2024-09-16T05:20:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436b7fe640 1 -- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43580051d0 msgr2=0x7f436c1389a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:20:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436b7fe640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43580051d0 0x7f436c1389a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436b7fe640 1 -- 172.21.15.114:0/1124130313 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f436c0980f0 msgr2=0x7f436c138ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436b7fe640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f436c0980f0 0x7f436c138ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:20:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436b7fe640 1 monclient: found mon.noname-a 2024-09-16T05:20:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436b7fe640 1 -- 172.21.15.114:0/1124130313 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436c097970 con 0x7f436c06d990 2024-09-16T05:20:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436a7fc640 10 monclient: get_auth_request con 0x7f436c0980f0 auth_method 0 2024-09-16T05:20:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436a7fc640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f436c0980f0 0x7f436c138ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:20:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436b7fe640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f436c06d990 0x7f436c06de40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f436c09a730 tx=0x7f4358005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 10 client.?.objecter ms_handle_connect 0x7f436c06d990 2024-09-16T05:20:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:20:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 1 -- 172.21.15.114:0/1124130313 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354007de0 con 0x7f436c06d990 2024-09-16T05:20:38.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 10 monclient: dump: 2024-09-16T05:20:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:38.072 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 10 monclient: _finish_auth 0 2024-09-16T05:20:38.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 10 monclient: _check_auth_tickets 2024-09-16T05:20:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43714c3640 5 monclient: authenticate success, global_id 74559 2024-09-16T05:20:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 1 -- 172.21.15.114:0/1124130313 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43540af070 con 0x7f436c06d990 2024-09-16T05:20:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43714c3640 10 monclient: _renew_subs 2024-09-16T05:20:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43714c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43714c3640 1 -- 172.21.15.114:0/1124130313 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f436c09b2e0 con 0x7f436c06d990 2024-09-16T05:20:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 1 -- 172.21.15.114:0/1124130313 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354006d60 con 0x7f436c06d990 2024-09-16T05:20:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 10 monclient: dump: 2024-09-16T05:20:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:38.074 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43697fa640 4 set_mon_vals no callback set 2024-09-16T05:20:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43714c3640 10 client.74559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:20:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43714c3640 10 monclient: _renew_subs 2024-09-16T05:20:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43714c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43714c3640 1 -- 172.21.15.114:0/1124130313 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f436c139420 con 0x7f436c06d990 2024-09-16T05:20:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43714c3640 1 librados: init done 2024-09-16T05:20:38.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43714c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:38.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43714c3640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 0x7f436c06e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:38.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43714c3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:38.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43714c3640 1 -- 172.21.15.114:0/1124130313 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f436c069390 con 0x7f436c02d7d0 2024-09-16T05:20:38.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436affd640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 0x7f436c06e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:38.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436affd640 10 monclient: get_auth_request con 0x7f436c02d7d0 auth_method 0 2024-09-16T05:20:38.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:38.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:38.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:38.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 1 -- 172.21.15.114:0/1124130313 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4354007030 con 0x7f436c06d990 2024-09-16T05:20:38.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436affd640 1 -- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 msgr2=0x7f436c06e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:20:38.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436affd640 1 -- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 msgr2=0x7f436c06e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:20:38.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436affd640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 0x7f436c06e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:38.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436affd640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 0x7f436c06e380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:20:38.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 1 --2- 172.21.15.114:0/1124130313 >> v2:172.21.15.149:6812/1185531135 conn(0x7f434c037810 0x7f434c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:38.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 1 -- 172.21.15.114:0/1124130313 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(528..528 src has 1..528) ==== 5861+0+0 (secure 0 0 0) 0x7f43540e2ca0 con 0x7f436c06d990 2024-09-16T05:20:38.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 10 client.74559.objecter ms_dispatch 0x7f436c000ba0 osd_map(528..528 src has 1..528) 2024-09-16T05:20:38.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 3 client.74559.objecter handle_osd_map got epochs [528,528] > 0 2024-09-16T05:20:38.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 3 client.74559.objecter handle_osd_map decoding full epoch 528 2024-09-16T05:20:38.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 20 client.74559.objecter dump_active .. 0 homeless 2024-09-16T05:20:38.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436a7fc640 1 --2- 172.21.15.114:0/1124130313 >> v2:172.21.15.149:6812/1185531135 conn(0x7f434c037810 0x7f434c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:38.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436a7fc640 10 monclient: get_auth_request con 0x7f434c037810 auth_method 0 2024-09-16T05:20:38.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f43637fe640 10 client.74559.objecter ms_handle_connect 0x7f434c037810 2024-09-16T05:20:38.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.055+0000 7f436a7fc640 1 --2- 172.21.15.114:0/1124130313 >> v2:172.21.15.149:6812/1185531135 conn(0x7f434c037810 0x7f434c039cd0 secure :-1 s=READY pgs=3564 cs=0 l=1 rev1=1 crypto rx=0x7f435c0010f0 tx=0x7f435c001170 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:38.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.259+0000 7f436affd640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 0x7f436c06e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:38.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.259+0000 7f436affd640 10 monclient: get_auth_request con 0x7f436c02d7d0 auth_method 0 2024-09-16T05:20:38.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.259+0000 7f436affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:38.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.259+0000 7f436affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:38.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.259+0000 7f436affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:38.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.259+0000 7f436affd640 1 -- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 msgr2=0x7f436c06e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:20:38.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.259+0000 7f436affd640 1 -- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 msgr2=0x7f436c06e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:20:38.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.259+0000 7f436affd640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 0x7f436c06e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:38.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.259+0000 7f436affd640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 0x7f436c06e380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:20:38.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.663+0000 7f436affd640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 0x7f436c06e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:38.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.663+0000 7f436affd640 10 monclient: get_auth_request con 0x7f436c02d7d0 auth_method 0 2024-09-16T05:20:38.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.663+0000 7f436affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:38.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.663+0000 7f436affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:38.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.663+0000 7f436affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:38.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.663+0000 7f436affd640 1 -- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 msgr2=0x7f436c06e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:20:38.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.663+0000 7f436affd640 1 -- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 msgr2=0x7f436c06e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:20:38.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.663+0000 7f436affd640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 0x7f436c06e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:38.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:38.663+0000 7f436affd640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 0x7f436c06e380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:20:39.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f436affd640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 0x7f436c06e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f436affd640 10 monclient: get_auth_request con 0x7f436c02d7d0 auth_method 0 2024-09-16T05:20:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f436affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f436affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f436affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f436affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f436affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:39.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f436affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f436affd640 10 monclient(hunting): handle_auth_done global_id 104099 payload 824 2024-09-16T05:20:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f436affd640 1 -- 172.21.15.114:0/1124130313 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f436c02dbb0 con 0x7f436c02d7d0 2024-09-16T05:20:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f436affd640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 0x7f436c06e380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4364006a30 tx=0x7f4364019c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:39.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f43637fe640 10 client.74559.objecter ms_handle_connect 0x7f436c02d7d0 2024-09-16T05:20:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f43637fe640 10 client.74559.objecter resend_mon_ops 2024-09-16T05:20:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f43637fe640 1 -- 172.21.15.114:0/1124130313 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4364026070 con 0x7f436c02d7d0 2024-09-16T05:20:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f43637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f43637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:20:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f43637fe640 10 monclient: dump: 2024-09-16T05:20:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:39.466 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:39.466 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f43637fe640 1 -- 172.21.15.114:0/1124130313 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4364017e40 con 0x7f436c02d7d0 2024-09-16T05:20:39.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f43637fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:39.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f43637fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:20:39.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f43637fe640 1 -- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 msgr2=0x7f436c06e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f43637fe640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 0x7f436c06e380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4364006a30 tx=0x7f4364019c80 comp rx=0 tx=0).stop 2024-09-16T05:20:39.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f43637fe640 1 -- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 msgr2=0x7f436c06e380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:20:39.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.467+0000 7f43637fe640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c02d7d0 0x7f436c06e380 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4364006a30 tx=0x7f4364019c80 comp rx=0 tx=0).stop 2024-09-16T05:20:39.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f43714c3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:20:39.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f43714c3640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c062ea0 0x7f436c063280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:39.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f43714c3640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:39.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f43714c3640 1 -- 172.21.15.114:0/1124130313 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f436c09dac0 con 0x7f436c062ea0 2024-09-16T05:20:39.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f436affd640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c062ea0 0x7f436c063280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:39.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f436affd640 10 monclient: get_auth_request con 0x7f436c062ea0 auth_method 0 2024-09-16T05:20:39.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f436affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:39.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f436affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:39.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f436affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:39.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f436affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:39.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f436affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:39.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f436affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:39.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f436affd640 10 monclient(hunting): handle_auth_done global_id 104102 payload 824 2024-09-16T05:20:39.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f436affd640 1 -- 172.21.15.114:0/1124130313 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f436c08e710 con 0x7f436c062ea0 2024-09-16T05:20:39.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f43637fe640 10 client.74559.objecter ms_handle_connect 0x7f436c062ea0 2024-09-16T05:20:39.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f43637fe640 10 client.74559.objecter resend_mon_ops 2024-09-16T05:20:39.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f436affd640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c062ea0 0x7f436c063280 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4364006a30 tx=0x7f43640046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:39.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f43637fe640 1 -- 172.21.15.114:0/1124130313 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4364026040 con 0x7f436c062ea0 2024-09-16T05:20:39.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f43637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:39.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.475+0000 7f43637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:20:39.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43637fe640 10 monclient: dump: 2024-09-16T05:20:39.476 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:39.476 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:39.476 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.476 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.476 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:39.477 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.477 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:39.477 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:39.477 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:39.477 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43637fe640 1 -- 172.21.15.114:0/1124130313 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f43640066c0 con 0x7f436c062ea0 2024-09-16T05:20:39.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43637fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:20:39.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43637fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:20:39.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43637fe640 1 -- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c062ea0 msgr2=0x7f436c063280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43637fe640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c062ea0 0x7f436c063280 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4364006a30 tx=0x7f43640046d0 comp rx=0 tx=0).stop 2024-09-16T05:20:39.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43637fe640 1 -- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c062ea0 msgr2=0x7f436c063280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:20:39.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43637fe640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f436c062ea0 0x7f436c063280 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4364006a30 tx=0x7f43640046d0 comp rx=0 tx=0).stop 2024-09-16T05:20:39.478 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:20:39.478 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:20:39.478 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:20:39.478 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:20:39.478 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 56, 2024-09-16T05:20:39.478 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 15643, 2024-09-16T05:20:39.478 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:20:39.478 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:20:39.478 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:20:39.478 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:20:39.478 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:39.479 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 4, 2024-09-16T05:20:39.479 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:20:39.479 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:20:39.479 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:20:39.479 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:39.479 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:39.479 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:39.479 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:39.479 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:39.480 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:39.480 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:39.480 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:39.480 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:39.480 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:39.480 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:39.480 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:39.480 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:20:39.480 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:20:39.480 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:39.480 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:39.480 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:39.480 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:39.481 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:39.481 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:39.481 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:39.481 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:39.481 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:39.481 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:39.481 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:39.481 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:39.481 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.482 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:20:39.482 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:20:39.482 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:20:39.482 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:20:39.482 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:20:39.482 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:20:39.482 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:20:39.482 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:20:39.482 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:20:39.482 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:20:39.482 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:20:39.482 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:20:39.482 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:20:39.483 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:20:39.483 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:20:39.483 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:20:39.483 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:20:39.483 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:39.483 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:39.483 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:39.483 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:39.483 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:39.483 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:39.483 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:39.483 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:39.483 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:39.483 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:39.484 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:39.484 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:39.484 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:20:39.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.484 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:20:39.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.484 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:20:39.484 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:20:39.484 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:39.484 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:39.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.484 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:39.484 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:20:39.484 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:39.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.485 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.485 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:39.485 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:20:39.485 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:39.485 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:39.485 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:39.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.485 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:20:39.485 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:20:39.485 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:39.485 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:39.485 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:39.486 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.486 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.486 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:20:39.486 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:20:39.486 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:39.486 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:39.486 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.486 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:39.486 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:20:39.486 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:39.486 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.486 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.486 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:39.487 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:20:39.487 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:39.487 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:39.487 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:39.487 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.487 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:20:39.487 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:20:39.487 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:39.487 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:39.487 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:39.487 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.487 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.487 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:20:39.487 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:20:39.488 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:39.488 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:39.488 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.488 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:39.488 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:20:39.488 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:39.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.488 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.488 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:39.488 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:20:39.488 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:39.488 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:39.489 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:39.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.489 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:20:39.489 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:20:39.489 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:39.489 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:39.489 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:39.489 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:39.489 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:39.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.489 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:20:39.489 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:20:39.489 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.490 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:39.490 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:39.490 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:20:39.490 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:39.490 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:39.490 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:20:39.490 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.490 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:39.490 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:39.490 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:20:39.490 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:39.490 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:39.490 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.490 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:20:39.491 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:20:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43714c3640 10 librados: watch_flush enter 2024-09-16T05:20:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43714c3640 10 librados: watch_flush exit 2024-09-16T05:20:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43714c3640 20 client.74559.objecter shutdown clearing up homeless session... 2024-09-16T05:20:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43714c3640 10 client.74559.objecter successfully canceled tick 2024-09-16T05:20:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43714c3640 1 -- 172.21.15.114:0/1124130313 >> v2:172.21.15.149:6812/1185531135 conn(0x7f434c037810 msgr2=0x7f434c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43714c3640 1 --2- 172.21.15.114:0/1124130313 >> v2:172.21.15.149:6812/1185531135 conn(0x7f434c037810 0x7f434c039cd0 secure :-1 s=READY pgs=3564 cs=0 l=1 rev1=1 crypto rx=0x7f435c0010f0 tx=0x7f435c001170 comp rx=0 tx=0).stop 2024-09-16T05:20:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43714c3640 10 monclient: shutdown 2024-09-16T05:20:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43714c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:39.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43714c3640 1 -- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f436c06d990 msgr2=0x7f436c06de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43714c3640 1 --2- 172.21.15.114:0/1124130313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f436c06d990 0x7f436c06de40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f436c09a730 tx=0x7f4358005840 comp rx=0 tx=0).stop 2024-09-16T05:20:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f436b7fe640 1 -- 172.21.15.114:0/1124130313 reap_dead start 2024-09-16T05:20:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f436b7fe640 1 -- 172.21.15.114:0/1124130313 reap_dead start 2024-09-16T05:20:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43714c3640 1 -- 172.21.15.114:0/1124130313 shutdown_connections 2024-09-16T05:20:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43714c3640 1 -- 172.21.15.114:0/1124130313 >> 172.21.15.114:0/1124130313 conn(0x7f436c0a1fb0 msgr2=0x7f436c073070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43714c3640 1 -- 172.21.15.114:0/1124130313 shutdown_connections 2024-09-16T05:20:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43714c3640 1 -- 172.21.15.114:0/1124130313 wait complete. 2024-09-16T05:20:39.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.479+0000 7f43714c3640 1 librados: shutdown 2024-09-16T05:20:39.498 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:20:39.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:20:39.499 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:20:39.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 10 monclient: get_monmap_and_config 2024-09-16T05:20:39.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 10 monclient: build_initial_monmap 2024-09-16T05:20:39.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 10 monclient: monmap: 2024-09-16T05:20:39.623 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:39.623 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:39.623 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:20:39.628389+0000 2024-09-16T05:20:39.623 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:20:39.628389+0000 2024-09-16T05:20:39.624 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:39.624 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.624 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:20:39.624 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:20:39.624 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:20:39.624 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:20:39.624 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:20:39.624 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 1 Processor -- start 2024-09-16T05:20:39.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 1 -- start start 2024-09-16T05:20:39.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 10 monclient: init 2024-09-16T05:20:39.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:39.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-16T05:20:39.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f2b3c1376e0 0x7f2b3c137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:39.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 10 monclient(hunting): picked mon.noname-c con 0x7f2b3c1376e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:20:39.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b3c1370a0 addr v1:172.21.15.114:6789/0 2024-09-16T05:20:39.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2b3c0a5d00 0x7f2b3c0a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:39.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 10 monclient(hunting): picked mon.noname-a con 0x7f2b3c0a5d00 addr v2:172.21.15.114:3300/0 2024-09-16T05:20:39.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b3c09bd20 con 0x7f2b3c1370a0 2024-09-16T05:20:39.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:39.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f2b3c138000 con 0x7f2b3c0a5d00 2024-09-16T05:20:39.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:39.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f2b3c09c5c0 con 0x7f2b3c1376e0 2024-09-16T05:20:39.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:39.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b40e30640 10 monclient(hunting): authenticate will time out at 1724.257618s 2024-09-16T05:20:39.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b39d9b640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2b3c1370a0 0x7f2b3c0a4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36746/0 (socket says 172.21.15.114:36746) 2024-09-16T05:20:39.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b39d9b640 1 -- 172.21.15.114:0/4143953431 learned_addr learned my addr 172.21.15.114:0/4143953431 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:20:39.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 1 --2- 172.21.15.114:0/4143953431 >> v2:172.21.15.114:3300/0 conn(0x7f2b3c0a5d00 0x7f2b3c0a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:39.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 10 monclient(hunting): get_auth_request con 0x7f2b3c0a5d00 auth_method 0 2024-09-16T05:20:39.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:39.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:39.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:39.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:39.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:39.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:39.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3a59c640 1 --2- 172.21.15.114:0/4143953431 >> v2:172.21.15.149:3300/0 conn(0x7f2b3c1376e0 0x7f2b3c137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:39.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3a59c640 10 monclient(hunting): get_auth_request con 0x7f2b3c1376e0 auth_method 0 2024-09-16T05:20:39.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:39.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:39.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:39.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 10 monclient(hunting): handle_auth_done global_id 74562 payload 274 2024-09-16T05:20:39.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 1 -- 172.21.15.114:0/4143953431 >> v2:172.21.15.149:3300/0 conn(0x7f2b3c1376e0 msgr2=0x7f2b3c137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:39.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 1 --2- 172.21.15.114:0/4143953431 >> v2:172.21.15.149:3300/0 conn(0x7f2b3c1376e0 0x7f2b3c137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:39.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 1 -- 172.21.15.114:0/4143953431 >> v1:172.21.15.114:6789/0 conn(0x7f2b3c1370a0 legacy=0x7f2b3c0a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 10 monclient: _finish_hunting 0 2024-09-16T05:20:39.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 1 monclient: found mon.noname-a 2024-09-16T05:20:39.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:39.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:39.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.627+0000 7f2b3ad9d640 1 -- 172.21.15.114:0/4143953431 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b3c09c000 con 0x7f2b3c0a5d00 2024-09-16T05:20:39.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3ad9d640 1 --2- 172.21.15.114:0/4143953431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0a5d00 0x7f2b3c0a4980 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0a7410 tx=0x7f2b2c01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4897bf1da2aaa816 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:39.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3959a640 1 -- 172.21.15.114:0/4143953431 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c0b0070 con 0x7f2b3c0a5d00 2024-09-16T05:20:39.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:39.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:39.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3959a640 10 monclient: dump: 2024-09-16T05:20:39.637 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:39.637 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:39.637 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.637 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.637 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:39.638 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.638 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:39.638 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:39.638 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:39.638 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3959a640 10 monclient: _finish_auth 0 2024-09-16T05:20:39.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3959a640 10 monclient: _check_auth_tickets 2024-09-16T05:20:39.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:39.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3959a640 1 -- 172.21.15.114:0/4143953431 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b2c0ab600 con 0x7f2b3c0a5d00 2024-09-16T05:20:39.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:39.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3959a640 1 -- 172.21.15.114:0/4143953431 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c0ab9d0 con 0x7f2b3c0a5d00 2024-09-16T05:20:39.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 5 monclient: authenticate success, global_id 74562 2024-09-16T05:20:39.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient: get_monmap_and_config success 2024-09-16T05:20:39.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 4 set_mon_vals no callback set 2024-09-16T05:20:39.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:39.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:39.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3959a640 10 monclient: dump: 2024-09-16T05:20:39.639 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:39.639 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:39.639 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.639 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.639 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:39.639 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.639 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:39.639 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:39.640 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:39.640 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient: shutdown 2024-09-16T05:20:39.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:39.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 -- 172.21.15.114:0/4143953431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0a5d00 msgr2=0x7f2b3c0a4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 --2- 172.21.15.114:0/4143953431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0a5d00 0x7f2b3c0a4980 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0a7410 tx=0x7f2b2c01c340 comp rx=0 tx=0).stop 2024-09-16T05:20:39.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 -- 172.21.15.114:0/4143953431 shutdown_connections 2024-09-16T05:20:39.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 --2- 172.21.15.114:0/4143953431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0a5d00 0x7f2b3c0a4980 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:39.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 --2- 172.21.15.114:0/4143953431 >> v2:172.21.15.149:3300/0 conn(0x7f2b3c1376e0 0x7f2b3c137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:39.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 -- 172.21.15.114:0/4143953431 >> 172.21.15.114:0/4143953431 conn(0x7f2b3c0a9a40 msgr2=0x7f2b3c0a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:39.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 -- 172.21.15.114:0/4143953431 shutdown_connections 2024-09-16T05:20:39.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 -- 172.21.15.114:0/4143953431 wait complete. 2024-09-16T05:20:39.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient: build_initial_monmap 2024-09-16T05:20:39.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient: monmap: 2024-09-16T05:20:39.641 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:39.641 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:20:39.641 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:20:39.641 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:20:39.641 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:39.641 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.641 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:20:39.641 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:20:39.641 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:20:39.641 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 librados: starting msgr at 2024-09-16T05:20:39.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 librados: starting objecter 2024-09-16T05:20:39.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 Processor -- start 2024-09-16T05:20:39.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 -- start start 2024-09-16T05:20:39.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 librados: setting wanted keys 2024-09-16T05:20:39.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 librados: calling monclient init 2024-09-16T05:20:39.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient: init 2024-09-16T05:20:39.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:39.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:20:39.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0a5d00 0x7f2b3c1cf030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:39.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient(hunting): picked mon.noname-a con 0x7f2b3c0a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:39.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c1370a0 0x7f2b3c1361b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:39.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient(hunting): picked mon.noname-b con 0x7f2b3c1370a0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:39.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b3c1376e0 0x7f2b3c13c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:39.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient(hunting): picked mon.noname-c con 0x7f2b3c1376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:39.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:39.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3c141d80 con 0x7f2b3c0a5d00 2024-09-16T05:20:39.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:39.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3c141c00 con 0x7f2b3c1370a0 2024-09-16T05:20:39.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:39.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2b3c141f00 con 0x7f2b3c1376e0 2024-09-16T05:20:39.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:39.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient(hunting): authenticate will time out at 1724.262689s 2024-09-16T05:20:39.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0a5d00 0x7f2b3c1cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:39.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3ad9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b3c1376e0 0x7f2b3c13c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:39.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0a5d00 0x7f2b3c1cf030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53502/0 (socket says 172.21.15.114:53502) 2024-09-16T05:20:39.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 1 -- 172.21.15.114:0/124314745 learned_addr learned my addr 172.21.15.114:0/124314745 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:39.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 10 monclient(hunting): get_auth_request con 0x7f2b3c0a5d00 auth_method 0 2024-09-16T05:20:39.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:39.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:39.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:39.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3ad9d640 10 monclient(hunting): get_auth_request con 0x7f2b3c1376e0 auth_method 0 2024-09-16T05:20:39.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:39.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:39.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:39.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:39.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:39.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:39.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:39.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:39.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:39.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b39d9b640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c1370a0 0x7f2b3c1361b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:39.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b39d9b640 10 monclient(hunting): get_auth_request con 0x7f2b3c1370a0 auth_method 0 2024-09-16T05:20:39.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b39d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:39.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b39d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:39.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:39.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 10 monclient(hunting): handle_auth_done global_id 74568 payload 824 2024-09-16T05:20:39.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 1 -- 172.21.15.114:0/124314745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b3c1376e0 msgr2=0x7f2b3c13c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b3c1376e0 0x7f2b3c13c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:39.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 1 -- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c1370a0 msgr2=0x7f2b3c1361b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c1370a0 0x7f2b3c1361b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:39.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 10 monclient: _finish_hunting 0 2024-09-16T05:20:39.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 1 monclient: found mon.noname-a 2024-09-16T05:20:39.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:39.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:39.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 1 -- 172.21.15.114:0/124314745 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b3c1cff40 con 0x7f2b3c0a5d00 2024-09-16T05:20:39.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3ad9d640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b3c1376e0 0x7f2b3c13c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:20:39.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b3a59c640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0a5d00 0x7f2b3c1cf030 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2b3400d800 tx=0x7f2b3400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:39.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b32ffd640 10 client.?.objecter ms_handle_connect 0x7f2b3c0a5d00 2024-09-16T05:20:39.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b32ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:20:39.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b32ffd640 1 -- 172.21.15.114:0/124314745 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3400b660 con 0x7f2b3c0a5d00 2024-09-16T05:20:39.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:39.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b32ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:39.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b32ffd640 10 monclient: dump: 2024-09-16T05:20:39.647 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:39.647 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:39.647 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.647 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.647 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:39.647 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.647 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:39.647 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:39.647 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:39.647 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b32ffd640 10 monclient: _finish_auth 0 2024-09-16T05:20:39.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b32ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:20:39.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:39.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b32ffd640 1 -- 172.21.15.114:0/124314745 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b34019070 con 0x7f2b3c0a5d00 2024-09-16T05:20:39.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 5 monclient: authenticate success, global_id 74568 2024-09-16T05:20:39.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient: _renew_subs 2024-09-16T05:20:39.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:39.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 -- 172.21.15.114:0/124314745 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b3c0a8f70 con 0x7f2b3c0a5d00 2024-09-16T05:20:39.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b32ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:39.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b32ffd640 1 -- 172.21.15.114:0/124314745 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3400ba00 con 0x7f2b3c0a5d00 2024-09-16T05:20:39.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:39.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:39.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b32ffd640 10 monclient: dump: 2024-09-16T05:20:39.648 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:39.648 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:39.648 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.648 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.649 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:39.649 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.649 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:39.650 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:39.650 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:39.650 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 client.74568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:20:39.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b38d99640 4 set_mon_vals no callback set 2024-09-16T05:20:39.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient: _renew_subs 2024-09-16T05:20:39.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:39.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 -- 172.21.15.114:0/124314745 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b3c0a25e0 con 0x7f2b3c0a5d00 2024-09-16T05:20:39.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.631+0000 7f2b40e30640 1 librados: init done 2024-09-16T05:20:39.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b40e30640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:39.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b40e30640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c13c6a0 0x7f2b3c13ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:39.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b40e30640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:39.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b40e30640 1 -- 172.21.15.114:0/124314745 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3c13cfc0 con 0x7f2b3c13c6a0 2024-09-16T05:20:39.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 1 -- 172.21.15.114:0/124314745 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2b3400bba0 con 0x7f2b3c0a5d00 2024-09-16T05:20:39.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b39d9b640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c13c6a0 0x7f2b3c13ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:39.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 1 --2- 172.21.15.114:0/124314745 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2b20037810 0x7f2b20039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:39.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b39d9b640 10 monclient: get_auth_request con 0x7f2b3c13c6a0 auth_method 0 2024-09-16T05:20:39.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b39d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:39.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b39d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:39.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:39.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 1 -- 172.21.15.114:0/124314745 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(529..529 src has 1..529) ==== 5861+0+0 (secure 0 0 0) 0x7f2b340493b0 con 0x7f2b3c0a5d00 2024-09-16T05:20:39.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 10 client.74568.objecter ms_dispatch 0x7f2b3c000ba0 osd_map(529..529 src has 1..529) 2024-09-16T05:20:39.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 3 client.74568.objecter handle_osd_map got epochs [529,529] > 0 2024-09-16T05:20:39.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 3 client.74568.objecter handle_osd_map decoding full epoch 529 2024-09-16T05:20:39.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 20 client.74568.objecter dump_active .. 0 homeless 2024-09-16T05:20:39.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b39d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:39.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b39d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:39.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b39d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:39.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b3ad9d640 1 --2- 172.21.15.114:0/124314745 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2b20037810 0x7f2b20039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:39.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b3ad9d640 10 monclient: get_auth_request con 0x7f2b20037810 auth_method 0 2024-09-16T05:20:39.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b39d9b640 10 monclient(hunting): handle_auth_done global_id 64648 payload 824 2024-09-16T05:20:39.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b39d9b640 1 -- 172.21.15.114:0/124314745 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b3c137ac0 con 0x7f2b3c13c6a0 2024-09-16T05:20:39.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 10 client.74568.objecter ms_handle_connect 0x7f2b3c13c6a0 2024-09-16T05:20:39.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 10 client.74568.objecter resend_mon_ops 2024-09-16T05:20:39.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b39d9b640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c13c6a0 0x7f2b3c13ca80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c13d920 tx=0x7f2b2400c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:39.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 1 -- 172.21.15.114:0/124314745 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2400cb60 con 0x7f2b3c13c6a0 2024-09-16T05:20:39.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:39.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:20:39.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 10 monclient: dump: 2024-09-16T05:20:39.653 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:39.653 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:39.653 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.653 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.653 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:39.653 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.653 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:39.653 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:39.653 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:39.653 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 10 client.74568.objecter ms_handle_connect 0x7f2b20037810 2024-09-16T05:20:39.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b3ad9d640 1 --2- 172.21.15.114:0/124314745 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2b20037810 0x7f2b20039cd0 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c0ad000 tx=0x7f2b2c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:39.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 1 -- 172.21.15.114:0/124314745 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2b24018070 con 0x7f2b3c13c6a0 2024-09-16T05:20:39.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:39.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:20:39.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 1 -- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c13c6a0 msgr2=0x7f2b3c13ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c13c6a0 0x7f2b3c13ca80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c13d920 tx=0x7f2b2400c4b0 comp rx=0 tx=0).stop 2024-09-16T05:20:39.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 1 -- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c13c6a0 msgr2=0x7f2b3c13ca80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:20:39.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.635+0000 7f2b32ffd640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c13c6a0 0x7f2b3c13ca80 secure :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c13d920 tx=0x7f2b2400c4b0 comp rx=0 tx=0).stop 2024-09-16T05:20:39.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:20:39.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c138190 0x7f2b3c138570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:39.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:39.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 1 -- 172.21.15.114:0/124314745 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3c13e600 con 0x7f2b3c138190 2024-09-16T05:20:39.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b39d9b640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c138190 0x7f2b3c138570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:39.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b39d9b640 10 monclient: get_auth_request con 0x7f2b3c138190 auth_method 0 2024-09-16T05:20:39.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b39d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:39.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b39d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:39.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:39.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b39d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:39.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b39d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:39.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b39d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:39.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b39d9b640 10 monclient(hunting): handle_auth_done global_id 64651 payload 824 2024-09-16T05:20:39.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b39d9b640 1 -- 172.21.15.114:0/124314745 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2b3c068820 con 0x7f2b3c138190 2024-09-16T05:20:39.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b32ffd640 10 client.74568.objecter ms_handle_connect 0x7f2b3c138190 2024-09-16T05:20:39.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b32ffd640 10 client.74568.objecter resend_mon_ops 2024-09-16T05:20:39.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b39d9b640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c138190 0x7f2b3c138570 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2b24017000 tx=0x7f2b24001260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:39.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b32ffd640 1 -- 172.21.15.114:0/124314745 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2401de10 con 0x7f2b3c138190 2024-09-16T05:20:39.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:39.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:20:39.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b32ffd640 10 monclient: dump: 2024-09-16T05:20:39.657 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:39.657 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:39.657 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.657 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.658 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:39.658 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.658 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:39.658 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:39.658 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:39.658 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b32ffd640 1 -- 172.21.15.114:0/124314745 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f2b240070d0 con 0x7f2b3c138190 2024-09-16T05:20:39.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b32ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:20:39.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b32ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:20:39.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b32ffd640 1 -- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c138190 msgr2=0x7f2b3c138570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b32ffd640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c138190 0x7f2b3c138570 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2b24017000 tx=0x7f2b24001260 comp rx=0 tx=0).stop 2024-09-16T05:20:39.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b32ffd640 1 -- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c138190 msgr2=0x7f2b3c138570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:20:39.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b32ffd640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2b3c138190 0x7f2b3c138570 secure :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2b24017000 tx=0x7f2b24001260 comp rx=0 tx=0).stop 2024-09-16T05:20:39.659 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:20:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:20:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:20:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:20:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 56, 2024-09-16T05:20:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 523777, 2024-09-16T05:20:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:20:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:20:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:20:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:20:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 5, 2024-09-16T05:20:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:20:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:20:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:20:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:20:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:20:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:20:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:20:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:20:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:20:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:20:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:20:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:20:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:20:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:20:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:20:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:20:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:20:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:20:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:20:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:20:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:20:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:20:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:20:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:20:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:20:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:20:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:20:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:20:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:20:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:20:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:20:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:20:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:20:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:20:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:20:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:20:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:20:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:20:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:20:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:20:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:20:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:20:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:20:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:20:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:20:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:20:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:20:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:20:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:20:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:20:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:20:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:39.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:39.674 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:20:39.674 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:20:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 10 librados: watch_flush enter 2024-09-16T05:20:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 10 librados: watch_flush exit 2024-09-16T05:20:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 20 client.74568.objecter shutdown clearing up homeless session... 2024-09-16T05:20:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 10 client.74568.objecter successfully canceled tick 2024-09-16T05:20:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 1 -- 172.21.15.114:0/124314745 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2b20037810 msgr2=0x7f2b20039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 1 --2- 172.21.15.114:0/124314745 >> v2:172.21.15.149:6812/1185531135 conn(0x7f2b20037810 0x7f2b20039cd0 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c0ad000 tx=0x7f2b2c008040 comp rx=0 tx=0).stop 2024-09-16T05:20:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 10 monclient: shutdown 2024-09-16T05:20:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 1 -- 172.21.15.114:0/124314745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0a5d00 msgr2=0x7f2b3c1cf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 1 --2- 172.21.15.114:0/124314745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0a5d00 0x7f2b3c1cf030 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2b3400d800 tx=0x7f2b3400dcc0 comp rx=0 tx=0).stop 2024-09-16T05:20:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b3ad9d640 1 -- 172.21.15.114:0/124314745 reap_dead start 2024-09-16T05:20:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b3ad9d640 1 -- 172.21.15.114:0/124314745 reap_dead start 2024-09-16T05:20:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 1 -- 172.21.15.114:0/124314745 shutdown_connections 2024-09-16T05:20:39.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 1 -- 172.21.15.114:0/124314745 >> 172.21.15.114:0/124314745 conn(0x7f2b3c0a9a40 msgr2=0x7f2b3c09fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:39.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 1 -- 172.21.15.114:0/124314745 shutdown_connections 2024-09-16T05:20:39.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 1 -- 172.21.15.114:0/124314745 wait complete. 2024-09-16T05:20:39.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.643+0000 7f2b40e30640 1 librados: shutdown 2024-09-16T05:20:39.677 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:20:39.677 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:20:39.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:20:39.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.791+0000 7f31f1e64640 10 monclient: get_monmap_and_config 2024-09-16T05:20:39.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.791+0000 7f31f1e64640 10 monclient: build_initial_monmap 2024-09-16T05:20:39.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.791+0000 7f31f1e64640 10 monclient: monmap: 2024-09-16T05:20:39.790 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:39.790 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:39.790 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:20:39.795387+0000 2024-09-16T05:20:39.791 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:20:39.795387+0000 2024-09-16T05:20:39.791 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:39.791 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.791 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:20:39.791 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:20:39.791 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:20:39.791 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:20:39.791 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:20:39.791 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 1 Processor -- start 2024-09-16T05:20:39.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 1 -- start start 2024-09-16T05:20:39.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 10 monclient: init 2024-09-16T05:20:39.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:39.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-16T05:20:39.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31ec137730 addr v1:172.21.15.149:6789/0 2024-09-16T05:20:39.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31ec0a5d00 addr v1:172.21.15.114:6789/0 2024-09-16T05:20:39.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f31ec13dbe0 0x7f31ec13ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:39.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 10 monclient(hunting): picked mon.noname-a con 0x7f31ec13dbe0 addr v2:172.21.15.114:3300/0 2024-09-16T05:20:39.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31ec0a55e0 con 0x7f31ec0a5d00 2024-09-16T05:20:39.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31ec0a3710 con 0x7f31ec137730 2024-09-16T05:20:39.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:39.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f31ec09c450 con 0x7f31ec13dbe0 2024-09-16T05:20:39.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:39.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 10 monclient(hunting): authenticate will time out at 1724.424906s 2024-09-16T05:20:39.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31eaffd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f31ec0a5d00 0x7f31ec13c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36756/0 (socket says 172.21.15.114:36756) 2024-09-16T05:20:39.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31eaffd640 1 -- 172.21.15.114:0/2977118234 learned_addr learned my addr 172.21.15.114:0/2977118234 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:20:39.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 1 --2- 172.21.15.114:0/2977118234 >> v2:172.21.15.114:3300/0 conn(0x7f31ec13dbe0 0x7f31ec13ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:39.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 10 monclient(hunting): get_auth_request con 0x7f31ec13dbe0 auth_method 0 2024-09-16T05:20:39.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:39.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:39.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:39.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:39.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:39.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:39.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ea7fc640 1 -- 172.21.15.114:0/2977118234 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2702532182 0 0) 0x7f31ec0a55e0 con 0x7f31ec0a5d00 2024-09-16T05:20:39.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 10 monclient(hunting): handle_auth_done global_id 74571 payload 274 2024-09-16T05:20:39.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 1 -- 172.21.15.114:0/2977118234 >> v1:172.21.15.149:6789/0 conn(0x7f31ec137730 legacy=0x7f31ec137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:39.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 1 -- 172.21.15.114:0/2977118234 >> v1:172.21.15.114:6789/0 conn(0x7f31ec0a5d00 legacy=0x7f31ec13c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 10 monclient: _finish_hunting 0 2024-09-16T05:20:39.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 1 monclient: found mon.noname-a 2024-09-16T05:20:39.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:39.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:39.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 1 -- 172.21.15.114:0/2977118234 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31ec09be00 con 0x7f31ec13dbe0 2024-09-16T05:20:39.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ea7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:20:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ebfff640 1 --2- 172.21.15.114:0/2977118234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31ec13dbe0 0x7f31ec13ffd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f31e4009d90 tx=0x7f31e401c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b4bfd8c06b7b52f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ea7fc640 1 -- 172.21.15.114:0/2977118234 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e40b2070 con 0x7f31ec13dbe0 2024-09-16T05:20:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:39.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ea7fc640 10 monclient: dump: 2024-09-16T05:20:39.802 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:39.802 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:39.802 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.802 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.802 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:39.802 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.802 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:39.802 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:39.802 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:39.802 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ea7fc640 10 monclient: _finish_auth 0 2024-09-16T05:20:39.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ea7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:20:39.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:39.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 5 monclient: authenticate success, global_id 74571 2024-09-16T05:20:39.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:39.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ea7fc640 1 -- 172.21.15.114:0/2977118234 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31e40a57c0 con 0x7f31ec13dbe0 2024-09-16T05:20:39.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:39.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 10 monclient: get_monmap_and_config success 2024-09-16T05:20:39.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ea7fc640 1 -- 172.21.15.114:0/2977118234 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e40a5b90 con 0x7f31ec13dbe0 2024-09-16T05:20:39.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31f1e64640 4 set_mon_vals no callback set 2024-09-16T05:20:39.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:39.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:39.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.795+0000 7f31ea7fc640 10 monclient: dump: 2024-09-16T05:20:39.803 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:39.804 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:39.804 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.804 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.804 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:39.804 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.804 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:39.804 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:39.804 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:39.804 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient: shutdown 2024-09-16T05:20:39.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:39.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 -- 172.21.15.114:0/2977118234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31ec13dbe0 msgr2=0x7f31ec13ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 --2- 172.21.15.114:0/2977118234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31ec13dbe0 0x7f31ec13ffd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f31e4009d90 tx=0x7f31e401c7f0 comp rx=0 tx=0).stop 2024-09-16T05:20:39.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 -- 172.21.15.114:0/2977118234 shutdown_connections 2024-09-16T05:20:39.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 --2- 172.21.15.114:0/2977118234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31ec13dbe0 0x7f31ec13ffd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:39.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 -- 172.21.15.114:0/2977118234 >> 172.21.15.114:0/2977118234 conn(0x7f31ec0a9c70 msgr2=0x7f31ec0aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:39.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 -- 172.21.15.114:0/2977118234 shutdown_connections 2024-09-16T05:20:39.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 -- 172.21.15.114:0/2977118234 wait complete. 2024-09-16T05:20:39.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient: build_initial_monmap 2024-09-16T05:20:39.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient: monmap: 2024-09-16T05:20:39.805 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:39.805 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:20:39.805 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:20:39.805 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:20:39.805 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:39.806 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.806 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:20:39.806 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:20:39.806 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:20:39.806 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 librados: starting msgr at 2024-09-16T05:20:39.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 librados: starting objecter 2024-09-16T05:20:39.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 Processor -- start 2024-09-16T05:20:39.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 -- start start 2024-09-16T05:20:39.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 librados: setting wanted keys 2024-09-16T05:20:39.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 librados: calling monclient init 2024-09-16T05:20:39.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient: init 2024-09-16T05:20:39.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:39.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:20:39.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31ec0a5d00 0x7f31ec1358e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:39.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient(hunting): picked mon.noname-c con 0x7f31ec0a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:39.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f31ec137730 0x7f31ec135e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:39.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient(hunting): picked mon.noname-b con 0x7f31ec137730 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:39.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31ec13dbe0 0x7f31ec09e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:39.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient(hunting): picked mon.noname-a con 0x7f31ec13dbe0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:39.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:39.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f31ec142eb0 con 0x7f31ec13dbe0 2024-09-16T05:20:39.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:39.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f31ec142d30 con 0x7f31ec137730 2024-09-16T05:20:39.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:39.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f31ec143030 con 0x7f31ec0a5d00 2024-09-16T05:20:39.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:39.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient(hunting): authenticate will time out at 1724.429751s 2024-09-16T05:20:39.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31ec13dbe0 0x7f31ec09e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:39.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31eb7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31ec0a5d00 0x7f31ec1358e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:39.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31ec13dbe0 0x7f31ec09e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53516/0 (socket says 172.21.15.114:53516) 2024-09-16T05:20:39.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 1 -- 172.21.15.114:0/2961968898 learned_addr learned my addr 172.21.15.114:0/2961968898 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:39.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 10 monclient(hunting): get_auth_request con 0x7f31ec13dbe0 auth_method 0 2024-09-16T05:20:39.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:39.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:39.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:39.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31eb7fe640 10 monclient(hunting): get_auth_request con 0x7f31ec0a5d00 auth_method 0 2024-09-16T05:20:39.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:39.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:39.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:39.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:39.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:39.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:39.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:39.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:39.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:39.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31eaffd640 1 --2- 172.21.15.114:0/2961968898 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f31ec137730 0x7f31ec135e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:39.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 10 monclient(hunting): handle_auth_done global_id 74577 payload 824 2024-09-16T05:20:39.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 1 -- 172.21.15.114:0/2961968898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31ec0a5d00 msgr2=0x7f31ec1358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 1 --2- 172.21.15.114:0/2961968898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31ec0a5d00 0x7f31ec1358e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:39.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 1 -- 172.21.15.114:0/2961968898 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f31ec137730 msgr2=0x7f31ec135e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:39.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 1 --2- 172.21.15.114:0/2961968898 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f31ec137730 0x7f31ec135e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:39.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 10 monclient: _finish_hunting 0 2024-09-16T05:20:39.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 1 monclient: found mon.noname-a 2024-09-16T05:20:39.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:39.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:39.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 1 -- 172.21.15.114:0/2961968898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31ec0a2180 con 0x7f31ec13dbe0 2024-09-16T05:20:39.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31eaffd640 10 monclient: get_auth_request con 0x7f31ec137730 auth_method 0 2024-09-16T05:20:39.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31eb7fe640 1 --2- 172.21.15.114:0/2961968898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31ec0a5d00 0x7f31ec1358e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:20:39.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31eaffd640 1 --2- 172.21.15.114:0/2961968898 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f31ec137730 0x7f31ec135e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:20:39.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31ebfff640 1 --2- 172.21.15.114:0/2961968898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31ec13dbe0 0x7f31ec09e800 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f31e40b0040 tx=0x7f31e4019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:39.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 10 client.?.objecter ms_handle_connect 0x7f31ec13dbe0 2024-09-16T05:20:39.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:20:39.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 1 -- 172.21.15.114:0/2961968898 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e40bd030 con 0x7f31ec13dbe0 2024-09-16T05:20:39.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:39.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:39.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 10 monclient: dump: 2024-09-16T05:20:39.811 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:39.811 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:39.811 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.811 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.811 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:39.811 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.811 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:39.811 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:39.811 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:39.811 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 10 monclient: _finish_auth 0 2024-09-16T05:20:39.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 10 monclient: _check_auth_tickets 2024-09-16T05:20:39.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:39.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 1 -- 172.21.15.114:0/2961968898 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31e40b8470 con 0x7f31ec13dbe0 2024-09-16T05:20:39.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:39.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 1 -- 172.21.15.114:0/2961968898 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e40b2040 con 0x7f31ec13dbe0 2024-09-16T05:20:39.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 5 monclient: authenticate success, global_id 74577 2024-09-16T05:20:39.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient: _renew_subs 2024-09-16T05:20:39.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:39.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e9ffb640 4 set_mon_vals no callback set 2024-09-16T05:20:39.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 -- 172.21.15.114:0/2961968898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31ec0a0d30 con 0x7f31ec13dbe0 2024-09-16T05:20:39.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:39.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:39.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 10 monclient: dump: 2024-09-16T05:20:39.813 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:39.813 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:39.813 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.813 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:39.813 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:39.813 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:39.813 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:39.814 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:39.814 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:39.814 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:39.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 client.74577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:20:39.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient: _renew_subs 2024-09-16T05:20:39.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:39.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 -- 172.21.15.114:0/2961968898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31ec0a2460 con 0x7f31ec13dbe0 2024-09-16T05:20:39.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 librados: init done 2024-09-16T05:20:39.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:39.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:39.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:39.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31f1e64640 1 -- 172.21.15.114:0/2961968898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31ec13f860 con 0x7f31ec13dbe0 2024-09-16T05:20:39.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 1 -- 172.21.15.114:0/2961968898 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f31e40c0070 con 0x7f31ec13dbe0 2024-09-16T05:20:39.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 1 --2- 172.21.15.114:0/2961968898 >> v2:172.21.15.149:6812/1185531135 conn(0x7f31cc037810 0x7f31cc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:39.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 1 -- 172.21.15.114:0/2961968898 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(529..529 src has 1..529) ==== 5861+0+0 (secure 0 0 0) 0x7f31e40e3e90 con 0x7f31ec13dbe0 2024-09-16T05:20:39.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 10 client.74577.objecter ms_dispatch 0x7f31ec000ba0 osd_map(529..529 src has 1..529) 2024-09-16T05:20:39.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 3 client.74577.objecter handle_osd_map got epochs [529,529] > 0 2024-09-16T05:20:39.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.799+0000 7f31e3fff640 3 client.74577.objecter handle_osd_map decoding full epoch 529 2024-09-16T05:20:39.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.803+0000 7f31e3fff640 20 client.74577.objecter dump_active .. 0 homeless 2024-09-16T05:20:39.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.803+0000 7f31eb7fe640 1 --2- 172.21.15.114:0/2961968898 >> v2:172.21.15.149:6812/1185531135 conn(0x7f31cc037810 0x7f31cc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:39.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.803+0000 7f31eb7fe640 10 monclient: get_auth_request con 0x7f31cc037810 auth_method 0 2024-09-16T05:20:39.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.803+0000 7f31e3fff640 10 client.74577.objecter ms_handle_connect 0x7f31cc037810 2024-09-16T05:20:39.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.803+0000 7f31eb7fe640 1 --2- 172.21.15.114:0/2961968898 >> v2:172.21.15.149:6812/1185531135 conn(0x7f31cc037810 0x7f31cc039cd0 secure :-1 s=READY pgs=3566 cs=0 l=1 rev1=1 crypto rx=0x7f31d4007fd0 tx=0x7f31d4007e40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:39.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.807+0000 7f31e3fff640 1 -- 172.21.15.114:0/2961968898 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f31e4016500 con 0x7f31ec13dbe0 2024-09-16T05:20:39.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.807+0000 7f31e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:39.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:39.807+0000 7f31e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:20:40.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.031+0000 7f31f1e64640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:20:40.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.031+0000 7f31f1e64640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:20:40.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.031+0000 7f31f1e64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:40.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.031+0000 7f31f1e64640 1 -- 172.21.15.114:0/2961968898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f31ec137b10 con 0x7f31ec13dbe0 2024-09-16T05:20:40.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.031+0000 7f31e3fff640 1 -- 172.21.15.114:0/2961968898 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f31e40ad440 con 0x7f31ec13dbe0 2024-09-16T05:20:40.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.031+0000 7f31e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:20:40.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.031+0000 7f31e3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:20:40.032 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.032 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=11,kv=11,logm=58} crc {auth=4105129816,config=2586702516,health=3601959098,kv=2812177559,logm=320448501}) 2024-09-16T05:20:40.032 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=11,logm=56} crc {auth=4034170520,config=2586702516,health=2701963991,kv=2812177559,logm=2557379116}) 2024-09-16T05:20:40.033 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.033 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=11,kv=11,logm=58} crc {auth=4105129816,config=2586702516,health=3601959098,kv=2812177559,logm=320448501}) 2024-09-16T05:20:40.033 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=11,logm=57} crc {auth=3675726669,config=2586702516,health=2701963991,kv=3340377701,logm=437974226}) 2024-09-16T05:20:40.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 10 librados: watch_flush enter 2024-09-16T05:20:40.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 10 librados: watch_flush exit 2024-09-16T05:20:40.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 20 client.74577.objecter shutdown clearing up homeless session... 2024-09-16T05:20:40.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 10 client.74577.objecter successfully canceled tick 2024-09-16T05:20:40.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 1 -- 172.21.15.114:0/2961968898 >> v2:172.21.15.149:6812/1185531135 conn(0x7f31cc037810 msgr2=0x7f31cc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:40.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 1 --2- 172.21.15.114:0/2961968898 >> v2:172.21.15.149:6812/1185531135 conn(0x7f31cc037810 0x7f31cc039cd0 secure :-1 s=READY pgs=3566 cs=0 l=1 rev1=1 crypto rx=0x7f31d4007fd0 tx=0x7f31d4007e40 comp rx=0 tx=0).stop 2024-09-16T05:20:40.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 10 monclient: shutdown 2024-09-16T05:20:40.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:40.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 1 -- 172.21.15.114:0/2961968898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31ec13dbe0 msgr2=0x7f31ec09e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:40.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 1 --2- 172.21.15.114:0/2961968898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31ec13dbe0 0x7f31ec09e800 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f31e40b0040 tx=0x7f31e4019590 comp rx=0 tx=0).stop 2024-09-16T05:20:40.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 1 -- 172.21.15.114:0/2961968898 shutdown_connections 2024-09-16T05:20:40.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 1 --2- 172.21.15.114:0/2961968898 >> v2:172.21.15.149:6812/1185531135 conn(0x7f31cc037810 0x7f31cc039cd0 unknown :-1 s=CLOSED pgs=3566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:40.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 1 --2- 172.21.15.114:0/2961968898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31ec13dbe0 0x7f31ec09e800 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:40.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 1 --2- 172.21.15.114:0/2961968898 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f31ec137730 0x7f31ec135e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:40.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 1 --2- 172.21.15.114:0/2961968898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31ec0a5d00 0x7f31ec1358e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:40.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 1 -- 172.21.15.114:0/2961968898 >> 172.21.15.114:0/2961968898 conn(0x7f31ec0a9c70 msgr2=0x7f31ec0a7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:40.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 1 -- 172.21.15.114:0/2961968898 shutdown_connections 2024-09-16T05:20:40.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 1 -- 172.21.15.114:0/2961968898 wait complete. 2024-09-16T05:20:40.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:40.035+0000 7f31f1e64640 1 librados: shutdown 2024-09-16T05:20:40.036 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.039+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.036 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.039+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2746185769}) 2024-09-16T05:20:40.036 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.039+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3838169724}) 2024-09-16T05:20:40.036 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.039+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.036 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.039+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2746185769}) 2024-09-16T05:20:40.036 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.039+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2911089855}) 2024-09-16T05:20:40.038 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.039+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.038 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.039+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4135085907}) 2024-09-16T05:20:40.038 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.039+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1845755580}) 2024-09-16T05:20:40.038 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.039+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.038 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.039+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4135085907}) 2024-09-16T05:20:40.038 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.039+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=337172031}) 2024-09-16T05:20:40.040 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.043+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.040 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.043+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3907932652}) 2024-09-16T05:20:40.040 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.043+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3628938371}) 2024-09-16T05:20:40.040 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.043+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.040 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.043+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3907932652}) 2024-09-16T05:20:40.040 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.043+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=320165568}) 2024-09-16T05:20:40.042 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.043+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.042 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.043+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2955493196}) 2024-09-16T05:20:40.042 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.043+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2959464800}) 2024-09-16T05:20:40.042 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.043+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.042 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.043+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2955493196}) 2024-09-16T05:20:40.043 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.043+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3457737146}) 2024-09-16T05:20:40.044 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.047+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.044 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.047+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000331375}) 2024-09-16T05:20:40.044 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.047+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3174522603}) 2024-09-16T05:20:40.044 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.047+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.045 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.047+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000331375}) 2024-09-16T05:20:40.045 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.047+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2005793236}) 2024-09-16T05:20:40.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.047+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.047+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=529665625}) 2024-09-16T05:20:40.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.047+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3279013500}) 2024-09-16T05:20:40.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.047+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.047+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=529665625}) 2024-09-16T05:20:40.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.047+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1924341903}) 2024-09-16T05:20:40.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.051+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.051+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1191076416}) 2024-09-16T05:20:40.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.051+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4044661584}) 2024-09-16T05:20:40.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.051+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.051+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1191076416}) 2024-09-16T05:20:40.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.051+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=194837485}) 2024-09-16T05:20:40.049 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.051+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.050 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.051+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957066548}) 2024-09-16T05:20:40.050 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.051+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2576553595}) 2024-09-16T05:20:40.050 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.051+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.050 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.051+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957066548}) 2024-09-16T05:20:40.050 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.051+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2305451965}) 2024-09-16T05:20:40.051 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.051 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2633529472}) 2024-09-16T05:20:40.051 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3826396537}) 2024-09-16T05:20:40.051 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.052 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2633529472}) 2024-09-16T05:20:40.052 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3588773084}) 2024-09-16T05:20:40.053 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:20:40.053 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.053 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151288033}) 2024-09-16T05:20:40.054 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565322964}) 2024-09-16T05:20:40.054 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.054 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151288033}) 2024-09-16T05:20:40.054 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4239273680}) 2024-09-16T05:20:40.054 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.055 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2436094462}) 2024-09-16T05:20:40.055 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2940928236}) 2024-09-16T05:20:40.055 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.055 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2436094462}) 2024-09-16T05:20:40.055 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.055+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1011491586}) 2024-09-16T05:20:40.056 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.059+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.056 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.059+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912208916}) 2024-09-16T05:20:40.056 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.059+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=420356950}) 2024-09-16T05:20:40.056 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.059+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.057 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.059+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912208916}) 2024-09-16T05:20:40.057 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.059+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2798022659}) 2024-09-16T05:20:40.058 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.059+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.058 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.059+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047668979}) 2024-09-16T05:20:40.058 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.059+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2542336654}) 2024-09-16T05:20:40.058 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.059+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.058 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.059+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047668979}) 2024-09-16T05:20:40.059 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.059+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3963021160}) 2024-09-16T05:20:40.059 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.063+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.060 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.063+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697403202}) 2024-09-16T05:20:40.060 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.063+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4103476869}) 2024-09-16T05:20:40.060 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.063+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.060 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.063+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697403202}) 2024-09-16T05:20:40.060 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.063+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2635688360}) 2024-09-16T05:20:40.061 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.063+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.061 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.063+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642338394}) 2024-09-16T05:20:40.061 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.063+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2069536100}) 2024-09-16T05:20:40.061 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.063+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.061 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.063+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642338394}) 2024-09-16T05:20:40.062 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.063+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1199952697}) 2024-09-16T05:20:40.063 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.063 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684490712}) 2024-09-16T05:20:40.063 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=759324371}) 2024-09-16T05:20:40.063 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.063 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684490712}) 2024-09-16T05:20:40.064 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3042198709}) 2024-09-16T05:20:40.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1264991433}) 2024-09-16T05:20:40.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2071141416}) 2024-09-16T05:20:40.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1264991433}) 2024-09-16T05:20:40.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279847745}) 2024-09-16T05:20:40.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=374507368}) 2024-09-16T05:20:40.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1565162081}) 2024-09-16T05:20:40.067 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.067 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=374507368}) 2024-09-16T05:20:40.067 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.067+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1893984443}) 2024-09-16T05:20:40.068 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.071+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.068 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.071+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1627900185}) 2024-09-16T05:20:40.069 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.071+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1035227063}) 2024-09-16T05:20:40.069 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.071+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.069 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.071+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1627900185}) 2024-09-16T05:20:40.069 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.071+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1394149256}) 2024-09-16T05:20:40.070 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.071+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.070 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.071+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3857757425}) 2024-09-16T05:20:40.071 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.071+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=580734403}) 2024-09-16T05:20:40.071 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.071+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.071 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.071+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3857757425}) 2024-09-16T05:20:40.071 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.071+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3304077652}) 2024-09-16T05:20:40.072 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.075+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.072 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.075+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987307610}) 2024-09-16T05:20:40.072 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.075+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=708900817}) 2024-09-16T05:20:40.072 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.075+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.072 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.075+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987307610}) 2024-09-16T05:20:40.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.075+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=763495970}) 2024-09-16T05:20:40.074 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.075+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.074 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.075+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642994311}) 2024-09-16T05:20:40.074 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.075+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=358095639}) 2024-09-16T05:20:40.074 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.075+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.074 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.075+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642994311}) 2024-09-16T05:20:40.075 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.075+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2191475396}) 2024-09-16T05:20:40.076 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.079+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.076 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.079+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425487028}) 2024-09-16T05:20:40.076 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.079+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2054711495}) 2024-09-16T05:20:40.076 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.079+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.076 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.079+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425487028}) 2024-09-16T05:20:40.076 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.079+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1528035350}) 2024-09-16T05:20:40.077 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.079+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.077 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.079+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3705563144}) 2024-09-16T05:20:40.078 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.079+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3186069172}) 2024-09-16T05:20:40.078 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.079+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.078 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.079+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3705563144}) 2024-09-16T05:20:40.078 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.079+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3367997094}) 2024-09-16T05:20:40.079 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.083+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.079 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.083+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3132279201}) 2024-09-16T05:20:40.080 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.083+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=18768573}) 2024-09-16T05:20:40.080 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.083+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.080 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.083+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3132279201}) 2024-09-16T05:20:40.080 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.083+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2004171700}) 2024-09-16T05:20:40.081 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.083+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.081 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.083+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3596914174}) 2024-09-16T05:20:40.082 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.083+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1425951866}) 2024-09-16T05:20:40.082 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.083+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.082 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.083+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3596914174}) 2024-09-16T05:20:40.082 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.083+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=524745535}) 2024-09-16T05:20:40.083 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.087+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.084 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.087+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2391505984}) 2024-09-16T05:20:40.084 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.087+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=608527159}) 2024-09-16T05:20:40.084 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.087+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.084 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.087+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2391505984}) 2024-09-16T05:20:40.084 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.087+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1445872513}) 2024-09-16T05:20:40.085 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.087+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.086 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.087+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314029446}) 2024-09-16T05:20:40.086 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.087+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1175852965}) 2024-09-16T05:20:40.086 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.087+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.086 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.087+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314029446}) 2024-09-16T05:20:40.086 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.087+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2186504522}) 2024-09-16T05:20:40.087 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.091+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.087 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.091+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1675843570}) 2024-09-16T05:20:40.087 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.091+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3516208283}) 2024-09-16T05:20:40.087 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.091+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.087 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.091+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1675843570}) 2024-09-16T05:20:40.088 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.091+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1125087550}) 2024-09-16T05:20:40.090 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.091+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.090 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.091+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1648304051}) 2024-09-16T05:20:40.090 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.091+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086942516}) 2024-09-16T05:20:40.090 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.091+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.090 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.091+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1648304051}) 2024-09-16T05:20:40.090 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.091+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=666471061}) 2024-09-16T05:20:40.091 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.095+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.091 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.095+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1184209577}) 2024-09-16T05:20:40.091 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.095+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=834579961}) 2024-09-16T05:20:40.091 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.095+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.091 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.095+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1184209577}) 2024-09-16T05:20:40.091 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.095+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3893250829}) 2024-09-16T05:20:40.093 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.095+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.093 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.095+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955075790}) 2024-09-16T05:20:40.093 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.095+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1736765777}) 2024-09-16T05:20:40.094 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.095+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.094 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.095+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955075790}) 2024-09-16T05:20:40.094 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.095+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1515303170}) 2024-09-16T05:20:40.095 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.095 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1428543753}) 2024-09-16T05:20:40.095 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2668046061}) 2024-09-16T05:20:40.095 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.095 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1428543753}) 2024-09-16T05:20:40.095 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=847374131}) 2024-09-16T05:20:40.097 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.097 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765252533}) 2024-09-16T05:20:40.097 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=176767440}) 2024-09-16T05:20:40.097 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.097 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765252533}) 2024-09-16T05:20:40.097 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1449894592}) 2024-09-16T05:20:40.098 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.098 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660098262}) 2024-09-16T05:20:40.098 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2163941585}) 2024-09-16T05:20:40.099 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.099 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660098262}) 2024-09-16T05:20:40.099 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.099+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170180770}) 2024-09-16T05:20:40.100 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.103+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.100 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.103+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1749951116}) 2024-09-16T05:20:40.100 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.103+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617758938}) 2024-09-16T05:20:40.100 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.103+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.101 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.103+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1749951116}) 2024-09-16T05:20:40.101 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.103+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2266004312}) 2024-09-16T05:20:40.102 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.103+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.102 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.103+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2579331331}) 2024-09-16T05:20:40.102 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.103+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3545312691}) 2024-09-16T05:20:40.102 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.103+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.102 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.103+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2579331331}) 2024-09-16T05:20:40.102 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.103+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1236361451}) 2024-09-16T05:20:40.104 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.107+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.104 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.107+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1686430293}) 2024-09-16T05:20:40.104 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.107+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=65364498}) 2024-09-16T05:20:40.104 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.107+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.104 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.107+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1686430293}) 2024-09-16T05:20:40.104 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.107+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1396415392}) 2024-09-16T05:20:40.105 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.107+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.105 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.107+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1512230705}) 2024-09-16T05:20:40.105 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.107+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=283496948}) 2024-09-16T05:20:40.106 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.107+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.106 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.107+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1512230705}) 2024-09-16T05:20:40.106 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.107+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2195001213}) 2024-09-16T05:20:40.107 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.111+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.107 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.111+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2769745715}) 2024-09-16T05:20:40.107 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.111+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=369100155}) 2024-09-16T05:20:40.107 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.111+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.108 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.111+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2769745715}) 2024-09-16T05:20:40.108 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.111+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3355658988}) 2024-09-16T05:20:40.109 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.111+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.109 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.111+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2218231810}) 2024-09-16T05:20:40.109 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.111+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4047707574}) 2024-09-16T05:20:40.109 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.111+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.109 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.111+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2218231810}) 2024-09-16T05:20:40.109 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.111+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4068722927}) 2024-09-16T05:20:40.111 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.111 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3389726282}) 2024-09-16T05:20:40.111 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1326982985}) 2024-09-16T05:20:40.111 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.111 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3389726282}) 2024-09-16T05:20:40.111 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3619510438}) 2024-09-16T05:20:40.113 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.113 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2886484850}) 2024-09-16T05:20:40.113 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2200551915}) 2024-09-16T05:20:40.113 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.113 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2886484850}) 2024-09-16T05:20:40.113 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=631428878}) 2024-09-16T05:20:40.115 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.115 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048311342}) 2024-09-16T05:20:40.115 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1716210196}) 2024-09-16T05:20:40.115 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.115 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.115+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048311342}) 2024-09-16T05:20:40.115 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.119+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2252808005}) 2024-09-16T05:20:40.117 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.119+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.117 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.119+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773427036}) 2024-09-16T05:20:40.117 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.119+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1824525463}) 2024-09-16T05:20:40.117 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.119+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.117 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.119+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773427036}) 2024-09-16T05:20:40.118 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.119+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=261476207}) 2024-09-16T05:20:40.119 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.119+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.119 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.119+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1247507725}) 2024-09-16T05:20:40.119 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.119+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1156056320}) 2024-09-16T05:20:40.119 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.119+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.119 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.119+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1247507725}) 2024-09-16T05:20:40.119 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.119+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3627849808}) 2024-09-16T05:20:40.120 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.123+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.120 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.123+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575313017}) 2024-09-16T05:20:40.120 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.123+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1052719409}) 2024-09-16T05:20:40.121 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.123+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.121 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.123+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575313017}) 2024-09-16T05:20:40.121 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.123+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=871929920}) 2024-09-16T05:20:40.126 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.123+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.126 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.123+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=937323913}) 2024-09-16T05:20:40.126 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.123+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4150208507}) 2024-09-16T05:20:40.126 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.123+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.126 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.123+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=937323913}) 2024-09-16T05:20:40.126 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.123+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=611078554}) 2024-09-16T05:20:40.126 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.127+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.126 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.127+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180024885}) 2024-09-16T05:20:40.126 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.127+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3586208366}) 2024-09-16T05:20:40.126 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.127+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.126 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.127+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180024885}) 2024-09-16T05:20:40.126 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.127+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=555054275}) 2024-09-16T05:20:40.127 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.127+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.127 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.127+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515320361}) 2024-09-16T05:20:40.127 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.127+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1494897621}) 2024-09-16T05:20:40.127 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.127+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.127 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.127+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515320361}) 2024-09-16T05:20:40.127 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.127+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2502511523}) 2024-09-16T05:20:40.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.131+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.131+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623465902}) 2024-09-16T05:20:40.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.131+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4046893747}) 2024-09-16T05:20:40.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.131+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.131+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623465902}) 2024-09-16T05:20:40.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.131+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1426182659}) 2024-09-16T05:20:40.130 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.131+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.130 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.131+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3523269209}) 2024-09-16T05:20:40.130 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.131+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=242981859}) 2024-09-16T05:20:40.130 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.131+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.130 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.131+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3523269209}) 2024-09-16T05:20:40.130 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.131+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=766928936}) 2024-09-16T05:20:40.133 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.135+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.133 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.135+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12} crc {logm=754815396,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=161914370}) 2024-09-16T05:20:40.133 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.135+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=76,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11} crc {logm=2900776915,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=3994884788}) 2024-09-16T05:20:40.133 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.135+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.133 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.135+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12} crc {logm=754815396,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=161914370}) 2024-09-16T05:20:40.133 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.135+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=76,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12} crc {logm=873737900,mdsmap=1078762840,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=161914370}) 2024-09-16T05:20:40.134 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.135+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.135 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.135+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=2,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=38} crc {mgrstat=2004832952,mon_config_key=1354545202,monmap=3045472550,osd_metadata=487805116,osd_pg_creating=3986562758,osd_snap=3820554222,osdmap=1345767598}) 2024-09-16T05:20:40.135 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.135+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=2,mon_config_key=16,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=33,osdmap=41} crc {mgrstat=2004832952,mon_config_key=2098536757,monmap=3045472550,osd_metadata=114301549,osd_pg_creating=3986562758,osd_snap=20400488,osdmap=4040736446}) 2024-09-16T05:20:40.135 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.135+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.135 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.135+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=2,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=38} crc {mgrstat=2004832952,mon_config_key=1354545202,monmap=3045472550,osd_metadata=487805116,osd_pg_creating=3986562758,osd_snap=3820554222,osdmap=1345767598}) 2024-09-16T05:20:40.135 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.135+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=2,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=43} crc {mgrstat=2004832952,mon_config_key=1589051853,monmap=3045472550,osd_metadata=487805116,osd_pg_creating=3986562758,osd_snap=2081235346,osdmap=2035437879}) 2024-09-16T05:20:40.137 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.139+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.137 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.139+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1105937446}) 2024-09-16T05:20:40.137 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.139+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2885012370}) 2024-09-16T05:20:40.137 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.139+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.137 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.139+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1105937446}) 2024-09-16T05:20:40.137 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.139+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2941156447}) 2024-09-16T05:20:40.139 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.143+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.139 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.143+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=27557007}) 2024-09-16T05:20:40.140 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.143+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2819501533}) 2024-09-16T05:20:40.140 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.143+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.140 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.143+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=27557007}) 2024-09-16T05:20:40.140 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.143+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3453140134}) 2024-09-16T05:20:40.141 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.143+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.141 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.143+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1600799945}) 2024-09-16T05:20:40.142 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.143+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1130634805}) 2024-09-16T05:20:40.142 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.143+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.142 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.143+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1600799945}) 2024-09-16T05:20:40.142 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.143+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3999977098}) 2024-09-16T05:20:40.143 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.147+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.143 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.147+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=129738475}) 2024-09-16T05:20:40.143 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.147+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=643874467}) 2024-09-16T05:20:40.144 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.147+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.144 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.147+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=129738475}) 2024-09-16T05:20:40.144 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.147+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1010542524}) 2024-09-16T05:20:40.146 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.147+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.146 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.147+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1243896118}) 2024-09-16T05:20:40.146 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.147+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1117591029}) 2024-09-16T05:20:40.146 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.147+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.147 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.147+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1243896118}) 2024-09-16T05:20:40.147 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.147+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1519972686}) 2024-09-16T05:20:40.150 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.151+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.150 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.151+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1478816267}) 2024-09-16T05:20:40.151 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.151+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1962177506}) 2024-09-16T05:20:40.151 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.151+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.151 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.151+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1478816267}) 2024-09-16T05:20:40.151 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.151+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=320923835}) 2024-09-16T05:20:40.154 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.155+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.154 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.155+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2668775256}) 2024-09-16T05:20:40.155 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.155+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3635063410}) 2024-09-16T05:20:40.155 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.155+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.155 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.155+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2668775256}) 2024-09-16T05:20:40.155 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.155+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1960388090}) 2024-09-16T05:20:40.157 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.159+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.157 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.159+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=28} crc {osdmap=4218663376}) 2024-09-16T05:20:40.157 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.159+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=28} crc {osdmap=3935588825}) 2024-09-16T05:20:40.157 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.159+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:20:40.157 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.159+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=28} crc {osdmap=4218663376}) 2024-09-16T05:20:40.157 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:20:40.159+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=28} crc {osdmap=1491208963}) 2024-09-16T05:20:41.054 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:20:41.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:20:41.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 10 monclient: get_monmap_and_config 2024-09-16T05:20:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 10 monclient: build_initial_monmap 2024-09-16T05:20:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 10 monclient: monmap: 2024-09-16T05:20:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:20:41.197823+0000 2024-09-16T05:20:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:20:41.197823+0000 2024-09-16T05:20:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:20:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:20:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:20:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:20:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:20:41.194 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 1 Processor -- start 2024-09-16T05:20:41.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 1 -- start start 2024-09-16T05:20:41.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 10 monclient: init 2024-09-16T05:20:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T05:20:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f799c09b6c0 0x7f799c0a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 10 monclient(hunting): picked mon.noname-b con 0x7f799c09b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f799c0a1400 addr v1:172.21.15.149:6789/0 2024-09-16T05:20:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f799c0a17e0 addr v1:172.21.15.114:6789/0 2024-09-16T05:20:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f799c09d4e0 con 0x7f799c0a17e0 2024-09-16T05:20:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f799c09f430 con 0x7f799c0a1400 2024-09-16T05:20:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f799c09e390 con 0x7f799c09b6c0 2024-09-16T05:20:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a3da7640 10 monclient(hunting): authenticate will time out at 1725.826931s 2024-09-16T05:20:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f799c09b6c0 0x7f799c0a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a2344640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f799c0a17e0 0x7f799c13cd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36770/0 (socket says 172.21.15.114:36770) 2024-09-16T05:20:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a2344640 1 -- 172.21.15.114:0/3230879196 learned_addr learned my addr 172.21.15.114:0/3230879196 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:20:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 10 monclient(hunting): get_auth_request con 0x7f799c09b6c0 auth_method 0 2024-09-16T05:20:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 10 monclient(hunting): handle_auth_done global_id 104111 payload 274 2024-09-16T05:20:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 1 -- 172.21.15.114:0/3230879196 >> v1:172.21.15.149:6789/0 conn(0x7f799c0a1400 legacy=0x7f799c139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 1 -- 172.21.15.114:0/3230879196 >> v1:172.21.15.114:6789/0 conn(0x7f799c0a17e0 legacy=0x7f799c13cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 10 monclient: _finish_hunting 0 2024-09-16T05:20:41.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 1 monclient: found mon.noname-b 2024-09-16T05:20:41.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:41.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:20:41.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a1b43640 1 -- 172.21.15.114:0/3230879196 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799c09c2f0 con 0x7f799c09b6c0 2024-09-16T05:20:41.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a0b41640 1 -- 172.21.15.114:0/3230879196 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2719667778 0 0) 0x7f799c09d4e0 con 0x7f799c0a17e0 2024-09-16T05:20:41.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.195+0000 7f79a0b41640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:20:41.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a1b43640 1 --2- 172.21.15.114:0/3230879196 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f799c09b6c0 0x7f799c0a0ec0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7998009d90 tx=0x7f799801c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9543e17361ffaa8a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a0b41640 1 -- 172.21.15.114:0/3230879196 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79980a66c0 con 0x7f799c09b6c0 2024-09-16T05:20:41.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a0b41640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a0b41640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:20:41.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a0b41640 10 monclient: dump: 2024-09-16T05:20:41.205 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.205 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.205 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.205 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.205 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.205 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.205 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.205 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.206 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.206 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a0b41640 10 monclient: _finish_auth 0 2024-09-16T05:20:41.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a0b41640 10 monclient: _check_auth_tickets 2024-09-16T05:20:41.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a0b41640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:41.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a0b41640 1 -- 172.21.15.114:0/3230879196 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79980a6860 con 0x7f799c09b6c0 2024-09-16T05:20:41.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 5 monclient: authenticate success, global_id 104111 2024-09-16T05:20:41.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:41.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a0b41640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:41.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 10 monclient: get_monmap_and_config success 2024-09-16T05:20:41.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a0b41640 1 -- 172.21.15.114:0/3230879196 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79980a6c30 con 0x7f799c09b6c0 2024-09-16T05:20:41.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 4 set_mon_vals no callback set 2024-09-16T05:20:41.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a0b41640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a0b41640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:20:41.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a0b41640 10 monclient: dump: 2024-09-16T05:20:41.207 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.207 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.207 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.207 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.207 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.207 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.207 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.207 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.207 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.207 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 10 monclient: shutdown 2024-09-16T05:20:41.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:41.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 -- 172.21.15.114:0/3230879196 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f799c09b6c0 msgr2=0x7f799c0a0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 --2- 172.21.15.114:0/3230879196 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f799c09b6c0 0x7f799c0a0ec0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7998009d90 tx=0x7f799801c7f0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 -- 172.21.15.114:0/3230879196 shutdown_connections 2024-09-16T05:20:41.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 --2- 172.21.15.114:0/3230879196 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f799c09b6c0 0x7f799c0a0ec0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 -- 172.21.15.114:0/3230879196 >> 172.21.15.114:0/3230879196 conn(0x7f799c0a1fb0 msgr2=0x7f799c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:41.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 -- 172.21.15.114:0/3230879196 shutdown_connections 2024-09-16T05:20:41.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 -- 172.21.15.114:0/3230879196 wait complete. 2024-09-16T05:20:41.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 10 monclient: build_initial_monmap 2024-09-16T05:20:41.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 10 monclient: monmap: 2024-09-16T05:20:41.208 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:41.208 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:20:41.208 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:20:41.208 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:20:41.208 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:41.209 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.209 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:20:41.209 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:20:41.209 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:20:41.209 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 librados: starting msgr at 2024-09-16T05:20:41.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 librados: starting objecter 2024-09-16T05:20:41.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 Processor -- start 2024-09-16T05:20:41.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 -- start start 2024-09-16T05:20:41.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 librados: setting wanted keys 2024-09-16T05:20:41.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 librados: calling monclient init 2024-09-16T05:20:41.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 10 monclient: init 2024-09-16T05:20:41.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:41.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:20:41.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f799c09b6c0 0x7f799c13dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 10 monclient(hunting): picked mon.noname-c con 0x7f799c09b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:41.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f799c0a1400 0x7f799c093fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 10 monclient(hunting): picked mon.noname-b con 0x7f799c0a1400 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:41.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f799c0a17e0 0x7f799c094510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 10 monclient(hunting): picked mon.noname-a con 0x7f799c0a17e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:41.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f799c140cb0 con 0x7f799c0a17e0 2024-09-16T05:20:41.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f799c0a1c10 con 0x7f799c0a1400 2024-09-16T05:20:41.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f799c140e30 con 0x7f799c09b6c0 2024-09-16T05:20:41.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:41.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a3da7640 10 monclient(hunting): authenticate will time out at 1725.831862s 2024-09-16T05:20:41.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a1b43640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f799c09b6c0 0x7f799c13dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a2344640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f799c0a17e0 0x7f799c094510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.199+0000 7f79a1b43640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f799c09b6c0 0x7f799c13dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:35612/0 (socket says 172.21.15.114:35612) 2024-09-16T05:20:41.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 1 -- 172.21.15.114:0/3081275197 learned_addr learned my addr 172.21.15.114:0/3081275197 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:41.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 10 monclient(hunting): get_auth_request con 0x7f799c09b6c0 auth_method 0 2024-09-16T05:20:41.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a2344640 10 monclient(hunting): get_auth_request con 0x7f799c0a17e0 auth_method 0 2024-09-16T05:20:41.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a2344640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a2344640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a2344640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a2344640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a2344640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a2344640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1342640 1 --2- 172.21.15.114:0/3081275197 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f799c0a1400 0x7f799c093fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1342640 10 monclient(hunting): get_auth_request con 0x7f799c0a1400 auth_method 0 2024-09-16T05:20:41.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1342640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1342640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1342640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 10 monclient(hunting): handle_auth_done global_id 104114 payload 824 2024-09-16T05:20:41.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 1 -- 172.21.15.114:0/3081275197 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f799c0a1400 msgr2=0x7f799c093fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 1 --2- 172.21.15.114:0/3081275197 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f799c0a1400 0x7f799c093fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 1 -- 172.21.15.114:0/3081275197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f799c0a17e0 msgr2=0x7f799c094510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 1 --2- 172.21.15.114:0/3081275197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f799c0a17e0 0x7f799c094510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 10 monclient: _finish_hunting 0 2024-09-16T05:20:41.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 1 monclient: found mon.noname-c 2024-09-16T05:20:41.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:41.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:20:41.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 1 -- 172.21.15.114:0/3081275197 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799c096110 con 0x7f799c09b6c0 2024-09-16T05:20:41.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a2344640 1 --2- 172.21.15.114:0/3081275197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f799c0a17e0 0x7f799c094510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:20:41.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1b43640 1 --2- 172.21.15.114:0/3081275197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f799c09b6c0 0x7f799c13dd00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f799801a420 tx=0x7f7998019590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 10 client.?.objecter ms_handle_connect 0x7f799c09b6c0 2024-09-16T05:20:41.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:20:41.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 1 -- 172.21.15.114:0/3081275197 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79980be030 con 0x7f799c09b6c0 2024-09-16T05:20:41.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:20:41.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 10 monclient: dump: 2024-09-16T05:20:41.214 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.214 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.214 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.214 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.214 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.214 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.214 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.214 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.215 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.215 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 10 monclient: _finish_auth 0 2024-09-16T05:20:41.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 10 monclient: _check_auth_tickets 2024-09-16T05:20:41.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:41.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 1 -- 172.21.15.114:0/3081275197 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79980ba740 con 0x7f799c09b6c0 2024-09-16T05:20:41.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a3da7640 5 monclient: authenticate success, global_id 104114 2024-09-16T05:20:41.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a3da7640 10 monclient: _renew_subs 2024-09-16T05:20:41.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a3da7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:20:41.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a3da7640 1 -- 172.21.15.114:0/3081275197 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f799c09aa90 con 0x7f799c09b6c0 2024-09-16T05:20:41.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:41.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 1 -- 172.21.15.114:0/3081275197 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79980bab30 con 0x7f799c09b6c0 2024-09-16T05:20:41.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:20:41.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 10 monclient: dump: 2024-09-16T05:20:41.216 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.216 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.216 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.216 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.216 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.216 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.216 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.216 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.216 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.216 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a3da7640 10 client.104114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:20:41.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a3da7640 10 monclient: _renew_subs 2024-09-16T05:20:41.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a3da7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:20:41.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a3da7640 1 -- 172.21.15.114:0/3081275197 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f799c097c80 con 0x7f799c09b6c0 2024-09-16T05:20:41.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a3da7640 1 librados: init done 2024-09-16T05:20:41.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f7993fff640 4 set_mon_vals no callback set 2024-09-16T05:20:41.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f797ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:41.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f797ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:41.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f797ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:20:41.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 1 -- 172.21.15.114:0/3081275197 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f79980c1070 con 0x7f799c09b6c0 2024-09-16T05:20:41.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f797ffff640 1 -- 172.21.15.114:0/3081275197 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7978005090 con 0x7f799c09b6c0 2024-09-16T05:20:41.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 1 --2- 172.21.15.114:0/3081275197 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7980037810 0x7f7980039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 1 -- 172.21.15.114:0/3081275197 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(529..529 src has 1..529) ==== 5861+0+0 (secure 0 0 0) 0x7f79980e4490 con 0x7f799c09b6c0 2024-09-16T05:20:41.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 10 client.104114.objecter ms_dispatch 0x7f799c000ba0 osd_map(529..529 src has 1..529) 2024-09-16T05:20:41.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 3 client.104114.objecter handle_osd_map got epochs [529,529] > 0 2024-09-16T05:20:41.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 3 client.104114.objecter handle_osd_map decoding full epoch 529 2024-09-16T05:20:41.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 20 client.104114.objecter dump_active .. 0 homeless 2024-09-16T05:20:41.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1342640 1 --2- 172.21.15.114:0/3081275197 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7980037810 0x7f7980039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1342640 10 monclient: get_auth_request con 0x7f7980037810 auth_method 0 2024-09-16T05:20:41.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79a1342640 1 --2- 172.21.15.114:0/3081275197 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7980037810 0x7f7980039cd0 secure :-1 s=READY pgs=3567 cs=0 l=1 rev1=1 crypto rx=0x7f798c007fd0 tx=0x7f798c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.203+0000 7f79927fc640 10 client.104114.objecter ms_handle_connect 0x7f7980037810 2024-09-16T05:20:41.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.207+0000 7f79927fc640 1 -- 172.21.15.114:0/3081275197 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7998016500 con 0x7f799c09b6c0 2024-09-16T05:20:41.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.207+0000 7f79927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:41.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.207+0000 7f79927fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:20:41.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.415+0000 7f797ffff640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:20:41.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.415+0000 7f797ffff640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:20:41.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.415+0000 7f797ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:20:41.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.415+0000 7f797ffff640 1 -- 172.21.15.114:0/3081275197 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7978005380 con 0x7f799c09b6c0 2024-09-16T05:20:41.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.415+0000 7f79927fc640 1 -- 172.21.15.114:0/3081275197 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f79980ad590 con 0x7f799c09b6c0 2024-09-16T05:20:41.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.415+0000 7f79927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:20:41.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.415+0000 7f79927fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:20:41.415 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:20:41.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 10 librados: watch_flush enter 2024-09-16T05:20:41.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 10 librados: watch_flush exit 2024-09-16T05:20:41.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 20 client.104114.objecter shutdown clearing up homeless session... 2024-09-16T05:20:41.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 10 client.104114.objecter successfully canceled tick 2024-09-16T05:20:41.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 1 -- 172.21.15.114:0/3081275197 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7980037810 msgr2=0x7f7980039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 1 --2- 172.21.15.114:0/3081275197 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7980037810 0x7f7980039cd0 secure :-1 s=READY pgs=3567 cs=0 l=1 rev1=1 crypto rx=0x7f798c007fd0 tx=0x7f798c009040 comp rx=0 tx=0).stop 2024-09-16T05:20:41.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 10 monclient: shutdown 2024-09-16T05:20:41.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:41.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 1 -- 172.21.15.114:0/3081275197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f799c09b6c0 msgr2=0x7f799c13dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 1 --2- 172.21.15.114:0/3081275197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f799c09b6c0 0x7f799c13dd00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f799801a420 tx=0x7f7998019590 comp rx=0 tx=0).stop 2024-09-16T05:20:41.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 1 -- 172.21.15.114:0/3081275197 shutdown_connections 2024-09-16T05:20:41.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 1 --2- 172.21.15.114:0/3081275197 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7980037810 0x7f7980039cd0 unknown :-1 s=CLOSED pgs=3567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 1 --2- 172.21.15.114:0/3081275197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f799c0a17e0 0x7f799c094510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 1 --2- 172.21.15.114:0/3081275197 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f799c0a1400 0x7f799c093fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 1 --2- 172.21.15.114:0/3081275197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f799c09b6c0 0x7f799c13dd00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 1 -- 172.21.15.114:0/3081275197 >> 172.21.15.114:0/3081275197 conn(0x7f799c0a1fb0 msgr2=0x7f799c072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:41.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 1 -- 172.21.15.114:0/3081275197 shutdown_connections 2024-09-16T05:20:41.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 1 -- 172.21.15.114:0/3081275197 wait complete. 2024-09-16T05:20:41.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.419+0000 7f797ffff640 1 librados: shutdown 2024-09-16T05:20:41.434 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:20:41.435 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:20:41.435 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:20:41.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:20:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 10 monclient: get_monmap_and_config 2024-09-16T05:20:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 10 monclient: build_initial_monmap 2024-09-16T05:20:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 10 monclient: monmap: 2024-09-16T05:20:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:20:41.556799+0000 2024-09-16T05:20:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:20:41.556799+0000 2024-09-16T05:20:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.553 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:20:41.553 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:20:41.553 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:20:41.553 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:20:41.553 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:20:41.553 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 1 Processor -- start 2024-09-16T05:20:41.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 1 -- start start 2024-09-16T05:20:41.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 10 monclient: init 2024-09-16T05:20:41.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:41.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-16T05:20:41.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f9c0809b6c0 0x7f9c080a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 10 monclient(hunting): picked mon.noname-c con 0x7f9c0809b6c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:20:41.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9c080a1400 addr v1:172.21.15.149:6789/0 2024-09-16T05:20:41.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c080a1810 0x7f9c0813cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 10 monclient(hunting): picked mon.noname-b con 0x7f9c080a1810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:41.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c0809d4e0 con 0x7f9c080a1400 2024-09-16T05:20:41.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f9c0809e390 con 0x7f9c0809b6c0 2024-09-16T05:20:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9c0809ece0 con 0x7f9c080a1810 2024-09-16T05:20:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0fc0d640 10 monclient(hunting): authenticate will time out at 1726.186065s 2024-09-16T05:20:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c080a1810 0x7f9c0813cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c080a1810 0x7f9c0813cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:35628/0 (socket says 172.21.15.114:35628) 2024-09-16T05:20:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 1 -- 172.21.15.114:0/3003139751 learned_addr learned my addr 172.21.15.114:0/3003139751 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 10 monclient(hunting): get_auth_request con 0x7f9c080a1810 auth_method 0 2024-09-16T05:20:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0d9a9640 1 --2- 172.21.15.114:0/3003139751 >> v2:172.21.15.149:3300/0 conn(0x7f9c0809b6c0 0x7f9c080a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0d9a9640 10 monclient(hunting): get_auth_request con 0x7f9c0809b6c0 auth_method 0 2024-09-16T05:20:41.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0d9a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0d9a9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0d9a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0d9a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0d9a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0d9a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0c9a7640 1 -- 172.21.15.114:0/3003139751 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 196098682 0 0) 0x7f9c0809d4e0 con 0x7f9c080a1400 2024-09-16T05:20:41.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0c9a7640 10 monclient(hunting): _negotiate 2024-09-16T05:20:41.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0c9a7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0c9a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0c9a7640 10 monclient(hunting): my global_id is 64660 2024-09-16T05:20:41.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0c9a7640 1 -- 172.21.15.114:0/3003139751 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bf4003530 con 0x7f9c080a1400 2024-09-16T05:20:41.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 10 monclient(hunting): handle_auth_done global_id 104117 payload 274 2024-09-16T05:20:41.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 1 -- 172.21.15.114:0/3003139751 >> v2:172.21.15.149:3300/0 conn(0x7f9c0809b6c0 msgr2=0x7f9c080a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:41.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 1 --2- 172.21.15.114:0/3003139751 >> v2:172.21.15.149:3300/0 conn(0x7f9c0809b6c0 0x7f9c080a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 1 -- 172.21.15.114:0/3003139751 >> v1:172.21.15.149:6789/0 conn(0x7f9c080a1400 legacy=0x7f9c08139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 10 monclient: _finish_hunting 0 2024-09-16T05:20:41.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 1 monclient: found mon.noname-b 2024-09-16T05:20:41.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:41.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:20:41.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.555+0000 7f9c0e1aa640 1 -- 172.21.15.114:0/3003139751 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c0809f430 con 0x7f9c080a1810 2024-09-16T05:20:41.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0e1aa640 1 --2- 172.21.15.114:0/3003139751 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c080a1810 0x7f9c0813cd70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009e50 tx=0x7f9c0001c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a626e28fed43629 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0c9a7640 1 -- 172.21.15.114:0/3003139751 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c000a56a0 con 0x7f9c080a1810 2024-09-16T05:20:41.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0c9a7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0c9a7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:20:41.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0c9a7640 10 monclient: dump: 2024-09-16T05:20:41.573 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.573 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.573 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.573 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.573 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.573 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.573 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.574 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.574 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.574 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0c9a7640 10 monclient: _finish_auth 0 2024-09-16T05:20:41.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0c9a7640 10 monclient: _check_auth_tickets 2024-09-16T05:20:41.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0c9a7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:41.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 5 monclient: authenticate success, global_id 104117 2024-09-16T05:20:41.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:41.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0c9a7640 1 -- 172.21.15.114:0/3003139751 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c000a5840 con 0x7f9c080a1810 2024-09-16T05:20:41.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0c9a7640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:41.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0c9a7640 1 -- 172.21.15.114:0/3003139751 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c000a5bf0 con 0x7f9c080a1810 2024-09-16T05:20:41.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 10 monclient: get_monmap_and_config success 2024-09-16T05:20:41.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 4 set_mon_vals no callback set 2024-09-16T05:20:41.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0c9a7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0c9a7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:20:41.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0c9a7640 10 monclient: dump: 2024-09-16T05:20:41.576 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.576 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.576 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.576 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.577 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.577 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.577 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.577 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.577 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.577 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 10 monclient: shutdown 2024-09-16T05:20:41.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:41.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 -- 172.21.15.114:0/3003139751 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c080a1810 msgr2=0x7f9c0813cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 --2- 172.21.15.114:0/3003139751 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c080a1810 0x7f9c0813cd70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009e50 tx=0x7f9c0001c8f0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 -- 172.21.15.114:0/3003139751 shutdown_connections 2024-09-16T05:20:41.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 --2- 172.21.15.114:0/3003139751 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c080a1810 0x7f9c0813cd70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 --2- 172.21.15.114:0/3003139751 >> v2:172.21.15.149:3300/0 conn(0x7f9c0809b6c0 0x7f9c080a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 -- 172.21.15.114:0/3003139751 >> 172.21.15.114:0/3003139751 conn(0x7f9c080a1fb0 msgr2=0x7f9c080a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:41.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 -- 172.21.15.114:0/3003139751 shutdown_connections 2024-09-16T05:20:41.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 -- 172.21.15.114:0/3003139751 wait complete. 2024-09-16T05:20:41.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 10 monclient: build_initial_monmap 2024-09-16T05:20:41.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 10 monclient: monmap: 2024-09-16T05:20:41.578 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:41.578 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:20:41.578 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:20:41.578 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:20:41.578 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:41.579 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.579 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:20:41.579 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:20:41.579 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:20:41.579 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 librados: starting msgr at 2024-09-16T05:20:41.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 librados: starting objecter 2024-09-16T05:20:41.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 Processor -- start 2024-09-16T05:20:41.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 -- start start 2024-09-16T05:20:41.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 librados: setting wanted keys 2024-09-16T05:20:41.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 librados: calling monclient init 2024-09-16T05:20:41.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 10 monclient: init 2024-09-16T05:20:41.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:41.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:20:41.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c0809b6c0 0x7f9c0813cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 10 monclient(hunting): picked mon.noname-b con 0x7f9c0809b6c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:41.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c080a1400 0x7f9c080994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 10 monclient(hunting): picked mon.noname-c con 0x7f9c080a1400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:41.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c080a1810 0x7f9c08099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 10 monclient(hunting): picked mon.noname-a con 0x7f9c080a1810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:41.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0809cad0 con 0x7f9c080a1810 2024-09-16T05:20:41.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0809dd40 con 0x7f9c0809b6c0 2024-09-16T05:20:41.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9c0813fad0 con 0x7f9c080a1400 2024-09-16T05:20:41.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:41.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.559+0000 7f9c0fc0d640 10 monclient(hunting): authenticate will time out at 1726.191962s 2024-09-16T05:20:41.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c080a1400 0x7f9c080994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c080a1400 0x7f9c080994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:35632/0 (socket says 172.21.15.114:35632) 2024-09-16T05:20:41.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 1 -- 172.21.15.114:0/179965563 learned_addr learned my addr 172.21.15.114:0/179965563 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:41.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 10 monclient(hunting): get_auth_request con 0x7f9c080a1400 auth_method 0 2024-09-16T05:20:41.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c0809b6c0 0x7f9c0813cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 10 monclient(hunting): get_auth_request con 0x7f9c0809b6c0 auth_method 0 2024-09-16T05:20:41.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0e1aa640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c080a1810 0x7f9c08099a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0e1aa640 10 monclient(hunting): get_auth_request con 0x7f9c080a1810 auth_method 0 2024-09-16T05:20:41.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0e1aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0e1aa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0e1aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0e1aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0e1aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0e1aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 10 monclient(hunting): handle_auth_done global_id 104120 payload 824 2024-09-16T05:20:41.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 1 -- 172.21.15.114:0/179965563 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c0809b6c0 msgr2=0x7f9c0813cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c0809b6c0 0x7f9c0813cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 1 -- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c080a1810 msgr2=0x7f9c08099a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c080a1810 0x7f9c08099a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 10 monclient: _finish_hunting 0 2024-09-16T05:20:41.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 1 monclient: found mon.noname-c 2024-09-16T05:20:41.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:41.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:20:41.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 1 -- 172.21.15.114:0/179965563 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c08097920 con 0x7f9c080a1400 2024-09-16T05:20:41.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d1a8640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c080a1400 0x7f9c080994e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9bf800c420 tx=0x7f9bf800c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 client.?.objecter ms_handle_connect 0x7f9c080a1400 2024-09-16T05:20:41.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:20:41.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 1 -- 172.21.15.114:0/179965563 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8017070 con 0x7f9c080a1400 2024-09-16T05:20:41.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:20:41.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 monclient: dump: 2024-09-16T05:20:41.585 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.585 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.585 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.585 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.585 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.585 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.585 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.585 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.585 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.585 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 monclient: _finish_auth 0 2024-09-16T05:20:41.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:20:41.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:41.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 1 -- 172.21.15.114:0/179965563 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bf80052d0 con 0x7f9c080a1400 2024-09-16T05:20:41.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:41.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 1 -- 172.21.15.114:0/179965563 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8003650 con 0x7f9c080a1400 2024-09-16T05:20:41.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:20:41.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 monclient: dump: 2024-09-16T05:20:41.586 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.586 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.587 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.587 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.587 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.587 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.587 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.587 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.587 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.587 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfffff640 4 set_mon_vals no callback set 2024-09-16T05:20:41.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0fc0d640 5 monclient: authenticate success, global_id 104120 2024-09-16T05:20:41.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0fc0d640 10 monclient: _renew_subs 2024-09-16T05:20:41.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0fc0d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:20:41.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0fc0d640 1 -- 172.21.15.114:0/179965563 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c08097c00 con 0x7f9c080a1400 2024-09-16T05:20:41.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0fc0d640 10 client.104120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:20:41.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0fc0d640 10 monclient: _renew_subs 2024-09-16T05:20:41.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0fc0d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:20:41.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0fc0d640 1 -- 172.21.15.114:0/179965563 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c08072ba0 con 0x7f9c080a1400 2024-09-16T05:20:41.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0fc0d640 1 librados: init done 2024-09-16T05:20:41.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0fc0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:41.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0fc0d640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c0802d7d0 0x7f9c08099f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0fc0d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0fc0d640 1 -- 172.21.15.114:0/179965563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0809b180 con 0x7f9c0802d7d0 2024-09-16T05:20:41.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 1 -- 172.21.15.114:0/179965563 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9bf801a070 con 0x7f9c080a1400 2024-09-16T05:20:41.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c0802d7d0 0x7f9c08099f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 10 monclient: get_auth_request con 0x7f9c0802d7d0 auth_method 0 2024-09-16T05:20:41.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 1 --2- 172.21.15.114:0/179965563 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9bec037840 0x7f9bec039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 1 -- 172.21.15.114:0/179965563 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(529..529 src has 1..529) ==== 5861+0+0 (secure 0 0 0) 0x7f9bf8048300 con 0x7f9c080a1400 2024-09-16T05:20:41.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 client.104120.objecter ms_dispatch 0x7f9c08000ba0 osd_map(529..529 src has 1..529) 2024-09-16T05:20:41.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 3 client.104120.objecter handle_osd_map got epochs [529,529] > 0 2024-09-16T05:20:41.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 3 client.104120.objecter handle_osd_map decoding full epoch 529 2024-09-16T05:20:41.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 10 monclient(hunting): handle_auth_done global_id 74589 payload 824 2024-09-16T05:20:41.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 1 -- 172.21.15.114:0/179965563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c0802dbb0 con 0x7f9c0802d7d0 2024-09-16T05:20:41.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 20 client.104120.objecter dump_active .. 0 homeless 2024-09-16T05:20:41.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0d9a9640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c0802d7d0 0x7f9c08099f60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9c04011900 tx=0x7f9c04011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 client.104120.objecter ms_handle_connect 0x7f9c0802d7d0 2024-09-16T05:20:41.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 client.104120.objecter resend_mon_ops 2024-09-16T05:20:41.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0e1aa640 1 --2- 172.21.15.114:0/179965563 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9bec037840 0x7f9bec039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 1 -- 172.21.15.114:0/179965563 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c04019070 con 0x7f9c0802d7d0 2024-09-16T05:20:41.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 monclient: dump: 2024-09-16T05:20:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.591 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.592 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0e1aa640 10 monclient: get_auth_request con 0x7f9bec037840 auth_method 0 2024-09-16T05:20:41.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 1 -- 172.21.15.114:0/179965563 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9c04004300 con 0x7f9c0802d7d0 2024-09-16T05:20:41.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:41.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:20:41.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 1 -- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c0802d7d0 msgr2=0x7f9c08099f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c0802d7d0 0x7f9c08099f60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9c04011900 tx=0x7f9c04011dc0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 1 -- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c0802d7d0 msgr2=0x7f9c08099f60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:20:41.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c0802d7d0 0x7f9c08099f60 secure :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9c04011900 tx=0x7f9c04011dc0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9bfe7fc640 10 client.104120.objecter ms_handle_connect 0x7f9bec037840 2024-09-16T05:20:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.563+0000 7f9c0e1aa640 1 --2- 172.21.15.114:0/179965563 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9bec037840 0x7f9bec039d00 secure :-1 s=READY pgs=3568 cs=0 l=1 rev1=1 crypto rx=0x7f9c000ae000 tx=0x7f9c000a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:20:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c08062ea0 0x7f9c080632b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 1 -- 172.21.15.114:0/179965563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9c08069390 con 0x7f9c08062ea0 2024-09-16T05:20:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0d9a9640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c08062ea0 0x7f9c080632b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0d9a9640 10 monclient: get_auth_request con 0x7f9c08062ea0 auth_method 0 2024-09-16T05:20:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0d9a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0d9a9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0d9a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0d9a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0d9a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0d9a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0d9a9640 10 monclient(hunting): handle_auth_done global_id 74592 payload 824 2024-09-16T05:20:41.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0d9a9640 1 -- 172.21.15.114:0/179965563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9c0806df20 con 0x7f9c08062ea0 2024-09-16T05:20:41.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0d9a9640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c08062ea0 0x7f9c080632b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9c04011900 tx=0x7f9c040046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9bfe7fc640 10 client.104120.objecter ms_handle_connect 0x7f9c08062ea0 2024-09-16T05:20:41.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9bfe7fc640 10 client.104120.objecter resend_mon_ops 2024-09-16T05:20:41.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9bfe7fc640 1 -- 172.21.15.114:0/179965563 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c04019040 con 0x7f9c08062ea0 2024-09-16T05:20:41.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9bfe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9bfe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:41.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9bfe7fc640 10 monclient: dump: 2024-09-16T05:20:41.594 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.594 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.594 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.595 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.595 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.595 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.595 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.595 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.595 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.595 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9bfe7fc640 1 -- 172.21.15.114:0/179965563 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f9c04003d10 con 0x7f9c08062ea0 2024-09-16T05:20:41.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9bfe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:20:41.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9bfe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:20:41.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9bfe7fc640 1 -- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c08062ea0 msgr2=0x7f9c080632b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9bfe7fc640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c08062ea0 0x7f9c080632b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9c04011900 tx=0x7f9c040046d0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9bfe7fc640 1 -- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c08062ea0 msgr2=0x7f9c080632b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:20:41.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9bfe7fc640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c08062ea0 0x7f9c080632b0 secure :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9c04011900 tx=0x7f9c040046d0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.596 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:20:41.596 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:20:41.596 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:20:41.596 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:20:41.596 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 56, 2024-09-16T05:20:41.596 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 400626, 2024-09-16T05:20:41.596 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:20:41.596 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:20:41.597 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:20:41.597 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:20:41.597 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:41.597 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 6, 2024-09-16T05:20:41.597 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:20:41.597 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:20:41.597 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:20:41.597 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:41.597 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:41.598 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:41.598 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:41.598 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:41.598 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:41.598 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:41.598 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:41.598 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:41.598 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:41.598 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:41.598 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:41.599 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:20:41.599 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:20:41.599 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:41.599 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:41.599 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:41.599 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:41.599 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:41.599 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:41.599 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:41.599 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:41.599 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:41.599 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:41.599 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:41.600 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.600 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.600 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:20:41.600 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:20:41.600 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:20:41.600 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:20:41.601 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:20:41.601 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:20:41.601 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:20:41.601 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:20:41.601 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:20:41.601 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:20:41.602 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:20:41.602 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:20:41.602 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:20:41.602 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:20:41.602 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:20:41.602 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:20:41.603 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:20:41.603 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:41.603 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:41.603 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:41.603 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:41.603 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:41.604 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:41.604 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:41.604 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:41.604 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:41.604 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:41.604 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:41.604 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:41.605 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:20:41.605 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.605 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:20:41.605 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:20:41.605 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:20:41.606 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:41.606 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:41.606 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.606 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:41.606 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:20:41.606 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.607 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.607 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:41.607 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:20:41.607 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.607 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.607 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.608 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.608 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:20:41.608 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:20:41.608 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:41.608 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:41.608 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:41.609 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.609 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:20:41.609 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:20:41.609 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:41.609 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:41.610 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.610 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:41.610 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:20:41.610 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.610 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.610 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.610 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:41.611 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:20:41.611 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.611 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.611 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.611 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:20:41.612 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:20:41.612 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:41.612 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:41.612 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:41.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.612 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:20:41.613 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:20:41.613 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:41.613 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:41.613 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.613 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:41.613 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:20:41.614 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.614 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.614 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.614 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:41.614 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:20:41.614 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.614 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.615 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.615 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.615 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:20:41.615 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:20:41.615 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:41.615 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:41.616 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:41.616 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.616 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.616 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.616 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:20:41.616 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:20:41.616 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.617 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:41.617 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:41.617 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:20:41.617 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.617 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:41.617 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:20:41.618 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.618 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:41.618 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:41.618 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:20:41.618 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.618 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:41.618 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:20:41.619 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.619 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:41.619 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:41.619 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:20:41.619 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.619 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.619 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.620 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:20:41.620 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:20:41.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 10 librados: watch_flush enter 2024-09-16T05:20:41.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 10 librados: watch_flush exit 2024-09-16T05:20:41.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 20 client.104120.objecter shutdown clearing up homeless session... 2024-09-16T05:20:41.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 10 client.104120.objecter successfully canceled tick 2024-09-16T05:20:41.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 1 -- 172.21.15.114:0/179965563 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9bec037840 msgr2=0x7f9bec039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 1 --2- 172.21.15.114:0/179965563 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9bec037840 0x7f9bec039d00 secure :-1 s=READY pgs=3568 cs=0 l=1 rev1=1 crypto rx=0x7f9c000ae000 tx=0x7f9c000a7070 comp rx=0 tx=0).stop 2024-09-16T05:20:41.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 10 monclient: shutdown 2024-09-16T05:20:41.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:41.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 1 -- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c080a1400 msgr2=0x7f9c080994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c080a1400 0x7f9c080994e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9bf800c420 tx=0x7f9bf800c8e0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0e1aa640 1 -- 172.21.15.114:0/179965563 reap_dead start 2024-09-16T05:20:41.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 1 -- 172.21.15.114:0/179965563 shutdown_connections 2024-09-16T05:20:41.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 1 --2- 172.21.15.114:0/179965563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c080a1400 0x7f9c080994e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 1 -- 172.21.15.114:0/179965563 >> 172.21.15.114:0/179965563 conn(0x7f9c080a1fb0 msgr2=0x7f9c08071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:41.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 1 -- 172.21.15.114:0/179965563 shutdown_connections 2024-09-16T05:20:41.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 1 -- 172.21.15.114:0/179965563 wait complete. 2024-09-16T05:20:41.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.571+0000 7f9c0fc0d640 1 librados: shutdown 2024-09-16T05:20:41.626 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:20:41.626 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:20:41.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:20:41.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 10 monclient: get_monmap_and_config 2024-09-16T05:20:41.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 10 monclient: build_initial_monmap 2024-09-16T05:20:41.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 10 monclient: monmap: 2024-09-16T05:20:41.748 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:41.748 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.748 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:20:41.752818+0000 2024-09-16T05:20:41.748 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:20:41.752818+0000 2024-09-16T05:20:41.748 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:41.748 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.748 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:20:41.748 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:20:41.748 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:20:41.748 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:20:41.748 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:20:41.748 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 1 Processor -- start 2024-09-16T05:20:41.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 1 -- start start 2024-09-16T05:20:41.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 10 monclient: init 2024-09-16T05:20:41.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:41.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-16T05:20:41.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f538c09b6c0 addr v1:172.21.15.149:6789/0 2024-09-16T05:20:41.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f538c0a15d0 0x7f538c13a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 10 monclient(hunting): picked mon.noname-c con 0x7f538c0a15d0 addr v2:172.21.15.149:3300/0 2024-09-16T05:20:41.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f538c0a19e0 0x7f538c13cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 10 monclient(hunting): picked mon.noname-a con 0x7f538c0a19e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:20:41.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f538c09d4e0 con 0x7f538c09b6c0 2024-09-16T05:20:41.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f538c09e390 con 0x7f538c0a19e0 2024-09-16T05:20:41.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f538c09ece0 con 0x7f538c0a15d0 2024-09-16T05:20:41.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:41.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f53927ff640 10 monclient(hunting): authenticate will time out at 1726.382013s 2024-09-16T05:20:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f538c0a19e0 0x7f538c13cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f538c0a19e0 0x7f538c13cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53558/0 (socket says 172.21.15.114:53558) 2024-09-16T05:20:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 1 -- 172.21.15.114:0/2298141649 learned_addr learned my addr 172.21.15.114:0/2298141649 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 10 monclient(hunting): get_auth_request con 0x7f538c0a19e0 auth_method 0 2024-09-16T05:20:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 10 monclient(hunting): handle_auth_done global_id 74595 payload 274 2024-09-16T05:20:41.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 1 -- 172.21.15.114:0/2298141649 >> v2:172.21.15.149:3300/0 conn(0x7f538c0a15d0 msgr2=0x7f538c13a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:41.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f538b7fe640 1 --2- 172.21.15.114:0/2298141649 >> v2:172.21.15.149:3300/0 conn(0x7f538c0a15d0 0x7f538c13a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 1 --2- 172.21.15.114:0/2298141649 >> v2:172.21.15.149:3300/0 conn(0x7f538c0a15d0 0x7f538c13a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 1 -- 172.21.15.114:0/2298141649 >> v1:172.21.15.149:6789/0 conn(0x7f538c09b6c0 legacy=0x7f538c0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:41.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 10 monclient: _finish_hunting 0 2024-09-16T05:20:41.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 1 monclient: found mon.noname-a 2024-09-16T05:20:41.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:41.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:41.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.751+0000 7f5390d9c640 1 -- 172.21.15.114:0/2298141649 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f538c09f430 con 0x7f538c0a19e0 2024-09-16T05:20:41.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f5390d9c640 1 --2- 172.21.15.114:0/2298141649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f538c0a19e0 0x7f538c13cba0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f538c09c630 tx=0x7f538401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc77baaeff616e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538affd640 1 -- 172.21.15.114:0/2298141649 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53840b0070 con 0x7f538c0a19e0 2024-09-16T05:20:41.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:41.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538affd640 10 monclient: dump: 2024-09-16T05:20:41.766 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.766 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.766 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.766 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.766 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.767 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.767 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.767 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.767 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.767 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538affd640 10 monclient: _finish_auth 0 2024-09-16T05:20:41.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538affd640 10 monclient: _check_auth_tickets 2024-09-16T05:20:41.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:41.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538affd640 1 -- 172.21.15.114:0/2298141649 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53840ab600 con 0x7f538c0a19e0 2024-09-16T05:20:41.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 5 monclient: authenticate success, global_id 74595 2024-09-16T05:20:41.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:41.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:41.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 10 monclient: get_monmap_and_config success 2024-09-16T05:20:41.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538affd640 1 -- 172.21.15.114:0/2298141649 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53840ab9d0 con 0x7f538c0a19e0 2024-09-16T05:20:41.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 4 set_mon_vals no callback set 2024-09-16T05:20:41.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:41.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538affd640 10 monclient: dump: 2024-09-16T05:20:41.768 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.768 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.768 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.768 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.768 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.769 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.769 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.769 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.769 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.769 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 10 monclient: shutdown 2024-09-16T05:20:41.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:41.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 -- 172.21.15.114:0/2298141649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f538c0a19e0 msgr2=0x7f538c13cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 --2- 172.21.15.114:0/2298141649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f538c0a19e0 0x7f538c13cba0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f538c09c630 tx=0x7f538401c430 comp rx=0 tx=0).stop 2024-09-16T05:20:41.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 -- 172.21.15.114:0/2298141649 shutdown_connections 2024-09-16T05:20:41.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 --2- 172.21.15.114:0/2298141649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f538c0a19e0 0x7f538c13cba0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 --2- 172.21.15.114:0/2298141649 >> v2:172.21.15.149:3300/0 conn(0x7f538c0a15d0 0x7f538c13a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 -- 172.21.15.114:0/2298141649 >> 172.21.15.114:0/2298141649 conn(0x7f538c0a1fb0 msgr2=0x7f538c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:41.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 -- 172.21.15.114:0/2298141649 shutdown_connections 2024-09-16T05:20:41.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 -- 172.21.15.114:0/2298141649 wait complete. 2024-09-16T05:20:41.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 10 monclient: build_initial_monmap 2024-09-16T05:20:41.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 10 monclient: monmap: 2024-09-16T05:20:41.770 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:41.770 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:20:41.770 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:20:41.770 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:20:41.770 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:41.770 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.770 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:20:41.770 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:20:41.770 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:20:41.770 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 librados: starting msgr at 2024-09-16T05:20:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 librados: starting objecter 2024-09-16T05:20:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 Processor -- start 2024-09-16T05:20:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 -- start start 2024-09-16T05:20:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 librados: setting wanted keys 2024-09-16T05:20:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 librados: calling monclient init 2024-09-16T05:20:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 10 monclient: init 2024-09-16T05:20:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:20:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f538c09b6c0 0x7f538c0947c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 10 monclient(hunting): picked mon.noname-a con 0x7f538c09b6c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f538c0a15d0 0x7f538c091b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 10 monclient(hunting): picked mon.noname-b con 0x7f538c0a15d0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c0a19e0 0x7f538c092040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 10 monclient(hunting): picked mon.noname-c con 0x7f538c0a19e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f538c13f900 con 0x7f538c09b6c0 2024-09-16T05:20:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f538c09cad0 con 0x7f538c0a15d0 2024-09-16T05:20:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f538c13fa80 con 0x7f538c0a19e0 2024-09-16T05:20:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f53927ff640 10 monclient(hunting): authenticate will time out at 1726.387179s 2024-09-16T05:20:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f538c09b6c0 0x7f538c0947c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f538c09b6c0 0x7f538c0947c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53562/0 (socket says 172.21.15.114:53562) 2024-09-16T05:20:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 1 -- 172.21.15.114:0/4027936289 learned_addr learned my addr 172.21.15.114:0/4027936289 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 10 monclient(hunting): get_auth_request con 0x7f538c09b6c0 auth_method 0 2024-09-16T05:20:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f5390d9c640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c0a19e0 0x7f538c092040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f5390d9c640 10 monclient(hunting): get_auth_request con 0x7f538c0a19e0 auth_method 0 2024-09-16T05:20:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f5390d9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f5390d9c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f5390d9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f5390d9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f5390d9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f5390d9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538b7fe640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f538c0a15d0 0x7f538c091b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538b7fe640 10 monclient(hunting): get_auth_request con 0x7f538c0a15d0 auth_method 0 2024-09-16T05:20:41.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 10 monclient(hunting): handle_auth_done global_id 74598 payload 824 2024-09-16T05:20:41.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 1 -- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c0a19e0 msgr2=0x7f538c092040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c0a19e0 0x7f538c092040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 1 -- 172.21.15.114:0/4027936289 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f538c0a15d0 msgr2=0x7f538c091b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f538c0a15d0 0x7f538c091b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:20:41.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 1 monclient: found mon.noname-a 2024-09-16T05:20:41.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:41.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 1 -- 172.21.15.114:0/4027936289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f538c094d00 con 0x7f538c09b6c0 2024-09-16T05:20:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f5390d9c640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c0a19e0 0x7f538c092040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:20:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f538bfff640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f538c09b6c0 0x7f538c0947c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f537c00ba80 tx=0x7f537c00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f5388ff9640 10 client.?.objecter ms_handle_connect 0x7f538c09b6c0 2024-09-16T05:20:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.755+0000 7f5388ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:20:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 1 -- 172.21.15.114:0/4027936289 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c019070 con 0x7f538c09b6c0 2024-09-16T05:20:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 monclient: dump: 2024-09-16T05:20:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.776 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 monclient: _finish_auth 0 2024-09-16T05:20:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:20:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 1 -- 172.21.15.114:0/4027936289 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f537c00cc90 con 0x7f538c09b6c0 2024-09-16T05:20:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 1 -- 172.21.15.114:0/4027936289 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c014680 con 0x7f538c09b6c0 2024-09-16T05:20:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f53927ff640 5 monclient: authenticate success, global_id 74598 2024-09-16T05:20:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 monclient: dump: 2024-09-16T05:20:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.778 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f538a7fc640 4 set_mon_vals no callback set 2024-09-16T05:20:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f53927ff640 10 monclient: _renew_subs 2024-09-16T05:20:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f53927ff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f53927ff640 1 -- 172.21.15.114:0/4027936289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f538c06e0d0 con 0x7f538c09b6c0 2024-09-16T05:20:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f53927ff640 10 client.74598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:20:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f53927ff640 10 monclient: _renew_subs 2024-09-16T05:20:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f53927ff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f53927ff640 1 -- 172.21.15.114:0/4027936289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f538c0954d0 con 0x7f538c09b6c0 2024-09-16T05:20:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f53927ff640 1 librados: init done 2024-09-16T05:20:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f53927ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f53927ff640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c092580 0x7f538c095710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f53927ff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f53927ff640 1 -- 172.21.15.114:0/4027936289 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f538c069390 con 0x7f538c092580 2024-09-16T05:20:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f538b7fe640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c092580 0x7f538c095710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f538b7fe640 10 monclient: get_auth_request con 0x7f538c092580 auth_method 0 2024-09-16T05:20:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f538b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f538b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f538b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f538b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f538b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f538b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 1 -- 172.21.15.114:0/4027936289 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f537c01c070 con 0x7f538c09b6c0 2024-09-16T05:20:41.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f538b7fe640 10 monclient(hunting): handle_auth_done global_id 104126 payload 824 2024-09-16T05:20:41.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f538b7fe640 1 -- 172.21.15.114:0/4027936289 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f538c092960 con 0x7f538c092580 2024-09-16T05:20:41.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 1 --2- 172.21.15.114:0/4027936289 >> v2:172.21.15.149:6812/1185531135 conn(0x7f536c037810 0x7f536c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 1 -- 172.21.15.114:0/4027936289 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(529..529 src has 1..529) ==== 5861+0+0 (secure 0 0 0) 0x7f537c048d60 con 0x7f538c09b6c0 2024-09-16T05:20:41.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 client.74598.objecter ms_dispatch 0x7f538c000ba0 osd_map(529..529 src has 1..529) 2024-09-16T05:20:41.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 3 client.74598.objecter handle_osd_map got epochs [529,529] > 0 2024-09-16T05:20:41.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 3 client.74598.objecter handle_osd_map decoding full epoch 529 2024-09-16T05:20:41.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f538b7fe640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c092580 0x7f538c095710 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f538c093ab0 tx=0x7f537400ec00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 20 client.74598.objecter dump_active .. 0 homeless 2024-09-16T05:20:41.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 client.74598.objecter ms_handle_connect 0x7f538c092580 2024-09-16T05:20:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 client.74598.objecter resend_mon_ops 2024-09-16T05:20:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 1 -- 172.21.15.114:0/4027936289 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374019070 con 0x7f538c092580 2024-09-16T05:20:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:20:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 monclient: dump: 2024-09-16T05:20:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.783 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.783 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.783 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.783 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5390d9c640 1 --2- 172.21.15.114:0/4027936289 >> v2:172.21.15.149:6812/1185531135 conn(0x7f536c037810 0x7f536c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5390d9c640 10 monclient: get_auth_request con 0x7f536c037810 auth_method 0 2024-09-16T05:20:41.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 1 -- 172.21.15.114:0/4027936289 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5374004020 con 0x7f538c092580 2024-09-16T05:20:41.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:41.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:20:41.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 1 -- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c092580 msgr2=0x7f538c095710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c092580 0x7f538c095710 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f538c093ab0 tx=0x7f537400ec00 comp rx=0 tx=0).stop 2024-09-16T05:20:41.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 1 -- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c092580 msgr2=0x7f538c095710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:20:41.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c092580 0x7f538c095710 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5388ff9640 10 client.74598.objecter ms_handle_connect 0x7f536c037810 2024-09-16T05:20:41.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.759+0000 7f5390d9c640 1 --2- 172.21.15.114:0/4027936289 >> v2:172.21.15.149:6812/1185531135 conn(0x7f536c037810 0x7f536c039cd0 secure :-1 s=READY pgs=3569 cs=0 l=1 rev1=1 crypto rx=0x7f53840ad000 tx=0x7f5384008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.763+0000 7f53927ff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:20:41.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.763+0000 7f53927ff640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c08e010 0x7f538c08e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.763+0000 7f53927ff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.763+0000 7f53927ff640 1 -- 172.21.15.114:0/4027936289 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f538c0a0ec0 con 0x7f538c08e010 2024-09-16T05:20:41.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f538b7fe640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c08e010 0x7f538c08e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f538b7fe640 10 monclient: get_auth_request con 0x7f538c08e010 auth_method 0 2024-09-16T05:20:41.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f538b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f538b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f538b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f538b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f538b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f538b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f538b7fe640 10 monclient(hunting): handle_auth_done global_id 104129 payload 824 2024-09-16T05:20:41.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f538b7fe640 1 -- 172.21.15.114:0/4027936289 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f538c094080 con 0x7f538c08e010 2024-09-16T05:20:41.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f538b7fe640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c08e010 0x7f538c08e420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5374018000 tx=0x7f53740046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f5388ff9640 10 client.74598.objecter ms_handle_connect 0x7f538c08e010 2024-09-16T05:20:41.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f5388ff9640 10 client.74598.objecter resend_mon_ops 2024-09-16T05:20:41.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f5388ff9640 1 -- 172.21.15.114:0/4027936289 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537400a980 con 0x7f538c08e010 2024-09-16T05:20:41.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f5388ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f5388ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:20:41.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f5388ff9640 10 monclient: dump: 2024-09-16T05:20:41.787 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.787 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.787 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.787 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.787 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.787 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.787 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.787 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.787 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.787 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f5388ff9640 1 -- 172.21.15.114:0/4027936289 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f537400a980 con 0x7f538c08e010 2024-09-16T05:20:41.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f5388ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:20:41.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f5388ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:20:41.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f5388ff9640 1 -- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c08e010 msgr2=0x7f538c08e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f5388ff9640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c08e010 0x7f538c08e420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5374018000 tx=0x7f53740046d0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f5388ff9640 1 -- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c08e010 msgr2=0x7f538c08e420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:20:41.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f5388ff9640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f538c08e010 0x7f538c08e420 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5374018000 tx=0x7f53740046d0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.788 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:20:41.788 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:20:41.788 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:20:41.789 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:20:41.789 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 56, 2024-09-16T05:20:41.789 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 17931, 2024-09-16T05:20:41.789 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:20:41.789 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:20:41.789 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:20:41.789 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:20:41.789 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:41.789 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 7, 2024-09-16T05:20:41.789 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:20:41.789 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:20:41.789 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:20:41.789 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:41.789 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:20:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:20:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:41.790 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:20:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:20:41.791 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:20:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:20:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:20:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:20:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:20:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:20:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:20:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:20:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:20:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:20:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:20:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:20:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:20:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:20:41.792 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:20:41.793 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:41.793 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:41.793 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:41.793 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:41.793 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:41.793 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:41.793 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:41.793 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:41.793 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:41.793 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:41.793 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:41.793 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:41.793 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:20:41.794 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.794 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:20:41.794 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.794 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:20:41.794 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:20:41.794 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:41.794 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:41.794 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.794 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:41.794 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:20:41.795 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.795 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.795 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.795 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:41.795 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:20:41.795 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.795 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.795 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.795 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.795 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:20:41.795 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:20:41.795 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:41.795 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:41.795 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:41.796 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.796 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.796 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:20:41.796 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:20:41.796 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:41.796 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:41.796 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.796 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:41.796 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:20:41.796 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.796 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.796 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.796 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:41.797 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:20:41.797 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.797 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.797 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.797 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.797 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:20:41.797 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:20:41.797 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:41.797 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:41.798 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:41.798 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.798 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.798 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:20:41.798 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:20:41.798 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:41.798 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:41.798 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.798 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:41.798 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:20:41.798 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.798 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.798 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.798 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:20:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:20:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:20:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:20:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:20:41.800 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.800 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:41.800 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:41.800 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:20:41.800 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.800 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:41.800 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:20:41.800 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.800 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:41.800 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:41.800 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:20:41.800 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.800 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.800 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.801 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:20:41.801 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:20:41.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f53927ff640 10 librados: watch_flush enter 2024-09-16T05:20:41.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f53927ff640 10 librados: watch_flush exit 2024-09-16T05:20:41.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f53927ff640 20 client.74598.objecter shutdown clearing up homeless session... 2024-09-16T05:20:41.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f53927ff640 10 client.74598.objecter successfully canceled tick 2024-09-16T05:20:41.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f53927ff640 1 -- 172.21.15.114:0/4027936289 >> v2:172.21.15.149:6812/1185531135 conn(0x7f536c037810 msgr2=0x7f536c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f53927ff640 1 --2- 172.21.15.114:0/4027936289 >> v2:172.21.15.149:6812/1185531135 conn(0x7f536c037810 0x7f536c039cd0 secure :-1 s=READY pgs=3569 cs=0 l=1 rev1=1 crypto rx=0x7f53840ad000 tx=0x7f5384008040 comp rx=0 tx=0).stop 2024-09-16T05:20:41.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f53927ff640 10 monclient: shutdown 2024-09-16T05:20:41.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f53927ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:41.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f53927ff640 1 -- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f538c09b6c0 msgr2=0x7f538c0947c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f53927ff640 1 --2- 172.21.15.114:0/4027936289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f538c09b6c0 0x7f538c0947c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f537c00ba80 tx=0x7f537c00bf40 comp rx=0 tx=0).stop 2024-09-16T05:20:41.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f5390d9c640 1 -- 172.21.15.114:0/4027936289 reap_dead start 2024-09-16T05:20:41.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f5390d9c640 1 -- 172.21.15.114:0/4027936289 reap_dead start 2024-09-16T05:20:41.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f53927ff640 1 -- 172.21.15.114:0/4027936289 shutdown_connections 2024-09-16T05:20:41.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f53927ff640 1 -- 172.21.15.114:0/4027936289 >> 172.21.15.114:0/4027936289 conn(0x7f538c0a1fb0 msgr2=0x7f538c071380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:41.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f53927ff640 1 -- 172.21.15.114:0/4027936289 shutdown_connections 2024-09-16T05:20:41.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f53927ff640 1 -- 172.21.15.114:0/4027936289 wait complete. 2024-09-16T05:20:41.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.767+0000 7f53927ff640 1 librados: shutdown 2024-09-16T05:20:41.804 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:20:41.804 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:20:41.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:20:41.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 10 monclient: get_monmap_and_config 2024-09-16T05:20:41.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 10 monclient: build_initial_monmap 2024-09-16T05:20:41.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 10 monclient: monmap: 2024-09-16T05:20:41.917 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:41.917 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.917 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:20:41.921949+0000 2024-09-16T05:20:41.917 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:20:41.921949+0000 2024-09-16T05:20:41.917 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:41.917 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.917 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:20:41.917 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:20:41.917 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:20:41.917 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:20:41.917 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:20:41.918 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 1 Processor -- start 2024-09-16T05:20:41.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 1 -- start start 2024-09-16T05:20:41.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 10 monclient: init 2024-09-16T05:20:41.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:41.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-16T05:20:41.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb6a809b6c0 0x7fb6a80a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 10 monclient(hunting): picked mon.noname-a con 0x7fb6a809b6c0 addr v2:172.21.15.114:3300/0 2024-09-16T05:20:41.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6a80a1400 addr v1:172.21.15.149:6789/0 2024-09-16T05:20:41.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb6a80a1810 0x7fb6a813cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 10 monclient(hunting): picked mon.noname-c con 0x7fb6a80a1810 addr v2:172.21.15.149:3300/0 2024-09-16T05:20:41.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6a809d4e0 con 0x7fb6a80a1400 2024-09-16T05:20:41.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb6a809e390 con 0x7fb6a809b6c0 2024-09-16T05:20:41.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fb6a809ece0 con 0x7fb6a80a1810 2024-09-16T05:20:41.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:41.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6acca7640 10 monclient(hunting): authenticate will time out at 1726.551046s 2024-09-16T05:20:41.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb6a809b6c0 0x7fb6a80a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb6a809b6c0 0x7fb6a80a0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53574/0 (socket says 172.21.15.114:53574) 2024-09-16T05:20:41.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 1 -- 172.21.15.114:0/3569483764 learned_addr learned my addr 172.21.15.114:0/3569483764 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:41.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 10 monclient(hunting): get_auth_request con 0x7fb6a809b6c0 auth_method 0 2024-09-16T05:20:41.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a6d9d640 1 --2- 172.21.15.114:0/3569483764 >> v2:172.21.15.149:3300/0 conn(0x7fb6a80a1810 0x7fb6a813cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 10 monclient(hunting): handle_auth_done global_id 74601 payload 274 2024-09-16T05:20:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 1 -- 172.21.15.114:0/3569483764 >> v2:172.21.15.149:3300/0 conn(0x7fb6a80a1810 msgr2=0x7fb6a813cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 1 --2- 172.21.15.114:0/3569483764 >> v2:172.21.15.149:3300/0 conn(0x7fb6a80a1810 0x7fb6a813cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 1 -- 172.21.15.114:0/3569483764 >> v1:172.21.15.149:6789/0 conn(0x7fb6a80a1400 legacy=0x7fb6a8139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:20:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 10 monclient: _finish_hunting 0 2024-09-16T05:20:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 1 monclient: found mon.noname-a 2024-09-16T05:20:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a659c640 1 -- 172.21.15.114:0/3569483764 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a809f430 con 0x7fb6a809b6c0 2024-09-16T05:20:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a6d9d640 10 monclient: get_auth_request con 0x7fb6a80a1810 auth_method 0 2024-09-16T05:20:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.919+0000 7fb6a6d9d640 1 --2- 172.21.15.114:0/3569483764 >> v2:172.21.15.149:3300/0 conn(0x7fb6a80a1810 0x7fb6a813cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:20:41.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6a659c640 1 --2- 172.21.15.114:0/3569483764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a809b6c0 0x7fb6a80a0ec0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80a2f70 tx=0x7fb6a001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96bf15c15b60b49b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6a559a640 1 -- 172.21.15.114:0/3569483764 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a00a5600 con 0x7fb6a809b6c0 2024-09-16T05:20:41.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6a559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:41.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6a559a640 10 monclient: dump: 2024-09-16T05:20:41.929 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.929 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.929 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.929 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.929 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.929 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.930 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6a559a640 10 monclient: _finish_auth 0 2024-09-16T05:20:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6a559a640 10 monclient: _check_auth_tickets 2024-09-16T05:20:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6a559a640 1 -- 172.21.15.114:0/3569483764 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6a00a57a0 con 0x7fb6a809b6c0 2024-09-16T05:20:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 5 monclient: authenticate success, global_id 74601 2024-09-16T05:20:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:20:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6a559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 10 monclient: get_monmap_and_config success 2024-09-16T05:20:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6a559a640 1 -- 172.21.15.114:0/3569483764 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a00a5b70 con 0x7fb6a809b6c0 2024-09-16T05:20:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 4 set_mon_vals no callback set 2024-09-16T05:20:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6a559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6a559a640 10 monclient: dump: 2024-09-16T05:20:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.931 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 10 monclient: shutdown 2024-09-16T05:20:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 1 -- 172.21.15.114:0/3569483764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a809b6c0 msgr2=0x7fb6a80a0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 1 --2- 172.21.15.114:0/3569483764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a809b6c0 0x7fb6a80a0ec0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80a2f70 tx=0x7fb6a001c850 comp rx=0 tx=0).stop 2024-09-16T05:20:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 1 -- 172.21.15.114:0/3569483764 shutdown_connections 2024-09-16T05:20:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 1 --2- 172.21.15.114:0/3569483764 >> v2:172.21.15.149:3300/0 conn(0x7fb6a80a1810 0x7fb6a813cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 1 --2- 172.21.15.114:0/3569483764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a809b6c0 0x7fb6a80a0ec0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 1 -- 172.21.15.114:0/3569483764 >> 172.21.15.114:0/3569483764 conn(0x7fb6a80a1fb0 msgr2=0x7fb6a80a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 1 -- 172.21.15.114:0/3569483764 shutdown_connections 2024-09-16T05:20:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 1 -- 172.21.15.114:0/3569483764 wait complete. 2024-09-16T05:20:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 10 monclient: build_initial_monmap 2024-09-16T05:20:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 10 monclient: monmap: 2024-09-16T05:20:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:20:41.932 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:20:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:20:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:20:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:20:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:20:41.933 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 1 librados: starting msgr at 2024-09-16T05:20:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 1 librados: starting objecter 2024-09-16T05:20:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 1 Processor -- start 2024-09-16T05:20:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 1 -- start start 2024-09-16T05:20:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 1 librados: setting wanted keys 2024-09-16T05:20:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 1 librados: calling monclient init 2024-09-16T05:20:41.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.923+0000 7fb6acca7640 10 monclient: init 2024-09-16T05:20:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:20:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a809b6c0 0x7fb6a813ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 monclient(hunting): picked mon.noname-b con 0x7fb6a809b6c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:20:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a80a1400 0x7fb6a80994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 monclient(hunting): picked mon.noname-a con 0x7fb6a80a1400 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:20:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6a80a1810 0x7fb6a80999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 monclient(hunting): picked mon.noname-c con 0x7fb6a80a1810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a809cad0 con 0x7fb6a80a1400 2024-09-16T05:20:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a809dd40 con 0x7fb6a809b6c0 2024-09-16T05:20:41.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a813fad0 con 0x7fb6a80a1810 2024-09-16T05:20:41.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:41.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 monclient(hunting): authenticate will time out at 1726.556564s 2024-09-16T05:20:41.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a6d9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6a80a1810 0x7fb6a80999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a80a1400 0x7fb6a80994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a6d9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6a80a1810 0x7fb6a80999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:35664/0 (socket says 172.21.15.114:35664) 2024-09-16T05:20:41.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a6d9d640 1 -- 172.21.15.114:0/4270669467 learned_addr learned my addr 172.21.15.114:0/4270669467 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:41.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a6d9d640 10 monclient(hunting): get_auth_request con 0x7fb6a80a1810 auth_method 0 2024-09-16T05:20:41.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a80a1400 0x7fb6a80994b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53590/0 (socket says 172.21.15.114:53590) 2024-09-16T05:20:41.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 10 monclient(hunting): get_auth_request con 0x7fb6a80a1400 auth_method 0 2024-09-16T05:20:41.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a809b6c0 0x7fb6a813ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 10 monclient(hunting): get_auth_request con 0x7fb6a809b6c0 auth_method 0 2024-09-16T05:20:41.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 10 monclient(hunting): handle_auth_done global_id 74604 payload 824 2024-09-16T05:20:41.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 1 -- 172.21.15.114:0/4270669467 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6a80a1810 msgr2=0x7fb6a80999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6a80a1810 0x7fb6a80999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 1 -- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a809b6c0 msgr2=0x7fb6a813ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a809b6c0 0x7fb6a813ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:20:41.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 1 monclient: found mon.noname-a 2024-09-16T05:20:41.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:20:41.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:20:41.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 1 -- 172.21.15.114:0/4270669467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a8097cd0 con 0x7fb6a80a1400 2024-09-16T05:20:41.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a6d9d640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6a80a1810 0x7fb6a80999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:20:41.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a5d9b640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a80a1400 0x7fb6a80994b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00c410 tx=0x7fb69c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 client.?.objecter ms_handle_connect 0x7fb6a80a1400 2024-09-16T05:20:41.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:20:41.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 1 -- 172.21.15.114:0/4270669467 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c011790 con 0x7fb6a80a1400 2024-09-16T05:20:41.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:20:41.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 monclient: dump: 2024-09-16T05:20:41.939 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.939 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.939 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.939 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.939 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.939 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.939 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.939 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.940 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.940 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 monclient: _finish_auth 0 2024-09-16T05:20:41.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:20:41.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:20:41.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 5 monclient: authenticate success, global_id 74604 2024-09-16T05:20:41.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 1 -- 172.21.15.114:0/4270669467 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb69c017070 con 0x7fb6a80a1400 2024-09-16T05:20:41.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 monclient: _renew_subs 2024-09-16T05:20:41.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:41.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 1 -- 172.21.15.114:0/4270669467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a806e1e0 con 0x7fb6a80a1400 2024-09-16T05:20:41.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:20:41.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 1 -- 172.21.15.114:0/4270669467 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c011b70 con 0x7fb6a80a1400 2024-09-16T05:20:41.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:20:41.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 monclient: dump: 2024-09-16T05:20:41.941 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.941 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.941 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.941 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.941 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.941 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.941 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.941 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.941 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.941 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 client.74604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:20:41.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 monclient: _renew_subs 2024-09-16T05:20:41.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a4d99640 4 set_mon_vals no callback set 2024-09-16T05:20:41.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:20:41.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 1 -- 172.21.15.114:0/4270669467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a809b0e0 con 0x7fb6a80a1400 2024-09-16T05:20:41.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 1 librados: init done 2024-09-16T05:20:41.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:41.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a8099f30 0x7fb6a808e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6acca7640 1 -- 172.21.15.114:0/4270669467 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a8069390 con 0x7fb6a8099f30 2024-09-16T05:20:41.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 1 -- 172.21.15.114:0/4270669467 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb69c011d10 con 0x7fb6a80a1400 2024-09-16T05:20:41.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a8099f30 0x7fb6a808e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 1 --2- 172.21.15.114:0/4270669467 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb688037810 0x7fb688039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 10 monclient: get_auth_request con 0x7fb6a8099f30 auth_method 0 2024-09-16T05:20:41.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 1 -- 172.21.15.114:0/4270669467 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(529..529 src has 1..529) ==== 5861+0+0 (secure 0 0 0) 0x7fb69c048940 con 0x7fb6a80a1400 2024-09-16T05:20:41.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 client.74604.objecter ms_dispatch 0x7fb6a8000ba0 osd_map(529..529 src has 1..529) 2024-09-16T05:20:41.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 3 client.74604.objecter handle_osd_map got epochs [529,529] > 0 2024-09-16T05:20:41.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 3 client.74604.objecter handle_osd_map decoding full epoch 529 2024-09-16T05:20:41.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 20 client.74604.objecter dump_active .. 0 homeless 2024-09-16T05:20:41.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a6d9d640 1 --2- 172.21.15.114:0/4270669467 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb688037810 0x7fb688039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a6d9d640 10 monclient: get_auth_request con 0x7fb688037810 auth_method 0 2024-09-16T05:20:41.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 10 monclient(hunting): handle_auth_done global_id 64672 payload 824 2024-09-16T05:20:41.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 1 -- 172.21.15.114:0/4270669467 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6a809a310 con 0x7fb6a8099f30 2024-09-16T05:20:41.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 client.74604.objecter ms_handle_connect 0x7fb6a8099f30 2024-09-16T05:20:41.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 client.74604.objecter resend_mon_ops 2024-09-16T05:20:41.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a659c640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a8099f30 0x7fb6a808e010 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb6a809c630 tx=0x7fb6a00195f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 client.74604.objecter ms_handle_connect 0x7fb688037810 2024-09-16T05:20:41.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb6a6d9d640 1 --2- 172.21.15.114:0/4270669467 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb688037810 0x7fb688039cd0 secure :-1 s=READY pgs=3570 cs=0 l=1 rev1=1 crypto rx=0x7fb6a806dfb0 tx=0x7fb698008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 1 -- 172.21.15.114:0/4270669467 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0009510 con 0x7fb6a8099f30 2024-09-16T05:20:41.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:20:41.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.927+0000 7fb696ffd640 10 monclient: dump: 2024-09-16T05:20:41.944 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.944 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.944 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.944 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.944 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.945 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.945 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.945 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.945 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.945 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.931+0000 7fb696ffd640 1 -- 172.21.15.114:0/4270669467 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb6a00c06c0 con 0x7fb6a8099f30 2024-09-16T05:20:41.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.931+0000 7fb696ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:20:41.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.931+0000 7fb696ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:20:41.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.931+0000 7fb696ffd640 1 -- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a8099f30 msgr2=0x7fb6a808e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.931+0000 7fb696ffd640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a8099f30 0x7fb6a808e010 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb6a809c630 tx=0x7fb6a00195f0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.931+0000 7fb696ffd640 1 -- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a8099f30 msgr2=0x7fb6a808e010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:20:41.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.931+0000 7fb696ffd640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a8099f30 0x7fb6a808e010 secure :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb6a809c630 tx=0x7fb6a00195f0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.935+0000 7fb6acca7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:20:41.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.935+0000 7fb6acca7640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a808e550 0x7fb6a808e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:41.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.935+0000 7fb6acca7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:41.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.935+0000 7fb6acca7640 1 -- 172.21.15.114:0/4270669467 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a80a0ec0 con 0x7fb6a808e550 2024-09-16T05:20:41.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.935+0000 7fb6a659c640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a808e550 0x7fb6a808e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:41.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.935+0000 7fb6a659c640 10 monclient: get_auth_request con 0x7fb6a808e550 auth_method 0 2024-09-16T05:20:41.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.935+0000 7fb6a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:41.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.935+0000 7fb6a659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:41.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.935+0000 7fb6a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:41.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.935+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:20:41.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.935+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:20:41.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.935+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:20:41.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.935+0000 7fb6a659c640 10 monclient(hunting): handle_auth_done global_id 64675 payload 824 2024-09-16T05:20:41.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.935+0000 7fb6a659c640 1 -- 172.21.15.114:0/4270669467 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb6a80686f0 con 0x7fb6a808e550 2024-09-16T05:20:41.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6a659c640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a808e550 0x7fb6a808e960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb6a001af00 tx=0x7fb6a00a7000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:20:41.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb696ffd640 10 client.74604.objecter ms_handle_connect 0x7fb6a808e550 2024-09-16T05:20:41.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb696ffd640 10 client.74604.objecter resend_mon_ops 2024-09-16T05:20:41.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb696ffd640 1 -- 172.21.15.114:0/4270669467 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a00c0a80 con 0x7fb6a808e550 2024-09-16T05:20:41.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb696ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:20:41.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb696ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:20:41.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb696ffd640 10 monclient: dump: 2024-09-16T05:20:41.949 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:20:41.949 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:41.949 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.949 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:20:41.949 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:20:41.949 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:41.949 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:20:41.949 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:20:41.949 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:20:41.949 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:41.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb696ffd640 1 -- 172.21.15.114:0/4270669467 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fb6a00c0a80 con 0x7fb6a808e550 2024-09-16T05:20:41.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb696ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:20:41.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb696ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:20:41.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb696ffd640 1 -- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a808e550 msgr2=0x7fb6a808e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb696ffd640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a808e550 0x7fb6a808e960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb6a001af00 tx=0x7fb6a00a7000 comp rx=0 tx=0).stop 2024-09-16T05:20:41.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb696ffd640 1 -- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a808e550 msgr2=0x7fb6a808e960 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:20:41.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb696ffd640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6a808e550 0x7fb6a808e960 secure :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb6a001af00 tx=0x7fb6a00a7000 comp rx=0 tx=0).stop 2024-09-16T05:20:41.950 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:20:41.950 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:20:41.950 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:20:41.950 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:20:41.950 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 56, 2024-09-16T05:20:41.950 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 526073, 2024-09-16T05:20:41.951 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:20:41.951 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:20:41.951 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:20:41.951 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:20:41.951 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:41.951 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 7, 2024-09-16T05:20:41.951 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:20:41.951 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:20:41.951 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:20:41.951 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:41.951 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:41.951 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:41.952 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:41.952 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:41.952 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:41.952 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:41.952 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:41.952 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:41.952 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:41.952 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:41.952 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:41.952 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:20:41.952 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:20:41.953 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:41.953 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:41.953 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:41.953 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:41.953 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:41.953 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:41.953 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:41.953 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:41.953 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:41.953 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:41.953 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:41.953 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.953 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.953 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:20:41.954 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:20:41.954 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:20:41.954 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:20:41.954 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:20:41.954 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:20:41.954 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:20:41.954 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:20:41.954 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:20:41.954 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:20:41.954 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:20:41.954 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:20:41.954 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:20:41.954 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:20:41.954 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:20:41.955 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:20:41.955 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:20:41.955 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:20:41.955 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:20:41.955 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:20:41.955 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:20:41.955 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:20:41.955 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:20:41.955 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:20:41.955 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:20:41.955 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:20:41.955 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:20:41.955 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:20:41.955 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:41.956 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:20:41.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.956 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:20:41.956 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.956 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:20:41.956 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:20:41.956 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:41.956 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:41.956 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.956 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:41.956 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:20:41.956 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.957 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.957 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:41.957 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:20:41.957 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.957 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.957 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.957 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:20:41.957 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:20:41.957 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:41.958 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:41.958 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:41.958 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.958 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:20:41.958 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:20:41.958 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:41.958 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:41.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.958 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:41.958 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:20:41.958 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.958 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.959 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:41.959 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:20:41.959 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.959 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.959 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.959 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:20:41.959 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:20:41.959 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:41.959 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:41.959 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:41.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.959 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.959 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:20:41.960 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:20:41.960 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:20:41.960 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:20:41.960 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.960 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:20:41.960 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:20:41.960 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.960 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.960 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.960 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:20:41.960 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:20:41.960 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:20:41.960 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.960 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.961 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.961 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:20:41.961 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:20:41.961 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:20:41.961 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:20:41.961 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:20:41.961 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.961 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.961 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.961 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:20:41.961 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:20:41.961 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.961 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:41.961 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:41.962 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:20:41.962 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.962 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:41.962 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:20:41.962 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.962 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:41.962 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:41.962 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:20:41.962 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.962 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:41.962 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:20:41.962 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.962 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:41.962 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:41.963 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:20:41.963 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.963 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:20:41.963 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:20:41.963 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:20:41.963 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:20:41.963 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:20:41.963 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:20:41.963 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:20:41.963 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:20:41.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:20:41.963 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:20:41.963 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:20:41.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 10 librados: watch_flush enter 2024-09-16T05:20:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 10 librados: watch_flush exit 2024-09-16T05:20:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 20 client.74604.objecter shutdown clearing up homeless session... 2024-09-16T05:20:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 10 client.74604.objecter successfully canceled tick 2024-09-16T05:20:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 1 -- 172.21.15.114:0/4270669467 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb688037810 msgr2=0x7fb688039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 1 --2- 172.21.15.114:0/4270669467 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb688037810 0x7fb688039cd0 secure :-1 s=READY pgs=3570 cs=0 l=1 rev1=1 crypto rx=0x7fb6a806dfb0 tx=0x7fb698008040 comp rx=0 tx=0).stop 2024-09-16T05:20:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 10 monclient: shutdown 2024-09-16T05:20:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:20:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 1 -- 172.21.15.114:0/4270669467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a80a1400 msgr2=0x7fb6a80994b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:20:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a80a1400 0x7fb6a80994b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00c410 tx=0x7fb69c00c8d0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6a6d9d640 1 -- 172.21.15.114:0/4270669467 reap_dead start 2024-09-16T05:20:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 1 -- 172.21.15.114:0/4270669467 shutdown_connections 2024-09-16T05:20:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 1 --2- 172.21.15.114:0/4270669467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a80a1400 0x7fb6a80994b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:20:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 1 -- 172.21.15.114:0/4270669467 >> 172.21.15.114:0/4270669467 conn(0x7fb6a80a1fb0 msgr2=0x7fb6a8071460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:20:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 1 -- 172.21.15.114:0/4270669467 shutdown_connections 2024-09-16T05:20:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 1 -- 172.21.15.114:0/4270669467 wait complete. 2024-09-16T05:20:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:41.939+0000 7fb6acca7640 1 librados: shutdown 2024-09-16T05:20:41.967 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:20:41.967 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T05:20:41.967 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-16T05:20:41.967 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T05:20:41.967 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:20:41.968 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T05:20:41.968 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:20:41.996 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T05:20:41.996 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T05:20:41.996 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T05:20:41.996 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T05:20:56.997 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T05:20:56.998 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T05:20:56.998 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:20:56.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:20:57.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 10 monclient: get_monmap_and_config 2024-09-16T05:20:57.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 10 monclient: build_initial_monmap 2024-09-16T05:20:57.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 10 monclient: monmap: 2024-09-16T05:20:57.125 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:20:57.125 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:20:57.125 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:20:57.130314+0000 2024-09-16T05:20:57.125 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:20:57.130314+0000 2024-09-16T05:20:57.125 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:20:57.125 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:20:57.125 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:20:57.126 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:20:57.126 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:20:57.126 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:20:57.126 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:20:57.126 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:20:57.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 1 Processor -- start 2024-09-16T05:20:57.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 1 -- start start 2024-09-16T05:20:57.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 10 monclient: init 2024-09-16T05:20:57.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 10 monclient: _reopen_session rank -1 2024-09-16T05:20:57.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-16T05:20:57.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4bcc09b6c0 0x7f4bcc0a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:57.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 10 monclient(hunting): picked mon.noname-c con 0x7f4bcc09b6c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:20:57.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 0x7f4bcc0a1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:20:57.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 10 monclient(hunting): picked mon.noname-b con 0x7f4bcc0a1400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:20:57.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4bcc139640 addr v1:172.21.15.149:6789/0 2024-09-16T05:20:57.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bcc09d4e0 con 0x7f4bcc139640 2024-09-16T05:20:57.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:57.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f4bcc09e390 con 0x7f4bcc09b6c0 2024-09-16T05:20:57.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:20:57.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4bcc0a1d50 con 0x7f4bcc0a1400 2024-09-16T05:20:57.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 10 monclient(hunting): _renew_subs 2024-09-16T05:20:57.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd47b4640 10 monclient(hunting): authenticate will time out at 1741.759485s 2024-09-16T05:20:57.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd1d4f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 0x7f4bcc0a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:57.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd2d51640 1 -- >> v1:172.21.15.149:6789/0 conn(0x7f4bcc139640 legacy=0x7f4bcc13ba30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:20:57.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd2550640 1 -- >> v2:172.21.15.149:3300/0 conn(0x7f4bcc09b6c0 msgr2=0x7f4bcc0a0ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:20:57.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd2550640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4bcc09b6c0 0x7f4bcc0a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:20:57.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd1d4f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 0x7f4bcc0a1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:42314/0 (socket says 172.21.15.114:42314) 2024-09-16T05:20:57.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd1d4f640 1 -- 172.21.15.114:0/839948169 learned_addr learned my addr 172.21.15.114:0/839948169 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:20:57.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd1d4f640 10 monclient(hunting): get_auth_request con 0x7f4bcc0a1400 auth_method 0 2024-09-16T05:20:57.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd1d4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:57.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd1d4f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:57.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd1d4f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:20:57.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd1d4f640 1 -- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 msgr2=0x7f4bcc0a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:20:57.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd1d4f640 1 -- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 msgr2=0x7f4bcc0a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:20:57.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd1d4f640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 0x7f4bcc0a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:57.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.127+0000 7f4bd1d4f640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 0x7f4bcc0a1810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:20:57.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.331+0000 7f4bd2550640 1 -- 172.21.15.114:0/839948169 >> v2:172.21.15.149:3300/0 conn(0x7f4bcc09b6c0 msgr2=0x7f4bcc0a0ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:20:57.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.331+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.149:3300/0 conn(0x7f4bcc09b6c0 0x7f4bcc0a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:20:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.331+0000 7f4bd1d4f640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 0x7f4bcc0a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.331+0000 7f4bd2d51640 1 -- 172.21.15.114:0/839948169 >> v1:172.21.15.149:6789/0 conn(0x7f4bcc139640 legacy=0x7f4bcc13ba30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:20:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.331+0000 7f4bd1d4f640 10 monclient(hunting): get_auth_request con 0x7f4bcc0a1400 auth_method 0 2024-09-16T05:20:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.331+0000 7f4bd1d4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.331+0000 7f4bd1d4f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.331+0000 7f4bd1d4f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.331+0000 7f4bd1d4f640 1 -- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 msgr2=0x7f4bcc0a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:20:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.331+0000 7f4bd1d4f640 1 -- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 msgr2=0x7f4bcc0a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:20:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.331+0000 7f4bd1d4f640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 0x7f4bcc0a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.331+0000 7f4bd1d4f640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 0x7f4bcc0a1810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:20:57.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.735+0000 7f4bd2d51640 1 -- 172.21.15.114:0/839948169 >> v1:172.21.15.149:6789/0 conn(0x7f4bcc139640 legacy=0x7f4bcc13ba30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:20:57.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.735+0000 7f4bd2550640 1 -- 172.21.15.114:0/839948169 >> v2:172.21.15.149:3300/0 conn(0x7f4bcc09b6c0 msgr2=0x7f4bcc0a0ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:20:57.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.735+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.149:3300/0 conn(0x7f4bcc09b6c0 0x7f4bcc0a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:20:57.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.735+0000 7f4bd1d4f640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 0x7f4bcc0a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:57.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.735+0000 7f4bd1d4f640 10 monclient(hunting): get_auth_request con 0x7f4bcc0a1400 auth_method 0 2024-09-16T05:20:57.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.735+0000 7f4bd1d4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:57.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.735+0000 7f4bd1d4f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:57.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.735+0000 7f4bd1d4f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:57.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.735+0000 7f4bd1d4f640 1 -- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 msgr2=0x7f4bcc0a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:20:57.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.735+0000 7f4bd1d4f640 1 -- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 msgr2=0x7f4bcc0a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:20:57.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.735+0000 7f4bd1d4f640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 0x7f4bcc0a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:57.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:57.735+0000 7f4bd1d4f640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 0x7f4bcc0a1810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:20:58.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:58.539+0000 7f4bd2d51640 1 -- 172.21.15.114:0/839948169 >> v1:172.21.15.149:6789/0 conn(0x7f4bcc139640 legacy=0x7f4bcc13ba30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:20:58.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:58.539+0000 7f4bd2550640 1 -- 172.21.15.114:0/839948169 >> v2:172.21.15.149:3300/0 conn(0x7f4bcc09b6c0 msgr2=0x7f4bcc0a0ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:20:58.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:58.539+0000 7f4bd1d4f640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 0x7f4bcc0a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:20:58.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:58.539+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.149:3300/0 conn(0x7f4bcc09b6c0 0x7f4bcc0a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:20:58.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:58.539+0000 7f4bd1d4f640 10 monclient(hunting): get_auth_request con 0x7f4bcc0a1400 auth_method 0 2024-09-16T05:20:58.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:58.539+0000 7f4bd1d4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:20:58.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:58.539+0000 7f4bd1d4f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:20:58.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:58.539+0000 7f4bd1d4f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:20:58.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:58.539+0000 7f4bd1d4f640 1 -- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 msgr2=0x7f4bcc0a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:20:58.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:58.539+0000 7f4bd1d4f640 1 -- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 msgr2=0x7f4bcc0a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:20:58.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:58.539+0000 7f4bd1d4f640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 0x7f4bcc0a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:20:58.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:20:58.539+0000 7f4bd1d4f640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 0x7f4bcc0a1810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:21:00.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 10 monclient(hunting): tick 2024-09-16T05:21:00.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:21:00.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 1 monclient(hunting): continuing hunt 2024-09-16T05:21:00.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:21:00.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 1 -- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 msgr2=0x7f4bcc0a1810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:21:00.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bcc0a1400 0x7f4bcc0a1810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:00.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 1 -- 172.21.15.114:0/839948169 >> v2:172.21.15.149:3300/0 conn(0x7f4bcc09b6c0 msgr2=0x7f4bcc0a0ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:21:00.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.149:3300/0 conn(0x7f4bcc09b6c0 0x7f4bcc0a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:00.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 1 -- 172.21.15.114:0/839948169 >> v1:172.21.15.149:6789/0 conn(0x7f4bcc139640 legacy=0x7f4bcc13ba30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:21:00.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 10 monclient: _add_conns ranks=[3,0] 2024-09-16T05:21:00.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 0x7f4bb0005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:00.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f4bb00036b0 addr v2:172.21.15.114:3300/0 2024-09-16T05:21:00.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bb0006220 addr v1:172.21.15.114:6789/0 2024-09-16T05:21:00.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 1 -- 172.21.15.114:0/839948169 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bcc09f430 con 0x7f4bb0006220 2024-09-16T05:21:00.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:00.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 1 -- 172.21.15.114:0/839948169 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f4bcc09e390 con 0x7f4bb00036b0 2024-09-16T05:21:00.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bc3fff640 10 monclient(hunting): _renew_subs 2024-09-16T05:21:00.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 0x7f4bb0005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:00.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bd2550640 10 monclient(hunting): get_auth_request con 0x7f4bb00036b0 auth_method 0 2024-09-16T05:21:00.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bd2550640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:00.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bd2550640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:00.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bd2550640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:21:00.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bd2550640 1 -- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 msgr2=0x7f4bb0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:21:00.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bd2550640 1 -- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 msgr2=0x7f4bb0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:21:00.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 0x7f4bb0005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:21:00.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.131+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 0x7f4bb0005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:21:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.335+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 0x7f4bb0005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.335+0000 7f4bd2550640 10 monclient(hunting): get_auth_request con 0x7f4bb00036b0 auth_method 0 2024-09-16T05:21:00.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.335+0000 7f4bd2550640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.335+0000 7f4bd2550640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.335+0000 7f4bd2550640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:21:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.335+0000 7f4bd2550640 1 -- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 msgr2=0x7f4bb0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:21:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.335+0000 7f4bd2550640 1 -- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 msgr2=0x7f4bb0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:21:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.335+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 0x7f4bb0005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:21:00.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.335+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 0x7f4bb0005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:21:00.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.739+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 0x7f4bb0005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:00.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.739+0000 7f4bd2550640 10 monclient(hunting): get_auth_request con 0x7f4bb00036b0 auth_method 0 2024-09-16T05:21:00.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.739+0000 7f4bd2550640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:00.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.739+0000 7f4bd2550640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:00.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.739+0000 7f4bd2550640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:21:00.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.739+0000 7f4bd2550640 1 -- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 msgr2=0x7f4bb0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:21:00.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.739+0000 7f4bd2550640 1 -- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 msgr2=0x7f4bb0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:21:00.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.739+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 0x7f4bb0005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:21:00.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:00.739+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 0x7f4bb0005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:21:01.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:01.543+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 0x7f4bb0005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:01.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:01.543+0000 7f4bd2550640 10 monclient(hunting): get_auth_request con 0x7f4bb00036b0 auth_method 0 2024-09-16T05:21:01.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:01.543+0000 7f4bd2550640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:01.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:01.543+0000 7f4bd2550640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:01.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:01.543+0000 7f4bd2550640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:21:01.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:01.543+0000 7f4bd2550640 1 -- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 msgr2=0x7f4bb0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:21:01.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:01.543+0000 7f4bd2550640 1 -- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 msgr2=0x7f4bb0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:21:01.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:01.543+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 0x7f4bb0005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:21:01.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:01.543+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 0x7f4bb0005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:21:02.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.067+0000 7f4bd154e640 1 -- 172.21.15.114:0/839948169 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1443807757 0 0) 0x7f4bcc09f430 con 0x7f4bb0006220 2024-09-16T05:21:02.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.067+0000 7f4bd154e640 10 monclient(hunting): _negotiate 2024-09-16T05:21:02.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.067+0000 7f4bd154e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.067+0000 7f4bd154e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:21:02.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.067+0000 7f4bd154e640 10 monclient(hunting): my global_id is 74607 2024-09-16T05:21:02.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.067+0000 7f4bd154e640 1 -- 172.21.15.114:0/839948169 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bb8003530 con 0x7f4bb0006220 2024-09-16T05:21:02.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 1 -- 172.21.15.114:0/839948169 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2015426214 0 0) 0x7f4bb8003530 con 0x7f4bb0006220 2024-09-16T05:21:02.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 1 -- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 msgr2=0x7f4bb0005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:21:02.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 1 --2- 172.21.15.114:0/839948169 >> v2:172.21.15.114:3300/0 conn(0x7f4bb00036b0 0x7f4bb0005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient: _finish_hunting 0 2024-09-16T05:21:02.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:21:02.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:21:02.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:21:02.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 1 -- 172.21.15.114:0/839948169 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bcc09d4e0 con 0x7f4bb0006220 2024-09-16T05:21:02.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient: _finish_auth 0 2024-09-16T05:21:02.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient: _check_auth_tickets 2024-09-16T05:21:02.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:21:02.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 1 -- 172.21.15.114:0/839948169 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f4bbc00f040 con 0x7f4bb0006220 2024-09-16T05:21:02.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd47b4640 5 monclient: authenticate success, global_id 74607 2024-09-16T05:21:02.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd47b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:21:02.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:21:02.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient: dump: 2024-09-16T05:21:02.075 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.076 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.076 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.076 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.076 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.076 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.076 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.076 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.076 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.076 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:21:02.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient: _reopen_session rank -1 2024-09-16T05:21:02.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 1 -- 172.21.15.114:0/839948169 >> v1:172.21.15.114:6789/0 conn(0x7f4bb0006220 legacy=0x7f4bb00086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2d51640 1 -- 172.21.15.114:0/839948169 reap_dead start 2024-09-16T05:21:02.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:21:02.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bb80039c0 0x7f4bb8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient(hunting): picked mon.c con 0x7f4bb80039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:21:02.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4bb8004810 0x7f4bb8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient(hunting): picked mon.b con 0x7f4bb8004810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:21:02.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb8005220 0x7f4bb800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient(hunting): picked mon.a con 0x7f4bb8005220 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:21:02.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 1 -- 172.21.15.114:0/839948169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc09e390 con 0x7f4bb8005220 2024-09-16T05:21:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 1 -- 172.21.15.114:0/839948169 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb800c150 con 0x7f4bb8004810 2024-09-16T05:21:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 1 -- 172.21.15.114:0/839948169 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4bb800c350 con 0x7f4bb80039c0 2024-09-16T05:21:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient(hunting): _renew_subs 2024-09-16T05:21:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd154e640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:21:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd47b4640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:21:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd1d4f640 1 -- 172.21.15.114:0/839948169 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4bb8004810 msgr2=0x7f4bb8004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:21:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd1d4f640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4bb8004810 0x7f4bb8004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:21:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bb80039c0 0x7f4bb8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2d51640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb8005220 0x7f4bb800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 10 monclient(hunting): get_auth_request con 0x7f4bb80039c0 auth_method 0 2024-09-16T05:21:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:21:02.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2d51640 10 monclient(hunting): get_auth_request con 0x7f4bb8005220 auth_method 0 2024-09-16T05:21:02.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2d51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:02.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2d51640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2d51640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:21:02.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:21:02.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:21:02.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:21:02.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2d51640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:21:02.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2d51640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:21:02.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2d51640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:21:02.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 10 monclient(hunting): handle_auth_done global_id 74607 payload 293 2024-09-16T05:21:02.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 1 -- 172.21.15.114:0/839948169 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4bb8004810 msgr2=0x7f4bb8004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:21:02.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4bb8004810 0x7f4bb8004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 1 -- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb8005220 msgr2=0x7f4bb800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:21:02.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb8005220 0x7f4bb800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 10 monclient: _finish_hunting 0 2024-09-16T05:21:02.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 1 monclient: found mon.c 2024-09-16T05:21:02.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:21:02.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:21:02.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 1 -- 172.21.15.114:0/839948169 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb8003530 con 0x7f4bb80039c0 2024-09-16T05:21:02.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 10 monclient: _finish_auth 0 2024-09-16T05:21:02.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 10 monclient: _check_auth_tickets 2024-09-16T05:21:02.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2550640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:21:02.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.075+0000 7f4bd2d51640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb8005220 0x7f4bb800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:21:02.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd2550640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bb80039c0 0x7f4bb8003dc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80038c0 tx=0x7f4bc801d550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c214369e71e8236 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:21:02.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd154e640 1 -- 172.21.15.114:0/839948169 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc80b3070 con 0x7f4bb80039c0 2024-09-16T05:21:02.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd154e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd154e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:21:02.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd154e640 10 monclient: dump: 2024-09-16T05:21:02.083 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.083 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.083 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.083 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.083 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.083 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.083 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.084 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.084 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.084 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd47b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:21:02.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd154e640 1 -- 172.21.15.114:0/839948169 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bc80ae480 con 0x7f4bb80039c0 2024-09-16T05:21:02.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd154e640 10 monclient: handle_config config(0 keys) 2024-09-16T05:21:02.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd47b4640 10 monclient: get_monmap_and_config success 2024-09-16T05:21:02.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd47b4640 4 set_mon_vals no callback set 2024-09-16T05:21:02.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd154e640 1 -- 172.21.15.114:0/839948169 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc80050f0 con 0x7f4bb80039c0 2024-09-16T05:21:02.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd154e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd154e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:21:02.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd154e640 10 monclient: dump: 2024-09-16T05:21:02.085 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.085 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.085 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.085 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.085 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.085 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.085 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.085 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.085 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.085 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd47b4640 10 monclient: shutdown 2024-09-16T05:21:02.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd47b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:21:02.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd47b4640 1 -- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bb80039c0 msgr2=0x7f4bb8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd47b4640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bb80039c0 0x7f4bb8003dc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80038c0 tx=0x7f4bc801d550 comp rx=0 tx=0).stop 2024-09-16T05:21:02.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd47b4640 1 -- 172.21.15.114:0/839948169 shutdown_connections 2024-09-16T05:21:02.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd47b4640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb8005220 0x7f4bb800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd47b4640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4bb8004810 0x7f4bb8004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd47b4640 1 --2- 172.21.15.114:0/839948169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bb80039c0 0x7f4bb8003dc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd47b4640 1 -- 172.21.15.114:0/839948169 >> 172.21.15.114:0/839948169 conn(0x7f4bcc0a1fb0 msgr2=0x7f4bcc0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:21:02.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd47b4640 1 -- 172.21.15.114:0/839948169 shutdown_connections 2024-09-16T05:21:02.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.083+0000 7f4bd47b4640 1 -- 172.21.15.114:0/839948169 wait complete. 2024-09-16T05:21:02.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient: build_initial_monmap 2024-09-16T05:21:02.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient: monmap: 2024-09-16T05:21:02.086 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:21:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:21:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:21:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:21:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:21:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:21:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:21:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:21:02.087 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 1 librados: starting msgr at 2024-09-16T05:21:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 1 librados: starting objecter 2024-09-16T05:21:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 1 Processor -- start 2024-09-16T05:21:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 1 -- start start 2024-09-16T05:21:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 1 librados: setting wanted keys 2024-09-16T05:21:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 1 librados: calling monclient init 2024-09-16T05:21:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient: init 2024-09-16T05:21:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient: _reopen_session rank -1 2024-09-16T05:21:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:21:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bb80039c0 0x7f4bcc0989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient(hunting): picked mon.noname-c con 0x7f4bb80039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:21:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb8004810 0x7f4bcc0912e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient(hunting): picked mon.noname-a con 0x7f4bb8004810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:21:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4bb8005220 0x7f4bcc091820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient(hunting): picked mon.noname-b con 0x7f4bb8005220 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:21:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc09da60 con 0x7f4bb8004810 2024-09-16T05:21:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc09baa0 con 0x7f4bb8005220 2024-09-16T05:21:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4bcc09dd40 con 0x7f4bb80039c0 2024-09-16T05:21:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient(hunting): _renew_subs 2024-09-16T05:21:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient(hunting): authenticate will time out at 1746.717973s 2024-09-16T05:21:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bb80039c0 0x7f4bcc0989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd1d4f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb8004810 0x7f4bcc0912e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bb80039c0 0x7f4bcc0989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:42364/0 (socket says 172.21.15.114:42364) 2024-09-16T05:21:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2d51640 1 -- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4bb8005220 msgr2=0x7f4bcc091820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:21:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 1 -- 172.21.15.114:0/2857745166 learned_addr learned my addr 172.21.15.114:0/2857745166 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:21:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd1d4f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb8004810 0x7f4bcc0912e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34792/0 (socket says 172.21.15.114:34792) 2024-09-16T05:21:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 10 monclient(hunting): get_auth_request con 0x7f4bb80039c0 auth_method 0 2024-09-16T05:21:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:21:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd1d4f640 10 monclient(hunting): get_auth_request con 0x7f4bb8004810 auth_method 0 2024-09-16T05:21:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd1d4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd1d4f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd1d4f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:21:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2d51640 1 --2- 172.21.15.114:0/2857745166 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4bb8005220 0x7f4bcc091820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:21:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:21:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:21:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:21:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd1d4f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:21:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd1d4f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:21:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd1d4f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:21:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 10 monclient(hunting): handle_auth_done global_id 104135 payload 824 2024-09-16T05:21:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 1 -- 172.21.15.114:0/2857745166 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4bb8005220 msgr2=0x7f4bcc091820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:21:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 1 --2- 172.21.15.114:0/2857745166 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4bb8005220 0x7f4bcc091820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 1 -- 172.21.15.114:0/2857745166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb8004810 msgr2=0x7f4bcc0912e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 1 --2- 172.21.15.114:0/2857745166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb8004810 0x7f4bcc0912e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 10 monclient: _finish_hunting 0 2024-09-16T05:21:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 1 monclient: found mon.noname-c 2024-09-16T05:21:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:21:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:21:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 1 -- 172.21.15.114:0/2857745166 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bcc095260 con 0x7f4bb80039c0 2024-09-16T05:21:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd1d4f640 1 --2- 172.21.15.114:0/2857745166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb8004810 0x7f4bcc0912e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:21:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd2550640 1 --2- 172.21.15.114:0/2857745166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bb80039c0 0x7f4bcc0989b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4bc801cfc0 tx=0x7f4bc801a2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:21:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bc2ffd640 10 client.?.objecter ms_handle_connect 0x7f4bb80039c0 2024-09-16T05:21:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bc2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:21:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bc2ffd640 1 -- 172.21.15.114:0/2857745166 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc80bf070 con 0x7f4bb80039c0 2024-09-16T05:21:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bc2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:21:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bc2ffd640 10 monclient: dump: 2024-09-16T05:21:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.096 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bc2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:21:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bc2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:21:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:21:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 5 monclient: authenticate success, global_id 104135 2024-09-16T05:21:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient: _renew_subs 2024-09-16T05:21:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:21:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 1 -- 172.21.15.114:0/2857745166 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bcc093290 con 0x7f4bb80039c0 2024-09-16T05:21:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bc2ffd640 1 -- 172.21.15.114:0/2857745166 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bc80b4040 con 0x7f4bb80039c0 2024-09-16T05:21:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bc2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:21:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bc2ffd640 1 -- 172.21.15.114:0/2857745166 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc80b3050 con 0x7f4bb80039c0 2024-09-16T05:21:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bc2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:21:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bc2ffd640 10 monclient: dump: 2024-09-16T05:21:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.098 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.098 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.098 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.098 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.098 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.098 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd0d4d640 4 set_mon_vals no callback set 2024-09-16T05:21:02.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 client.104135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:21:02.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient: _renew_subs 2024-09-16T05:21:02.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:21:02.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 1 -- 172.21.15.114:0/2857745166 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bcc092bb0 con 0x7f4bb80039c0 2024-09-16T05:21:02.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.086+0000 7f4bd47b4640 1 librados: init done 2024-09-16T05:21:02.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.090+0000 7f4bc2ffd640 1 -- 172.21.15.114:0/2857745166 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4bc80aed00 con 0x7f4bb80039c0 2024-09-16T05:21:02.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.090+0000 7f4bd47b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:21:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.090+0000 7f4bd47b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:21:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.090+0000 7f4bd47b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:21:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.090+0000 7f4bd47b4640 1 -- 172.21.15.114:0/2857745166 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bcc0a0ec0 con 0x7f4bb80039c0 2024-09-16T05:21:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.090+0000 7f4bc2ffd640 1 --2- 172.21.15.114:0/2857745166 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4bb0037e80 0x7f4bb003a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.090+0000 7f4bc2ffd640 1 -- 172.21.15.114:0/2857745166 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(529..529 src has 1..529) ==== 5861+0+0 (secure 0 0 0) 0x7f4bc80e3eb0 con 0x7f4bb80039c0 2024-09-16T05:21:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.090+0000 7f4bc2ffd640 10 client.104135.objecter ms_dispatch 0x7f4bcc000ba0 osd_map(529..529 src has 1..529) 2024-09-16T05:21:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.090+0000 7f4bc2ffd640 3 client.104135.objecter handle_osd_map got epochs [529,529] > 0 2024-09-16T05:21:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.090+0000 7f4bc2ffd640 3 client.104135.objecter handle_osd_map decoding full epoch 529 2024-09-16T05:21:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.090+0000 7f4bc2ffd640 20 client.104135.objecter dump_active .. 0 homeless 2024-09-16T05:21:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.090+0000 7f4bd1d4f640 1 --2- 172.21.15.114:0/2857745166 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4bb0037e80 0x7f4bb003a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.090+0000 7f4bd1d4f640 10 monclient: get_auth_request con 0x7f4bb0037e80 auth_method 0 2024-09-16T05:21:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.090+0000 7f4bc2ffd640 10 client.104135.objecter ms_handle_connect 0x7f4bb0037e80 2024-09-16T05:21:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.090+0000 7f4bd1d4f640 1 --2- 172.21.15.114:0/2857745166 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4bb0037e80 0x7f4bb003a340 secure :-1 s=READY pgs=3572 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc001ec0 tx=0x7f4bbc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:21:02.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.094+0000 7f4bc2ffd640 1 -- 172.21.15.114:0/2857745166 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f4bc80a8070 con 0x7f4bb80039c0 2024-09-16T05:21:02.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.094+0000 7f4bc2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:21:02.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.094+0000 7f4bc2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:21:02.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.318+0000 7f4bd47b4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:21:02.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.318+0000 7f4bd47b4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:21:02.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.318+0000 7f4bd47b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:21:02.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.318+0000 7f4bd47b4640 1 -- 172.21.15.114:0/2857745166 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4bcc0a4b20 con 0x7f4bb80039c0 2024-09-16T05:21:02.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bc2ffd640 1 -- 172.21.15.114:0/2857745166 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f4bc8017560 con 0x7f4bb80039c0 2024-09-16T05:21:02.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bc2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:21:02.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bc2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:21:02.319 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":58,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:21:02.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 10 librados: watch_flush enter 2024-09-16T05:21:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 10 librados: watch_flush exit 2024-09-16T05:21:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 20 client.104135.objecter shutdown clearing up homeless session... 2024-09-16T05:21:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 10 client.104135.objecter successfully canceled tick 2024-09-16T05:21:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 1 -- 172.21.15.114:0/2857745166 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4bb0037e80 msgr2=0x7f4bb003a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 1 --2- 172.21.15.114:0/2857745166 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4bb0037e80 0x7f4bb003a340 secure :-1 s=READY pgs=3572 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc001ec0 tx=0x7f4bbc008040 comp rx=0 tx=0).stop 2024-09-16T05:21:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 10 monclient: shutdown 2024-09-16T05:21:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:21:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 1 -- 172.21.15.114:0/2857745166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bb80039c0 msgr2=0x7f4bcc0989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 1 --2- 172.21.15.114:0/2857745166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bb80039c0 0x7f4bcc0989b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4bc801cfc0 tx=0x7f4bc801a2f0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 1 -- 172.21.15.114:0/2857745166 shutdown_connections 2024-09-16T05:21:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 1 --2- 172.21.15.114:0/2857745166 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4bb0037e80 0x7f4bb003a340 unknown :-1 s=CLOSED pgs=3572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 1 --2- 172.21.15.114:0/2857745166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb8004810 0x7f4bcc0912e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 1 --2- 172.21.15.114:0/2857745166 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4bb8005220 0x7f4bcc091820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 1 --2- 172.21.15.114:0/2857745166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bb80039c0 0x7f4bcc0989b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 1 -- 172.21.15.114:0/2857745166 >> 172.21.15.114:0/2857745166 conn(0x7f4bcc0a1fb0 msgr2=0x7f4bcc0a32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:21:02.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 1 -- 172.21.15.114:0/2857745166 shutdown_connections 2024-09-16T05:21:02.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 1 -- 172.21.15.114:0/2857745166 wait complete. 2024-09-16T05:21:02.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.322+0000 7f4bd47b4640 1 librados: shutdown 2024-09-16T05:21:02.338 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:21:02.338 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T05:21:02.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:21:02.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 10 monclient: get_monmap_and_config 2024-09-16T05:21:02.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 10 monclient: build_initial_monmap 2024-09-16T05:21:02.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 10 monclient: monmap: 2024-09-16T05:21:02.460 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:21:02.460 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.460 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:21:02.465332+0000 2024-09-16T05:21:02.461 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:21:02.465332+0000 2024-09-16T05:21:02.461 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:21:02.461 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.461 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:21:02.461 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:21:02.461 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:21:02.461 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:21:02.461 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:21:02.461 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 1 Processor -- start 2024-09-16T05:21:02.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 1 -- start start 2024-09-16T05:21:02.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 10 monclient: init 2024-09-16T05:21:02.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 10 monclient: _reopen_session rank -1 2024-09-16T05:21:02.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-16T05:21:02.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff5580a0ec0 0x7ff55809dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 10 monclient(hunting): picked mon.noname-c con 0x7ff5580a0ec0 addr v2:172.21.15.149:3300/0 2024-09-16T05:21:02.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff55809e000 0x7ff55809e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 10 monclient(hunting): picked mon.noname-a con 0x7ff55809e000 addr v2:172.21.15.114:3300/0 2024-09-16T05:21:02.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff558138630 addr v1:172.21.15.149:6789/0 2024-09-16T05:21:02.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff55809e950 con 0x7ff558138630 2024-09-16T05:21:02.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff5580994b0 con 0x7ff55809e000 2024-09-16T05:21:02.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7ff55809a760 con 0x7ff5580a0ec0 2024-09-16T05:21:02.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 10 monclient(hunting): _renew_subs 2024-09-16T05:21:02.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55f7bb640 10 monclient(hunting): authenticate will time out at 1747.094660s 2024-09-16T05:21:02.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff55809e000 0x7ff55809e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55dd58640 1 -- >> v1:172.21.15.149:6789/0 conn(0x7ff558138630 legacy=0x7ff55813aa20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:21:02.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff55809e000 0x7ff55809e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34798/0 (socket says 172.21.15.114:34798) 2024-09-16T05:21:02.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55d557640 1 -- >> v2:172.21.15.149:3300/0 conn(0x7ff5580a0ec0 msgr2=0x7ff55809dac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:21:02.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 1 -- 172.21.15.114:0/1809216530 learned_addr learned my addr 172.21.15.114:0/1809216530 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:21:02.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 10 monclient(hunting): get_auth_request con 0x7ff55809e000 auth_method 0 2024-09-16T05:21:02.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:02.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:21:02.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55d557640 1 --2- 172.21.15.114:0/1809216530 >> v2:172.21.15.149:3300/0 conn(0x7ff5580a0ec0 0x7ff55809dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:21:02.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:21:02.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:21:02.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:21:02.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 10 monclient(hunting): handle_auth_done global_id 74613 payload 274 2024-09-16T05:21:02.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 1 -- 172.21.15.114:0/1809216530 >> v2:172.21.15.149:3300/0 conn(0x7ff5580a0ec0 msgr2=0x7ff55809dac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:21:02.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 1 --2- 172.21.15.114:0/1809216530 >> v2:172.21.15.149:3300/0 conn(0x7ff5580a0ec0 0x7ff55809dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 1 -- 172.21.15.114:0/1809216530 >> v1:172.21.15.149:6789/0 conn(0x7ff558138630 legacy=0x7ff55813aa20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:21:02.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 10 monclient: _finish_hunting 0 2024-09-16T05:21:02.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 1 monclient: found mon.noname-a 2024-09-16T05:21:02.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:21:02.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:21:02.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.462+0000 7ff55cd56640 1 -- 172.21.15.114:0/1809216530 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5580a1870 con 0x7ff55809e000 2024-09-16T05:21:02.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55cd56640 1 --2- 172.21.15.114:0/1809216530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff55809e000 0x7ff55809e410 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff5580a2fd0 tx=0x7ff54801eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0feeb6c1c2c1a8e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:21:02.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff557fff640 1 -- 172.21.15.114:0/1809216530 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54800b2d0 con 0x7ff55809e000 2024-09-16T05:21:02.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff557fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff557fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:21:02.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff557fff640 10 monclient: dump: 2024-09-16T05:21:02.468 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.468 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.468 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.468 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.469 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.469 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.469 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.469 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.469 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.469 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff557fff640 10 monclient: _finish_auth 0 2024-09-16T05:21:02.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff557fff640 10 monclient: _check_auth_tickets 2024-09-16T05:21:02.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff557fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:21:02.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff557fff640 1 -- 172.21.15.114:0/1809216530 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff54800b470 con 0x7ff55809e000 2024-09-16T05:21:02.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 5 monclient: authenticate success, global_id 74613 2024-09-16T05:21:02.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:21:02.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff557fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:21:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 10 monclient: get_monmap_and_config success 2024-09-16T05:21:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 4 set_mon_vals no callback set 2024-09-16T05:21:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff557fff640 1 -- 172.21.15.114:0/1809216530 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5480a7b40 con 0x7ff55809e000 2024-09-16T05:21:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff557fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff557fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:21:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff557fff640 10 monclient: dump: 2024-09-16T05:21:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.471 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.471 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 10 monclient: shutdown 2024-09-16T05:21:02.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:21:02.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 -- 172.21.15.114:0/1809216530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff55809e000 msgr2=0x7ff55809e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 --2- 172.21.15.114:0/1809216530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff55809e000 0x7ff55809e410 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff5580a2fd0 tx=0x7ff54801eb50 comp rx=0 tx=0).stop 2024-09-16T05:21:02.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 -- 172.21.15.114:0/1809216530 shutdown_connections 2024-09-16T05:21:02.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 --2- 172.21.15.114:0/1809216530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff55809e000 0x7ff55809e410 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 --2- 172.21.15.114:0/1809216530 >> v2:172.21.15.149:3300/0 conn(0x7ff5580a0ec0 0x7ff55809dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 -- 172.21.15.114:0/1809216530 >> 172.21.15.114:0/1809216530 conn(0x7ff5580a1fb0 msgr2=0x7ff5580a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:21:02.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 -- 172.21.15.114:0/1809216530 shutdown_connections 2024-09-16T05:21:02.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 -- 172.21.15.114:0/1809216530 wait complete. 2024-09-16T05:21:02.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 10 monclient: build_initial_monmap 2024-09-16T05:21:02.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 10 monclient: monmap: 2024-09-16T05:21:02.479 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:21:02.479 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:21:02.480 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:21:02.480 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:21:02.480 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:21:02.480 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.480 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:21:02.480 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:21:02.480 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:21:02.480 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 librados: starting msgr at 2024-09-16T05:21:02.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 librados: starting objecter 2024-09-16T05:21:02.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 Processor -- start 2024-09-16T05:21:02.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 -- start start 2024-09-16T05:21:02.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 librados: setting wanted keys 2024-09-16T05:21:02.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 librados: calling monclient init 2024-09-16T05:21:02.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 10 monclient: init 2024-09-16T05:21:02.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 10 monclient: _reopen_session rank -1 2024-09-16T05:21:02.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:21:02.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff55809e000 0x7ff558098870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 10 monclient(hunting): picked mon.noname-b con 0x7ff55809e000 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:21:02.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5580a0ec0 0x7ff558091410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 10 monclient(hunting): picked mon.noname-a con 0x7ff5580a0ec0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:21:02.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff558138630 0x7ff558091950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 10 monclient(hunting): picked mon.noname-c con 0x7ff558138630 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:21:02.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff55809c3b0 con 0x7ff5580a0ec0 2024-09-16T05:21:02.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7ff55809c230 con 0x7ff55809e000 2024-09-16T05:21:02.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff55809d1c0 con 0x7ff558138630 2024-09-16T05:21:02.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 10 monclient(hunting): _renew_subs 2024-09-16T05:21:02.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55f7bb640 10 monclient(hunting): authenticate will time out at 1747.099576s 2024-09-16T05:21:02.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55cd56640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5580a0ec0 0x7ff558091410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55dd58640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff558138630 0x7ff558091950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55cd56640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5580a0ec0 0x7ff558091410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34810/0 (socket says 172.21.15.114:34810) 2024-09-16T05:21:02.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55d557640 1 -- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff55809e000 msgr2=0x7ff558098870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:21:02.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55cd56640 1 -- 172.21.15.114:0/1202632410 learned_addr learned my addr 172.21.15.114:0/1202632410 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:21:02.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55cd56640 10 monclient(hunting): get_auth_request con 0x7ff5580a0ec0 auth_method 0 2024-09-16T05:21:02.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55cd56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:02.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55cd56640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55cd56640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:21:02.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55d557640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff55809e000 0x7ff558098870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:21:02.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55dd58640 10 monclient(hunting): get_auth_request con 0x7ff558138630 auth_method 0 2024-09-16T05:21:02.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55dd58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:02.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55dd58640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55dd58640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:21:02.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55cd56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:21:02.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55cd56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:21:02.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55cd56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:21:02.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55dd58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:21:02.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55dd58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:21:02.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.466+0000 7ff55dd58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:21:02.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55cd56640 10 monclient(hunting): handle_auth_done global_id 74616 payload 824 2024-09-16T05:21:02.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55cd56640 1 -- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff558138630 msgr2=0x7ff558091950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55cd56640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff558138630 0x7ff558091950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55cd56640 1 -- 172.21.15.114:0/1202632410 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff55809e000 msgr2=0x7ff558098870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:21:02.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55cd56640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff55809e000 0x7ff558098870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55cd56640 10 monclient: _finish_hunting 0 2024-09-16T05:21:02.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55cd56640 1 monclient: found mon.noname-a 2024-09-16T05:21:02.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55cd56640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:21:02.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55cd56640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:21:02.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55cd56640 1 -- 172.21.15.114:0/1202632410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff558098db0 con 0x7ff5580a0ec0 2024-09-16T05:21:02.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55dd58640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff558138630 0x7ff558091950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:21:02.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55cd56640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5580a0ec0 0x7ff558091410 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff54801c780 tx=0x7ff54801b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:21:02.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 client.?.objecter ms_handle_connect 0x7ff5580a0ec0 2024-09-16T05:21:02.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:21:02.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 1 -- 172.21.15.114:0/1202632410 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff548009800 con 0x7ff5580a0ec0 2024-09-16T05:21:02.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:21:02.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 monclient: dump: 2024-09-16T05:21:02.485 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.485 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.485 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.485 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.485 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.485 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.485 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.485 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.485 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.486 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 monclient: _finish_auth 0 2024-09-16T05:21:02.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:21:02.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:21:02.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55f7bb640 5 monclient: authenticate success, global_id 74616 2024-09-16T05:21:02.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 1 -- 172.21.15.114:0/1202632410 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5480be070 con 0x7ff5580a0ec0 2024-09-16T05:21:02.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:21:02.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 1 -- 172.21.15.114:0/1202632410 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff548009be0 con 0x7ff5580a0ec0 2024-09-16T05:21:02.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:21:02.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 monclient: dump: 2024-09-16T05:21:02.486 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.486 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.486 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.487 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.487 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.487 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.487 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.487 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.487 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.487 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff5577fe640 4 set_mon_vals no callback set 2024-09-16T05:21:02.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55f7bb640 10 monclient: _renew_subs 2024-09-16T05:21:02.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55f7bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:21:02.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55f7bb640 1 -- 172.21.15.114:0/1202632410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff558094910 con 0x7ff5580a0ec0 2024-09-16T05:21:02.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55f7bb640 10 client.74616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:21:02.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55f7bb640 10 monclient: _renew_subs 2024-09-16T05:21:02.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55f7bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:21:02.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55f7bb640 1 -- 172.21.15.114:0/1202632410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff558092c00 con 0x7ff5580a0ec0 2024-09-16T05:21:02.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55f7bb640 1 librados: init done 2024-09-16T05:21:02.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55f7bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:21:02.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55f7bb640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff558091e90 0x7ff55806f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55f7bb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55f7bb640 1 -- 172.21.15.114:0/1202632410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff558069390 con 0x7ff558091e90 2024-09-16T05:21:02.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55d557640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff558091e90 0x7ff55806f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55d557640 10 monclient: get_auth_request con 0x7ff558091e90 auth_method 0 2024-09-16T05:21:02.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55d557640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:02.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55d557640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55d557640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:21:02.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 1 -- 172.21.15.114:0/1202632410 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff5480b6070 con 0x7ff5580a0ec0 2024-09-16T05:21:02.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55d557640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:21:02.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55d557640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:21:02.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55d557640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:21:02.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55d557640 10 monclient(hunting): handle_auth_done global_id 74619 payload 824 2024-09-16T05:21:02.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 1 --2- 172.21.15.114:0/1202632410 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff5380377c0 0x7ff538039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55d557640 1 -- 172.21.15.114:0/1202632410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff558092270 con 0x7ff558091e90 2024-09-16T05:21:02.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 1 -- 172.21.15.114:0/1202632410 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(529..529 src has 1..529) ==== 5861+0+0 (secure 0 0 0) 0x7ff5480e3a80 con 0x7ff5580a0ec0 2024-09-16T05:21:02.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 client.74616.objecter ms_dispatch 0x7ff558000ba0 osd_map(529..529 src has 1..529) 2024-09-16T05:21:02.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 3 client.74616.objecter handle_osd_map got epochs [529,529] > 0 2024-09-16T05:21:02.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 3 client.74616.objecter handle_osd_map decoding full epoch 529 2024-09-16T05:21:02.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55d557640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff558091e90 0x7ff55806f9a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff54000b450 tx=0x7ff54000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:21:02.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 20 client.74616.objecter dump_active .. 0 homeless 2024-09-16T05:21:02.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 client.74616.objecter ms_handle_connect 0x7ff558091e90 2024-09-16T05:21:02.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 client.74616.objecter resend_mon_ops 2024-09-16T05:21:02.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 1 -- 172.21.15.114:0/1202632410 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff540003d50 con 0x7ff558091e90 2024-09-16T05:21:02.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:21:02.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 monclient: dump: 2024-09-16T05:21:02.490 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.490 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.490 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.490 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.490 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.490 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.490 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.490 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.490 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.490 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55dd58640 1 --2- 172.21.15.114:0/1202632410 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff5380377c0 0x7ff538039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55dd58640 10 monclient: get_auth_request con 0x7ff5380377c0 auth_method 0 2024-09-16T05:21:02.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 1 -- 172.21.15.114:0/1202632410 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff540015070 con 0x7ff558091e90 2024-09-16T05:21:02.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:21:02.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:21:02.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 1 -- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff558091e90 msgr2=0x7ff55806f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff558091e90 0x7ff55806f9a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff54000b450 tx=0x7ff54000b910 comp rx=0 tx=0).stop 2024-09-16T05:21:02.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 1 -- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff558091e90 msgr2=0x7ff55806f9a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:21:02.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff558091e90 0x7ff55806f9a0 secure :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff54000b450 tx=0x7ff54000b910 comp rx=0 tx=0).stop 2024-09-16T05:21:02.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff555ffb640 10 client.74616.objecter ms_handle_connect 0x7ff5380377c0 2024-09-16T05:21:02.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.470+0000 7ff55dd58640 1 --2- 172.21.15.114:0/1202632410 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff5380377c0 0x7ff538039c80 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 crypto rx=0x7ff550003850 tx=0x7ff5500090e0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:21:02.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:21:02.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff55806fee0 0x7ff5580702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 1 -- 172.21.15.114:0/1202632410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff558094e30 con 0x7ff55806fee0 2024-09-16T05:21:02.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55d557640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff55806fee0 0x7ff5580702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55d557640 10 monclient: get_auth_request con 0x7ff55806fee0 auth_method 0 2024-09-16T05:21:02.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55d557640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:02.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55d557640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55d557640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:21:02.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55d557640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:21:02.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55d557640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:21:02.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55d557640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:21:02.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55d557640 10 monclient(hunting): handle_auth_done global_id 74622 payload 824 2024-09-16T05:21:02.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55d557640 1 -- 172.21.15.114:0/1202632410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff558093040 con 0x7ff55806fee0 2024-09-16T05:21:02.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff555ffb640 10 client.74616.objecter ms_handle_connect 0x7ff55806fee0 2024-09-16T05:21:02.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff555ffb640 10 client.74616.objecter resend_mon_ops 2024-09-16T05:21:02.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55d557640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff55806fee0 0x7ff5580702f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff540000c10 tx=0x7ff5400059a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:21:02.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff555ffb640 1 -- 172.21.15.114:0/1202632410 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff540005640 con 0x7ff55806fee0 2024-09-16T05:21:02.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff555ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff555ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:21:02.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff555ffb640 10 monclient: dump: 2024-09-16T05:21:02.493 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.493 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.493 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.493 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.493 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.493 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.493 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.493 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.493 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.493 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff555ffb640 1 -- 172.21.15.114:0/1202632410 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7ff540005640 con 0x7ff55806fee0 2024-09-16T05:21:02.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff555ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:21:02.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff555ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:21:02.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff555ffb640 1 -- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff55806fee0 msgr2=0x7ff5580702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff555ffb640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff55806fee0 0x7ff5580702f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff540000c10 tx=0x7ff5400059a0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff555ffb640 1 -- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff55806fee0 msgr2=0x7ff5580702f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:21:02.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff555ffb640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff55806fee0 0x7ff5580702f0 secure :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff540000c10 tx=0x7ff5400059a0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.494 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:21:02.494 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:21:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:21:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:21:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 58, 2024-09-16T05:21:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 421534, 2024-09-16T05:21:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:21:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:21:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:21:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:21:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:21:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:21:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:21:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:21:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:21:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:21:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:21:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:21:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:21:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:21:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:21:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:21:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:21:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:21:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:21:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:21:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:21:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:21:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:21:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:21:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:21:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:21:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:21:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:21:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:21:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:21:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:21:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:21:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:21:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:21:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:21:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:21:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:21:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:21:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:21:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:21:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:21:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:21:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:21:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:21:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:21:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:21:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:21:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:21:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:21:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:21:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:21:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:21:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:21:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:21:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:21:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:21:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:21:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:21:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:21:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:21:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:21:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:21:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:21:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:21:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:21:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:21:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:21:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:21:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:21:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:21:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:21:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:21:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:21:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:21:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:21:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:21:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:21:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:21:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:21:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:21:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:21:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:21:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:21:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:21:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:21:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:21:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:21:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:21:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:21:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:21:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:21:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:21:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:21:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:21:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:21:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:21:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:21:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:21:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:21:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:21:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:21:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:21:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:21:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:21:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:21:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:21:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:21:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:21:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:21:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:21:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:21:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:21:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:21:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:21:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:21:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:21:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:21:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:21:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:21:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:21:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:21:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:21:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:21:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:21:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:21:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:21:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:21:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:21:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:21:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:21:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:21:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:21:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:21:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:21:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:21:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:21:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:21:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:21:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:21:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:21:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:21:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:21:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:21:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:21:02.507 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:21:02.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 10 librados: watch_flush enter 2024-09-16T05:21:02.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 10 librados: watch_flush exit 2024-09-16T05:21:02.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 20 client.74616.objecter shutdown clearing up homeless session... 2024-09-16T05:21:02.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 10 client.74616.objecter successfully canceled tick 2024-09-16T05:21:02.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 1 -- 172.21.15.114:0/1202632410 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff5380377c0 msgr2=0x7ff538039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 1 --2- 172.21.15.114:0/1202632410 >> v2:172.21.15.149:6812/1185531135 conn(0x7ff5380377c0 0x7ff538039c80 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 crypto rx=0x7ff550003850 tx=0x7ff5500090e0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 10 monclient: shutdown 2024-09-16T05:21:02.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:21:02.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 1 -- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5580a0ec0 msgr2=0x7ff558091410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 1 --2- 172.21.15.114:0/1202632410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5580a0ec0 0x7ff558091410 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff54801c780 tx=0x7ff54801b8f0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55dd58640 1 -- 172.21.15.114:0/1202632410 reap_dead start 2024-09-16T05:21:02.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55dd58640 1 -- 172.21.15.114:0/1202632410 reap_dead start 2024-09-16T05:21:02.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 1 -- 172.21.15.114:0/1202632410 shutdown_connections 2024-09-16T05:21:02.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 1 -- 172.21.15.114:0/1202632410 >> 172.21.15.114:0/1202632410 conn(0x7ff5580a1fb0 msgr2=0x7ff5581396c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:21:02.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 1 -- 172.21.15.114:0/1202632410 shutdown_connections 2024-09-16T05:21:02.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 1 -- 172.21.15.114:0/1202632410 wait complete. 2024-09-16T05:21:02.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.478+0000 7ff55f7bb640 1 librados: shutdown 2024-09-16T05:21:02.510 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:21:02.510 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T05:21:02.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:21:02.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.626+0000 7f10f7cc4640 10 monclient: get_monmap_and_config 2024-09-16T05:21:02.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.626+0000 7f10f7cc4640 10 monclient: build_initial_monmap 2024-09-16T05:21:02.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.626+0000 7f10f7cc4640 10 monclient: monmap: 2024-09-16T05:21:02.627 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:21:02.627 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.628 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:21:02.632174+0000 2024-09-16T05:21:02.628 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:21:02.632174+0000 2024-09-16T05:21:02.628 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:21:02.628 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.628 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:21:02.628 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:21:02.628 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:21:02.628 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:21:02.628 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:21:02.628 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f7cc4640 1 Processor -- start 2024-09-16T05:21:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f7cc4640 1 -- start start 2024-09-16T05:21:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f7cc4640 10 monclient: init 2024-09-16T05:21:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f7cc4640 10 monclient: _reopen_session rank -1 2024-09-16T05:21:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f7cc4640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-16T05:21:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f7cc4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f0137730 0x7f10f0137b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f7cc4640 10 monclient(hunting): picked mon.noname-b con 0x7f10f0137730 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:21:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f7cc4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10f00a5d00 addr v1:172.21.15.114:6789/0 2024-09-16T05:21:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f7cc4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10f00a4980 addr v1:172.21.15.149:6789/0 2024-09-16T05:21:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f7cc4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10f00a55e0 con 0x7f10f00a5d00 2024-09-16T05:21:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f7cc4640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10f00a3710 con 0x7f10f00a4980 2024-09-16T05:21:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f7cc4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f7cc4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f10f009c450 con 0x7f10f0137730 2024-09-16T05:21:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f7cc4640 10 monclient(hunting): _renew_subs 2024-09-16T05:21:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f7cc4640 10 monclient(hunting): authenticate will time out at 1747.261510s 2024-09-16T05:21:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f525f640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f10f00a5d00 0x7f10f00a4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49234/0 (socket says 172.21.15.114:49234) 2024-09-16T05:21:02.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f525f640 1 -- 172.21.15.114:0/2727349205 learned_addr learned my addr 172.21.15.114:0/2727349205 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:21:02.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f6261640 1 -- 172.21.15.114:0/2727349205 >> v1:172.21.15.149:6789/0 conn(0x7f10f00a4980 legacy=0x7f10f013f4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:21:02.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 1 --2- 172.21.15.114:0/2727349205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f0137730 0x7f10f0137b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 10 monclient(hunting): get_auth_request con 0x7f10f0137730 auth_method 0 2024-09-16T05:21:02.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:02.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:21:02.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:21:02.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:21:02.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:21:02.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f4a5e640 1 -- 172.21.15.114:0/2727349205 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2288799810 0 0) 0x7f10f00a55e0 con 0x7f10f00a5d00 2024-09-16T05:21:02.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f4a5e640 10 monclient(hunting): _negotiate 2024-09-16T05:21:02.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f4a5e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f4a5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:21:02.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f4a5e640 10 monclient(hunting): my global_id is 74625 2024-09-16T05:21:02.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f4a5e640 1 -- 172.21.15.114:0/2727349205 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10dc003530 con 0x7f10f00a5d00 2024-09-16T05:21:02.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 10 monclient(hunting): handle_auth_done global_id 104141 payload 274 2024-09-16T05:21:02.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 1 -- 172.21.15.114:0/2727349205 >> v1:172.21.15.149:6789/0 conn(0x7f10f00a4980 legacy=0x7f10f013f4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:21:02.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 1 -- 172.21.15.114:0/2727349205 >> v1:172.21.15.114:6789/0 conn(0x7f10f00a5d00 legacy=0x7f10f00a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 10 monclient: _finish_hunting 0 2024-09-16T05:21:02.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 1 monclient: found mon.noname-b 2024-09-16T05:21:02.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:21:02.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:21:02.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.630+0000 7f10f5a60640 1 -- 172.21.15.114:0/2727349205 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f009be00 con 0x7f10f0137730 2024-09-16T05:21:02.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f5a60640 1 --2- 172.21.15.114:0/2727349205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f0137730 0x7f10f0137b10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f10ec0097f0 tx=0x7f10ec01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d88582fbe8a23f6e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:21:02.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f4a5e640 1 -- 172.21.15.114:0/2727349205 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0b0070 con 0x7f10f0137730 2024-09-16T05:21:02.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f4a5e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f4a5e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:21:02.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f4a5e640 10 monclient: dump: 2024-09-16T05:21:02.645 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.645 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.645 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.645 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.645 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.648 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.648 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.648 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.649 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f4a5e640 10 monclient: _finish_auth 0 2024-09-16T05:21:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f4a5e640 10 monclient: _check_auth_tickets 2024-09-16T05:21:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f4a5e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:21:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f4a5e640 1 -- 172.21.15.114:0/2727349205 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ec0ab600 con 0x7f10f0137730 2024-09-16T05:21:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 5 monclient: authenticate success, global_id 104141 2024-09-16T05:21:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:21:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f4a5e640 10 monclient: handle_config config(0 keys) 2024-09-16T05:21:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f4a5e640 1 -- 172.21.15.114:0/2727349205 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0ab9d0 con 0x7f10f0137730 2024-09-16T05:21:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 10 monclient: get_monmap_and_config success 2024-09-16T05:21:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 4 set_mon_vals no callback set 2024-09-16T05:21:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f4a5e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f4a5e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:21:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f4a5e640 10 monclient: dump: 2024-09-16T05:21:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.651 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.651 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.651 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.651 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.651 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 10 monclient: shutdown 2024-09-16T05:21:02.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:21:02.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 -- 172.21.15.114:0/2727349205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f0137730 msgr2=0x7f10f0137b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 --2- 172.21.15.114:0/2727349205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f0137730 0x7f10f0137b10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f10ec0097f0 tx=0x7f10ec01c360 comp rx=0 tx=0).stop 2024-09-16T05:21:02.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 -- 172.21.15.114:0/2727349205 shutdown_connections 2024-09-16T05:21:02.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 --2- 172.21.15.114:0/2727349205 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f0137730 0x7f10f0137b10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 -- 172.21.15.114:0/2727349205 >> 172.21.15.114:0/2727349205 conn(0x7f10f00a9c70 msgr2=0x7f10f00aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:21:02.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 -- 172.21.15.114:0/2727349205 shutdown_connections 2024-09-16T05:21:02.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 -- 172.21.15.114:0/2727349205 wait complete. 2024-09-16T05:21:02.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 10 monclient: build_initial_monmap 2024-09-16T05:21:02.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 10 monclient: monmap: 2024-09-16T05:21:02.652 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:21:02.652 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:21:02.652 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:21:02.652 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:21:02.652 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:21:02.652 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.652 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:21:02.653 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:21:02.653 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:21:02.653 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 librados: starting msgr at 2024-09-16T05:21:02.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 librados: starting objecter 2024-09-16T05:21:02.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 Processor -- start 2024-09-16T05:21:02.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 -- start start 2024-09-16T05:21:02.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 librados: setting wanted keys 2024-09-16T05:21:02.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 librados: calling monclient init 2024-09-16T05:21:02.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 10 monclient: init 2024-09-16T05:21:02.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 10 monclient: _reopen_session rank -1 2024-09-16T05:21:02.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:21:02.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f10f00a4980 0x7f10f0135a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 10 monclient(hunting): picked mon.noname-b con 0x7f10f00a4980 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:21:02.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f00a5d00 0x7f10f0135f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 10 monclient(hunting): picked mon.noname-c con 0x7f10f00a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:21:02.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10f0137730 0x7f10f013be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 10 monclient(hunting): picked mon.noname-a con 0x7f10f0137730 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:21:02.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f10f0137320 con 0x7f10f0137730 2024-09-16T05:21:02.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f10f01371a0 con 0x7f10f00a4980 2024-09-16T05:21:02.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f10f0143420 con 0x7f10f00a5d00 2024-09-16T05:21:02.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 10 monclient(hunting): _renew_subs 2024-09-16T05:21:02.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f7cc4640 10 monclient(hunting): authenticate will time out at 1747.267348s 2024-09-16T05:21:02.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f525f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f00a5d00 0x7f10f0135f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10f0137730 0x7f10f013be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f5a60640 1 -- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f10f00a4980 msgr2=0x7f10f0135a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:21:02.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f5a60640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f10f00a4980 0x7f10f0135a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:21:02.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10f0137730 0x7f10f013be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34846/0 (socket says 172.21.15.114:34846) 2024-09-16T05:21:02.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f525f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f00a5d00 0x7f10f0135f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:42384/0 (socket says 172.21.15.114:42384) 2024-09-16T05:21:02.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 1 -- 172.21.15.114:0/437911036 learned_addr learned my addr 172.21.15.114:0/437911036 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:21:02.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 10 monclient(hunting): get_auth_request con 0x7f10f0137730 auth_method 0 2024-09-16T05:21:02.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:02.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:21:02.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f525f640 10 monclient(hunting): get_auth_request con 0x7f10f00a5d00 auth_method 0 2024-09-16T05:21:02.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f525f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:02.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f525f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f525f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:21:02.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f525f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:21:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f525f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:21:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f525f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:21:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:21:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:21:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:21:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 10 monclient(hunting): handle_auth_done global_id 74628 payload 824 2024-09-16T05:21:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 1 -- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f00a5d00 msgr2=0x7f10f0135f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 1 --2- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f00a5d00 0x7f10f0135f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 1 -- 172.21.15.114:0/437911036 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f10f00a4980 msgr2=0x7f10f0135a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:21:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 1 --2- 172.21.15.114:0/437911036 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f10f00a4980 0x7f10f0135a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 10 monclient: _finish_hunting 0 2024-09-16T05:21:02.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 1 monclient: found mon.noname-a 2024-09-16T05:21:02.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:21:02.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:21:02.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 1 -- 172.21.15.114:0/437911036 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f013fa90 con 0x7f10f0137730 2024-09-16T05:21:02.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f525f640 1 --2- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f00a5d00 0x7f10f0135f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:21:02.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10f6261640 1 --2- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10f0137730 0x7f10f013be60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f10e800b760 tx=0x7f10e800bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:21:02.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10e67fc640 10 client.?.objecter ms_handle_connect 0x7f10f0137730 2024-09-16T05:21:02.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.634+0000 7f10e67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:21:02.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 1 -- 172.21.15.114:0/437911036 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e800c960 con 0x7f10f0137730 2024-09-16T05:21:02.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:21:02.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 monclient: dump: 2024-09-16T05:21:02.659 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.659 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.659 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.659 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.659 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.659 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.659 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.659 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.660 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.660 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 monclient: _finish_auth 0 2024-09-16T05:21:02.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 monclient: _check_auth_tickets 2024-09-16T05:21:02.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:21:02.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f7cc4640 5 monclient: authenticate success, global_id 74628 2024-09-16T05:21:02.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 1 -- 172.21.15.114:0/437911036 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10e8010070 con 0x7f10f0137730 2024-09-16T05:21:02.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f7cc4640 10 monclient: _renew_subs 2024-09-16T05:21:02.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f7cc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:21:02.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f7cc4640 1 -- 172.21.15.114:0/437911036 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f009cd50 con 0x7f10f0137730 2024-09-16T05:21:02.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:21:02.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 1 -- 172.21.15.114:0/437911036 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e800cdc0 con 0x7f10f0137730 2024-09-16T05:21:02.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:21:02.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 monclient: dump: 2024-09-16T05:21:02.661 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.661 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.661 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.661 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.661 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.661 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.661 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.662 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.662 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.662 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e7fff640 4 set_mon_vals no callback set 2024-09-16T05:21:02.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f7cc4640 10 client.74628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:21:02.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f7cc4640 10 monclient: _renew_subs 2024-09-16T05:21:02.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f7cc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:21:02.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f7cc4640 1 -- 172.21.15.114:0/437911036 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f01364d0 con 0x7f10f0137730 2024-09-16T05:21:02.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f7cc4640 1 librados: init done 2024-09-16T05:21:02.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f7cc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:21:02.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f7cc4640 1 --2- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f013c3a0 0x7f10f013c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f7cc4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f7cc4640 1 -- 172.21.15.114:0/437911036 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f10f0073820 con 0x7f10f013c3a0 2024-09-16T05:21:02.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f5a60640 1 --2- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f013c3a0 0x7f10f013c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f5a60640 10 monclient: get_auth_request con 0x7f10f013c3a0 auth_method 0 2024-09-16T05:21:02.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f5a60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:02.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f5a60640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f5a60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:21:02.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f5a60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:21:02.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f5a60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:21:02.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 1 -- 172.21.15.114:0/437911036 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f10e8015470 con 0x7f10f0137730 2024-09-16T05:21:02.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f5a60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:21:02.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f5a60640 10 monclient(hunting): handle_auth_done global_id 104147 payload 824 2024-09-16T05:21:02.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f5a60640 1 -- 172.21.15.114:0/437911036 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10f0137b10 con 0x7f10f013c3a0 2024-09-16T05:21:02.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 1 --2- 172.21.15.114:0/437911036 >> v2:172.21.15.149:6812/1185531135 conn(0x7f10d40377c0 0x7f10d4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 1 -- 172.21.15.114:0/437911036 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(529..529 src has 1..529) ==== 5861+0+0 (secure 0 0 0) 0x7f10e8048e90 con 0x7f10f0137730 2024-09-16T05:21:02.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 client.74628.objecter ms_dispatch 0x7f10f0000ba0 osd_map(529..529 src has 1..529) 2024-09-16T05:21:02.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 3 client.74628.objecter handle_osd_map got epochs [529,529] > 0 2024-09-16T05:21:02.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 3 client.74628.objecter handle_osd_map decoding full epoch 529 2024-09-16T05:21:02.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f5a60640 1 --2- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f013c3a0 0x7f10f013c780 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f10f0072f50 tx=0x7f10ec01aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:21:02.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 20 client.74628.objecter dump_active .. 0 homeless 2024-09-16T05:21:02.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 client.74628.objecter ms_handle_connect 0x7f10f013c3a0 2024-09-16T05:21:02.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 client.74628.objecter resend_mon_ops 2024-09-16T05:21:02.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 1 -- 172.21.15.114:0/437911036 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0b0040 con 0x7f10f013c3a0 2024-09-16T05:21:02.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:21:02.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 monclient: dump: 2024-09-16T05:21:02.665 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.665 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.665 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.665 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.665 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.665 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.665 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.665 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.665 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.665 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f525f640 1 --2- 172.21.15.114:0/437911036 >> v2:172.21.15.149:6812/1185531135 conn(0x7f10d40377c0 0x7f10d4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f525f640 10 monclient: get_auth_request con 0x7f10d40377c0 auth_method 0 2024-09-16T05:21:02.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 1 -- 172.21.15.114:0/437911036 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f10ec004220 con 0x7f10f013c3a0 2024-09-16T05:21:02.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:21:02.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:21:02.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 1 -- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f013c3a0 msgr2=0x7f10f013c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 1 --2- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f013c3a0 0x7f10f013c780 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f10f0072f50 tx=0x7f10ec01aba0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 1 -- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f013c3a0 msgr2=0x7f10f013c780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:21:02.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 1 --2- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f013c3a0 0x7f10f013c780 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f10f0072f50 tx=0x7f10ec01aba0 comp rx=0 tx=0).stop 2024-09-16T05:21:02.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10e67fc640 10 client.74628.objecter ms_handle_connect 0x7f10d40377c0 2024-09-16T05:21:02.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.638+0000 7f10f525f640 1 --2- 172.21.15.114:0/437911036 >> v2:172.21.15.149:6812/1185531135 conn(0x7f10d40377c0 0x7f10d4039c80 secure :-1 s=READY pgs=3574 cs=0 l=1 rev1=1 crypto rx=0x7f10f013ea80 tx=0x7f10e0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:21:02.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:21:02.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 1 --2- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f013ccc0 0x7f10f0138290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:21:02.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:21:02.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 1 -- 172.21.15.114:0/437911036 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f10f0140290 con 0x7f10f013ccc0 2024-09-16T05:21:02.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f5a60640 1 --2- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f013ccc0 0x7f10f0138290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:21:02.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f5a60640 10 monclient: get_auth_request con 0x7f10f013ccc0 auth_method 0 2024-09-16T05:21:02.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f5a60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:21:02.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f5a60640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:21:02.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f5a60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:21:02.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f5a60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:21:02.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f5a60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:21:02.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f5a60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:21:02.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f5a60640 10 monclient(hunting): handle_auth_done global_id 104150 payload 824 2024-09-16T05:21:02.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f5a60640 1 -- 172.21.15.114:0/437911036 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f10f0068650 con 0x7f10f013ccc0 2024-09-16T05:21:02.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f5a60640 1 --2- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f013ccc0 0x7f10f0138290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f10ec01aa10 tx=0x7f10ec019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:21:02.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10e67fc640 10 client.74628.objecter ms_handle_connect 0x7f10f013ccc0 2024-09-16T05:21:02.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10e67fc640 10 client.74628.objecter resend_mon_ops 2024-09-16T05:21:02.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10e67fc640 1 -- 172.21.15.114:0/437911036 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0b0040 con 0x7f10f013ccc0 2024-09-16T05:21:02.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:21:02.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:21:02.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10e67fc640 10 monclient: dump: 2024-09-16T05:21:02.668 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:21:02.668 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:21:02.668 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.669 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:21:02.669 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:21:02.669 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:21:02.669 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:21:02.669 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:21:02.669 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:21:02.669 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:21:02.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10e67fc640 1 -- 172.21.15.114:0/437911036 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f10ec0c3070 con 0x7f10f013ccc0 2024-09-16T05:21:02.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10e67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:21:02.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10e67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:21:02.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10e67fc640 1 -- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f013ccc0 msgr2=0x7f10f0138290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10e67fc640 1 --2- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f013ccc0 0x7f10f0138290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f10ec01aa10 tx=0x7f10ec019100 comp rx=0 tx=0).stop 2024-09-16T05:21:02.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10e67fc640 1 -- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f013ccc0 msgr2=0x7f10f0138290 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:21:02.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10e67fc640 1 --2- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10f013ccc0 0x7f10f0138290 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f10ec01aa10 tx=0x7f10ec019100 comp rx=0 tx=0).stop 2024-09-16T05:21:02.670 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:21:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:21:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:21:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:21:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 58, 2024-09-16T05:21:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 38811, 2024-09-16T05:21:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:21:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:21:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:21:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:21:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:21:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:21:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:21:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:21:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:21:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:21:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:21:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:21:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:21:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:21:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:21:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:21:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:21:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:21:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:21:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:21:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:21:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:21:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:21:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:21:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:21:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:21:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:21:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:21:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:21:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:21:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:21:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:21:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:21:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:21:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:21:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:21:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:21:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:21:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:21:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:21:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:21:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:21:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:21:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:21:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:21:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:21:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:21:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:21:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:21:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:21:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:21:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:21:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:21:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:21:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:21:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:21:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:21:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:21:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:21:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:21:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:21:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:21:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:21:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:21:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:21:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:21:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:21:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:21:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:21:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:21:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:21:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:21:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:21:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:21:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:21:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:21:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:21:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:21:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:21:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:21:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:21:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:21:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:21:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:21:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:21:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:21:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:21:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:21:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:21:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:21:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:21:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:21:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:21:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:21:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:21:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:21:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:21:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:21:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:21:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:21:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:21:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:21:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:21:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:21:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:21:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:21:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:21:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:21:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:21:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:21:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:21:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:21:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:21:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:21:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:21:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:21:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:21:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:21:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:21:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:21:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:21:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:21:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:21:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:21:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:21:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:21:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:21:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:21:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:21:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:21:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:21:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:21:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:21:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:21:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:21:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:21:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:21:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:21:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:21:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:21:02.685 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:21:02.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 10 librados: watch_flush enter 2024-09-16T05:21:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 10 librados: watch_flush exit 2024-09-16T05:21:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 20 client.74628.objecter shutdown clearing up homeless session... 2024-09-16T05:21:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 10 client.74628.objecter successfully canceled tick 2024-09-16T05:21:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 1 -- 172.21.15.114:0/437911036 >> v2:172.21.15.149:6812/1185531135 conn(0x7f10d40377c0 msgr2=0x7f10d4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 1 --2- 172.21.15.114:0/437911036 >> v2:172.21.15.149:6812/1185531135 conn(0x7f10d40377c0 0x7f10d4039c80 secure :-1 s=READY pgs=3574 cs=0 l=1 rev1=1 crypto rx=0x7f10f013ea80 tx=0x7f10e0009040 comp rx=0 tx=0).stop 2024-09-16T05:21:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 10 monclient: shutdown 2024-09-16T05:21:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:21:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 1 -- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10f0137730 msgr2=0x7f10f013be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:21:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 1 --2- 172.21.15.114:0/437911036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10f0137730 0x7f10f013be60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f10e800b760 tx=0x7f10e800bc20 comp rx=0 tx=0).stop 2024-09-16T05:21:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f6261640 1 -- 172.21.15.114:0/437911036 reap_dead start 2024-09-16T05:21:02.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f6261640 1 -- 172.21.15.114:0/437911036 reap_dead start 2024-09-16T05:21:02.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 1 -- 172.21.15.114:0/437911036 shutdown_connections 2024-09-16T05:21:02.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 1 -- 172.21.15.114:0/437911036 >> 172.21.15.114:0/437911036 conn(0x7f10f00a9c70 msgr2=0x7f10f00a0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:21:02.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 1 -- 172.21.15.114:0/437911036 shutdown_connections 2024-09-16T05:21:02.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 1 -- 172.21.15.114:0/437911036 wait complete. 2024-09-16T05:21:02.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:21:02.646+0000 7f10f7cc4640 1 librados: shutdown 2024-09-16T05:21:02.688 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:21:02.688 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:21:51.106 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:21:51.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (8980 ms) 2024-09-16T05:21:51.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-09-16T05:21:51.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:21:51.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (5984 ms) 2024-09-16T05:21:51.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-09-16T05:21:51.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:21:51.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5048 ms) 2024-09-16T05:21:51.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-09-16T05:21:51.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:21:51.107 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (31034 ms) 2024-09-16T05:21:51.108 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-09-16T05:21:51.108 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:21:51.108 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13301 ms) 2024-09-16T05:21:51.108 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-09-16T05:21:51.108 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (11633 ms) 2024-09-16T05:21:51.108 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-09-16T05:21:51.108 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-09-16T05:21:51.108 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: 00000010 69 73 |is| 2024-09-16T05:21:51.108 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: 00000012 2024-09-16T05:21:51.108 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12031 ms) 2024-09-16T05:21:51.108 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-09-16T05:21:51.108 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T05:21:51.109 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6054 ms) 2024-09-16T05:21:51.109 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (632054 ms total) 2024-09-16T05:21:51.109 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: 2024-09-16T05:21:51.109 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-09-16T05:21:51.109 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-09-16T05:21:51.109 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1034 ms) 2024-09-16T05:21:51.109 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-09-16T05:21:51.109 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10911 ms) 2024-09-16T05:21:51.109 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-09-16T05:21:51.109 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (17974 ms) 2024-09-16T05:21:51.109 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-09-16T05:21:51.109 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (5 ms) 2024-09-16T05:21:51.109 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (29924 ms total) 2024-09-16T05:21:51.110 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: 2024-09-16T05:21:51.110 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-09-16T05:21:51.110 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-09-16T05:21:51.110 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7050 ms) 2024-09-16T05:21:51.110 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-09-16T05:21:51.110 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8068 ms) 2024-09-16T05:21:51.110 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-09-16T05:21:51.110 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: waiting for scrub... 2024-09-16T05:21:51.111 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: done waiting 2024-09-16T05:21:51.111 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (34910 ms) 2024-09-16T05:21:51.111 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-09-16T05:21:51.111 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10011 ms) 2024-09-16T05:21:51.111 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-09-16T05:21:51.111 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8074 ms) 2024-09-16T05:21:51.111 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-09-16T05:21:51.111 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (7978 ms) 2024-09-16T05:21:51.111 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-09-16T05:21:51.111 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10001 ms) 2024-09-16T05:21:51.111 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-09-16T05:22:32.689 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:22:32.690 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T05:22:32.690 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T05:22:32.691 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T05:22:32.694 INFO:tasks.ceph.mon.b:Started 2024-09-16T05:22:32.694 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T05:22:32.694 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T05:22:32.695 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T05:22:47.695 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T05:22:47.696 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T05:22:47.696 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:22:47.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:22:47.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.829+0000 7fe17650d640 10 monclient: get_monmap_and_config 2024-09-16T05:22:47.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.829+0000 7fe17650d640 10 monclient: build_initial_monmap 2024-09-16T05:22:47.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.829+0000 7fe17650d640 10 monclient: monmap: 2024-09-16T05:22:47.830 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:47.830 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:47.830 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:22:47.835998+0000 2024-09-16T05:22:47.830 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:22:47.835998+0000 2024-09-16T05:22:47.831 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:47.831 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:47.831 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:22:47.831 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:22:47.831 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:22:47.831 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:22:47.831 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:22:47.831 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:47.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 1 Processor -- start 2024-09-16T05:22:47.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 1 -- start start 2024-09-16T05:22:47.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 10 monclient: init 2024-09-16T05:22:47.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:47.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-16T05:22:47.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe170131e00 0x7fe1700a1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:47.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 10 monclient(hunting): picked mon.noname-b con 0x7fe170131e00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:47.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe1701321e0 0x7fe1701325c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:47.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 10 monclient(hunting): picked mon.noname-a con 0x7fe1701321e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:22:47.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe170134f40 addr v1:172.21.15.114:6789/0 2024-09-16T05:22:47.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1700a1a80 con 0x7fe170134f40 2024-09-16T05:22:47.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:47.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe170131a40 con 0x7fe1701321e0 2024-09-16T05:22:47.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:47.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe1700a0e70 con 0x7fe170131e00 2024-09-16T05:22:47.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:47.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe17650d640 10 monclient(hunting): authenticate will time out at 1852.465148s 2024-09-16T05:22:47.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe170131e00 0x7fe1700a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:47.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16f7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe1701321e0 0x7fe1701325c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:47.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe174aaa640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe170134f40 0x7fe1701353b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34650/0 (socket says 172.21.15.114:34650) 2024-09-16T05:22:47.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe174aaa640 1 -- 172.21.15.114:0/225524366 learned_addr learned my addr 172.21.15.114:0/225524366 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:22:47.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 10 monclient(hunting): get_auth_request con 0x7fe170131e00 auth_method 0 2024-09-16T05:22:47.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:47.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16f7fe640 10 monclient(hunting): get_auth_request con 0x7fe1701321e0 auth_method 0 2024-09-16T05:22:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 10 monclient(hunting): handle_auth_done global_id 104153 payload 274 2024-09-16T05:22:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 1 -- 172.21.15.114:0/225524366 >> v2:172.21.15.114:3300/0 conn(0x7fe1701321e0 msgr2=0x7fe1701325c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:22:47.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 1 --2- 172.21.15.114:0/225524366 >> v2:172.21.15.114:3300/0 conn(0x7fe1701321e0 0x7fe1701325c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:47.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 1 -- 172.21.15.114:0/225524366 >> v1:172.21.15.114:6789/0 conn(0x7fe170134f40 legacy=0x7fe1701353b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:47.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:22:47.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 1 monclient: found mon.noname-b 2024-09-16T05:22:47.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:47.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:22:47.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16ffff640 1 -- 172.21.15.114:0/225524366 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1700a05d0 con 0x7fe170131e00 2024-09-16T05:22:47.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.833+0000 7fe16f7fe640 1 --2- 172.21.15.114:0/225524366 >> v2:172.21.15.114:3300/0 conn(0x7fe1701321e0 0x7fe1701325c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:22:47.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16ffff640 1 --2- 172.21.15.114:0/225524366 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe170131e00 0x7fe1700a1540 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe1600097c0 tx=0x7fe16001c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57cb9c3e91d044a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:47.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16effd640 1 -- 172.21.15.114:0/225524366 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1600b0070 con 0x7fe170131e00 2024-09-16T05:22:47.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:47.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:22:47.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16effd640 10 monclient: dump: 2024-09-16T05:22:47.845 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:47.845 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:47.845 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:47.845 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:47.845 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:47.845 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:47.845 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:47.845 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:47.845 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:47.845 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:47.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16effd640 10 monclient: _finish_auth 0 2024-09-16T05:22:47.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16effd640 10 monclient: _check_auth_tickets 2024-09-16T05:22:47.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:47.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16effd640 1 -- 172.21.15.114:0/225524366 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1600ab600 con 0x7fe170131e00 2024-09-16T05:22:47.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 5 monclient: authenticate success, global_id 104153 2024-09-16T05:22:47.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:22:47.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:47.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient: get_monmap_and_config success 2024-09-16T05:22:47.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 4 set_mon_vals no callback set 2024-09-16T05:22:47.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16effd640 1 -- 172.21.15.114:0/225524366 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1600ab9d0 con 0x7fe170131e00 2024-09-16T05:22:47.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:47.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:22:47.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16effd640 10 monclient: dump: 2024-09-16T05:22:47.846 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:47.846 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:47.847 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:47.847 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:47.847 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:47.847 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:47.847 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:47.847 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:47.847 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:47.847 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:47.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient: shutdown 2024-09-16T05:22:47.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:47.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 -- 172.21.15.114:0/225524366 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe170131e00 msgr2=0x7fe1700a1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:47.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 --2- 172.21.15.114:0/225524366 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe170131e00 0x7fe1700a1540 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe1600097c0 tx=0x7fe16001c3f0 comp rx=0 tx=0).stop 2024-09-16T05:22:47.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 -- 172.21.15.114:0/225524366 shutdown_connections 2024-09-16T05:22:47.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 --2- 172.21.15.114:0/225524366 >> v2:172.21.15.114:3300/0 conn(0x7fe1701321e0 0x7fe1701325c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:47.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 --2- 172.21.15.114:0/225524366 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe170131e00 0x7fe1700a1540 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:47.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 -- 172.21.15.114:0/225524366 >> 172.21.15.114:0/225524366 conn(0x7fe1700a4280 msgr2=0x7fe1700a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:47.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 -- 172.21.15.114:0/225524366 shutdown_connections 2024-09-16T05:22:47.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 -- 172.21.15.114:0/225524366 wait complete. 2024-09-16T05:22:47.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient: build_initial_monmap 2024-09-16T05:22:47.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient: monmap: 2024-09-16T05:22:47.848 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:47.848 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:22:47.848 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:22:47.848 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:22:47.848 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:22:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:22:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:22:47.849 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 librados: starting msgr at 2024-09-16T05:22:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 librados: starting objecter 2024-09-16T05:22:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 Processor -- start 2024-09-16T05:22:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 -- start start 2024-09-16T05:22:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 librados: setting wanted keys 2024-09-16T05:22:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 librados: calling monclient init 2024-09-16T05:22:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient: init 2024-09-16T05:22:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:22:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe170131e00 0x7fe1700703a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient(hunting): picked mon.noname-a con 0x7fe170131e00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:22:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1701321e0 0x7fe1700708e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient(hunting): picked mon.noname-c con 0x7fe1701321e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe170134f40 0x7fe170070e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient(hunting): picked mon.noname-b con 0x7fe170134f40 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:22:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe17013fe30 con 0x7fe170131e00 2024-09-16T05:22:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fe17013fcb0 con 0x7fe170134f40 2024-09-16T05:22:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe17013ffb0 con 0x7fe1701321e0 2024-09-16T05:22:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:47.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient(hunting): authenticate will time out at 1852.470437s 2024-09-16T05:22:47.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe170131e00 0x7fe1700703a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:47.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1701321e0 0x7fe1700708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:47.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe170131e00 0x7fe1700703a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33738/0 (socket says 172.21.15.114:33738) 2024-09-16T05:22:47.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16ffff640 1 -- 172.21.15.114:0/1504588608 learned_addr learned my addr 172.21.15.114:0/1504588608 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:47.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 10 monclient(hunting): get_auth_request con 0x7fe1701321e0 auth_method 0 2024-09-16T05:22:47.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:47.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:47.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:47.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16ffff640 10 monclient(hunting): get_auth_request con 0x7fe170131e00 auth_method 0 2024-09-16T05:22:47.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:47.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe174aaa640 1 --2- 172.21.15.114:0/1504588608 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe170134f40 0x7fe170070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 10 monclient(hunting): handle_auth_done global_id 104156 payload 824 2024-09-16T05:22:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 1 -- 172.21.15.114:0/1504588608 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe170134f40 msgr2=0x7fe170070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 1 --2- 172.21.15.114:0/1504588608 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe170134f40 0x7fe170070e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 1 -- 172.21.15.114:0/1504588608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe170131e00 msgr2=0x7fe1700703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 1 --2- 172.21.15.114:0/1504588608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe170131e00 0x7fe1700703a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:47.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:22:47.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 1 monclient: found mon.noname-c 2024-09-16T05:22:47.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:47.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:22:47.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 1 -- 172.21.15.114:0/1504588608 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe170072830 con 0x7fe1701321e0 2024-09-16T05:22:47.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16ffff640 1 --2- 172.21.15.114:0/1504588608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe170131e00 0x7fe1700703a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:22:47.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16f7fe640 1 --2- 172.21.15.114:0/1504588608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1701321e0 0x7fe1700708e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe16400e9b0 tx=0x7fe16400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:47.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16cff9640 10 client.?.objecter ms_handle_connect 0x7fe1701321e0 2024-09-16T05:22:47.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:22:47.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16cff9640 1 -- 172.21.15.114:0/1504588608 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164017070 con 0x7fe1701321e0 2024-09-16T05:22:47.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:47.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:22:47.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16cff9640 10 monclient: dump: 2024-09-16T05:22:47.854 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:47.854 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:47.854 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:47.854 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:47.854 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:47.854 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:47.854 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:47.854 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:47.854 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:47.854 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:47.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16cff9640 10 monclient: _finish_auth 0 2024-09-16T05:22:47.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16cff9640 10 monclient: _check_auth_tickets 2024-09-16T05:22:47.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:47.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16cff9640 1 -- 172.21.15.114:0/1504588608 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16400c9a0 con 0x7fe1701321e0 2024-09-16T05:22:47.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 5 monclient: authenticate success, global_id 104156 2024-09-16T05:22:47.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient: _renew_subs 2024-09-16T05:22:47.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:47.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 -- 172.21.15.114:0/1504588608 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe17013d050 con 0x7fe1701321e0 2024-09-16T05:22:47.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:47.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16cff9640 1 -- 172.21.15.114:0/1504588608 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16400cd70 con 0x7fe1701321e0 2024-09-16T05:22:47.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:47.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:22:47.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16cff9640 10 monclient: dump: 2024-09-16T05:22:47.855 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:47.855 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:47.855 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:47.855 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:47.855 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:47.855 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:47.855 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:47.856 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:47.856 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:47.856 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:47.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 client.104156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:22:47.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient: _renew_subs 2024-09-16T05:22:47.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe16e7fc640 4 set_mon_vals no callback set 2024-09-16T05:22:47.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:47.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 -- 172.21.15.114:0/1504588608 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe170071c90 con 0x7fe1701321e0 2024-09-16T05:22:47.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.837+0000 7fe17650d640 1 librados: init done 2024-09-16T05:22:47.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.841+0000 7fe17650d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:47.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.841+0000 7fe17650d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:47.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.841+0000 7fe17650d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:47.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.841+0000 7fe17650d640 1 -- 172.21.15.114:0/1504588608 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1700a1540 con 0x7fe1701321e0 2024-09-16T05:22:47.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.841+0000 7fe16cff9640 1 -- 172.21.15.114:0/1504588608 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe16401a070 con 0x7fe1701321e0 2024-09-16T05:22:47.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.841+0000 7fe16cff9640 1 --2- 172.21.15.114:0/1504588608 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe15c037810 0x7fe15c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:47.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.841+0000 7fe16cff9640 1 -- 172.21.15.114:0/1504588608 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(615..615 src has 1..615) ==== 5592+0+0 (secure 0 0 0) 0x7fe164048d80 con 0x7fe1701321e0 2024-09-16T05:22:47.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.841+0000 7fe16cff9640 10 client.104156.objecter ms_dispatch 0x7fe170000ba0 osd_map(615..615 src has 1..615) 2024-09-16T05:22:47.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.841+0000 7fe16cff9640 3 client.104156.objecter handle_osd_map got epochs [615,615] > 0 2024-09-16T05:22:47.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.841+0000 7fe16cff9640 3 client.104156.objecter handle_osd_map decoding full epoch 615 2024-09-16T05:22:47.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.841+0000 7fe16cff9640 20 client.104156.objecter dump_active .. 0 homeless 2024-09-16T05:22:47.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.841+0000 7fe16ffff640 1 --2- 172.21.15.114:0/1504588608 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe15c037810 0x7fe15c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:47.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.841+0000 7fe16ffff640 10 monclient: get_auth_request con 0x7fe15c037810 auth_method 0 2024-09-16T05:22:47.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.841+0000 7fe16cff9640 10 client.104156.objecter ms_handle_connect 0x7fe15c037810 2024-09-16T05:22:47.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.841+0000 7fe16ffff640 1 --2- 172.21.15.114:0/1504588608 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe15c037810 0x7fe15c039cd0 secure :-1 s=READY pgs=3581 cs=0 l=1 rev1=1 crypto rx=0x7fe1600097c0 tx=0x7fe160008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:47.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.845+0000 7fe16cff9640 1 -- 172.21.15.114:0/1504588608 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fe16404c030 con 0x7fe1701321e0 2024-09-16T05:22:47.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.845+0000 7fe16cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:47.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:47.845+0000 7fe16cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:22:48.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.073+0000 7fe17650d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:22:48.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.073+0000 7fe17650d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:22:48.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.073+0000 7fe17650d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:48.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.073+0000 7fe17650d640 1 -- 172.21.15.114:0/1504588608 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe1700686f0 con 0x7fe1701321e0 2024-09-16T05:22:48.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.073+0000 7fe16cff9640 1 -- 172.21.15.114:0/1504588608 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fe164011350 con 0x7fe1701321e0 2024-09-16T05:22:48.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.073+0000 7fe16cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:22:48.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.073+0000 7fe16cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:22:48.073 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:22:48.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 10 librados: watch_flush enter 2024-09-16T05:22:48.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 10 librados: watch_flush exit 2024-09-16T05:22:48.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 20 client.104156.objecter shutdown clearing up homeless session... 2024-09-16T05:22:48.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 10 client.104156.objecter successfully canceled tick 2024-09-16T05:22:48.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 1 -- 172.21.15.114:0/1504588608 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe15c037810 msgr2=0x7fe15c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 1 --2- 172.21.15.114:0/1504588608 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe15c037810 0x7fe15c039cd0 secure :-1 s=READY pgs=3581 cs=0 l=1 rev1=1 crypto rx=0x7fe1600097c0 tx=0x7fe160008040 comp rx=0 tx=0).stop 2024-09-16T05:22:48.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 10 monclient: shutdown 2024-09-16T05:22:48.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:48.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 1 -- 172.21.15.114:0/1504588608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1701321e0 msgr2=0x7fe1700708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 1 --2- 172.21.15.114:0/1504588608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1701321e0 0x7fe1700708e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe16400e9b0 tx=0x7fe16400ee70 comp rx=0 tx=0).stop 2024-09-16T05:22:48.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 1 -- 172.21.15.114:0/1504588608 shutdown_connections 2024-09-16T05:22:48.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 1 --2- 172.21.15.114:0/1504588608 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe15c037810 0x7fe15c039cd0 unknown :-1 s=CLOSED pgs=3581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 1 --2- 172.21.15.114:0/1504588608 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe170134f40 0x7fe170070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 1 --2- 172.21.15.114:0/1504588608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe1701321e0 0x7fe1700708e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 1 --2- 172.21.15.114:0/1504588608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe170131e00 0x7fe1700703a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 1 -- 172.21.15.114:0/1504588608 >> 172.21.15.114:0/1504588608 conn(0x7fe1700a4280 msgr2=0x7fe1700a62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:48.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 1 -- 172.21.15.114:0/1504588608 shutdown_connections 2024-09-16T05:22:48.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 1 -- 172.21.15.114:0/1504588608 wait complete. 2024-09-16T05:22:48.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.077+0000 7fe17650d640 1 librados: shutdown 2024-09-16T05:22:48.090 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:22:48.091 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:22:48.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:22:48.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 10 monclient: get_monmap_and_config 2024-09-16T05:22:48.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 10 monclient: build_initial_monmap 2024-09-16T05:22:48.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 10 monclient: monmap: 2024-09-16T05:22:48.211 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:48.211 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.211 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:22:48.217076+0000 2024-09-16T05:22:48.212 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:22:48.217076+0000 2024-09-16T05:22:48.212 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:48.212 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.212 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:22:48.212 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:22:48.212 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:22:48.212 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:22:48.212 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:22:48.212 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 1 Processor -- start 2024-09-16T05:22:48.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 1 -- start start 2024-09-16T05:22:48.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 10 monclient: init 2024-09-16T05:22:48.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:48.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-16T05:22:48.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f00c41376e0 0x7f00c4137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 10 monclient(hunting): picked mon.noname-a con 0x7f00c41376e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:22:48.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00c41370a0 0x7f00c40a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 10 monclient(hunting): picked mon.noname-b con 0x7f00c41370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:48.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f00c40a5d00 addr v1:172.21.15.114:6789/0 2024-09-16T05:22:48.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00c409bd20 con 0x7f00c40a5d00 2024-09-16T05:22:48.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f00c4138000 con 0x7f00c41376e0 2024-09-16T05:22:48.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f00c409c5c0 con 0x7f00c41370a0 2024-09-16T05:22:48.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:48.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00ca45c640 10 monclient(hunting): authenticate will time out at 1852.846630s 2024-09-16T05:22:48.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c37fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00c41370a0 0x7f00c40a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f00c41376e0 0x7f00c4137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c89f9640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f00c40a5d00 0x7f00c40a47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34654/0 (socket says 172.21.15.114:34654) 2024-09-16T05:22:48.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f00c41376e0 0x7f00c4137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33740/0 (socket says 172.21.15.114:33740) 2024-09-16T05:22:48.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c89f9640 1 -- 172.21.15.114:0/1781854836 learned_addr learned my addr 172.21.15.114:0/1781854836 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:22:48.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 10 monclient(hunting): get_auth_request con 0x7f00c41376e0 auth_method 0 2024-09-16T05:22:48.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c37fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00c41370a0 0x7f00c40a4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:36392/0 (socket says 172.21.15.114:36392) 2024-09-16T05:22:48.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c37fe640 10 monclient(hunting): get_auth_request con 0x7f00c41370a0 auth_method 0 2024-09-16T05:22:48.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 10 monclient(hunting): handle_auth_done global_id 74640 payload 274 2024-09-16T05:22:48.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 1 -- 172.21.15.114:0/1781854836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00c41370a0 msgr2=0x7f00c40a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:22:48.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 1 --2- 172.21.15.114:0/1781854836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00c41370a0 0x7f00c40a4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 1 -- 172.21.15.114:0/1781854836 >> v1:172.21.15.114:6789/0 conn(0x7f00c40a5d00 legacy=0x7f00c40a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 10 monclient: _finish_hunting 0 2024-09-16T05:22:48.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 1 monclient: found mon.noname-a 2024-09-16T05:22:48.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:48.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c2ffd640 1 -- 172.21.15.114:0/1781854836 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3077754240 0 0) 0x7f00c409bd20 con 0x7f00c40a5d00 2024-09-16T05:22:48.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:22:48.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c3fff640 1 -- 172.21.15.114:0/1781854836 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c409c000 con 0x7f00c41376e0 2024-09-16T05:22:48.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c2ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:22:48.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.213+0000 7f00c37fe640 1 --2- 172.21.15.114:0/1781854836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00c41370a0 0x7f00c40a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:22:48.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c3fff640 1 --2- 172.21.15.114:0/1781854836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c41376e0 0x7f00c4137ac0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f00b40097c0 tx=0x7f00b401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bdb6deff6b047a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c2ffd640 1 -- 172.21.15.114:0/1781854836 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b40b0070 con 0x7f00c41376e0 2024-09-16T05:22:48.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:22:48.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c2ffd640 10 monclient: dump: 2024-09-16T05:22:48.225 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.225 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.226 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.226 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.226 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.226 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.226 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.226 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.226 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.226 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:22:48.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:22:48.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:48.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 5 monclient: authenticate success, global_id 74640 2024-09-16T05:22:48.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:22:48.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c2ffd640 1 -- 172.21.15.114:0/1781854836 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00b40ab600 con 0x7f00c41376e0 2024-09-16T05:22:48.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:48.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 10 monclient: get_monmap_and_config success 2024-09-16T05:22:48.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 4 set_mon_vals no callback set 2024-09-16T05:22:48.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c2ffd640 1 -- 172.21.15.114:0/1781854836 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b40ab9d0 con 0x7f00c41376e0 2024-09-16T05:22:48.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:48.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c2ffd640 10 monclient: dump: 2024-09-16T05:22:48.227 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.227 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.227 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.227 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.227 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.227 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.228 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.228 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.228 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.228 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 10 monclient: shutdown 2024-09-16T05:22:48.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:48.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 -- 172.21.15.114:0/1781854836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c41376e0 msgr2=0x7f00c4137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 --2- 172.21.15.114:0/1781854836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c41376e0 0x7f00c4137ac0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f00b40097c0 tx=0x7f00b401c360 comp rx=0 tx=0).stop 2024-09-16T05:22:48.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 -- 172.21.15.114:0/1781854836 shutdown_connections 2024-09-16T05:22:48.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 --2- 172.21.15.114:0/1781854836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00c41370a0 0x7f00c40a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 --2- 172.21.15.114:0/1781854836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c41376e0 0x7f00c4137ac0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 -- 172.21.15.114:0/1781854836 >> 172.21.15.114:0/1781854836 conn(0x7f00c40a9a40 msgr2=0x7f00c40a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:48.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 -- 172.21.15.114:0/1781854836 shutdown_connections 2024-09-16T05:22:48.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 -- 172.21.15.114:0/1781854836 wait complete. 2024-09-16T05:22:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 10 monclient: build_initial_monmap 2024-09-16T05:22:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 10 monclient: monmap: 2024-09-16T05:22:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:22:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:22:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:22:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:22:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:22:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:22:48.229 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 librados: starting msgr at 2024-09-16T05:22:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 librados: starting objecter 2024-09-16T05:22:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 Processor -- start 2024-09-16T05:22:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 -- start start 2024-09-16T05:22:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 librados: setting wanted keys 2024-09-16T05:22:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 librados: calling monclient init 2024-09-16T05:22:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 10 monclient: init 2024-09-16T05:22:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:22:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c40a5d00 0x7f00c41358a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 10 monclient(hunting): picked mon.noname-a con 0x7f00c40a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:22:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f00c41370a0 0x7f00c4135de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 10 monclient(hunting): picked mon.noname-b con 0x7f00c41370a0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:22:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00c41376e0 0x7f00c413bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 10 monclient(hunting): picked mon.noname-c con 0x7f00c41376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:48.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4141d80 con 0x7f00c40a5d00 2024-09-16T05:22:48.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4141c00 con 0x7f00c41370a0 2024-09-16T05:22:48.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f00c4141f00 con 0x7f00c41376e0 2024-09-16T05:22:48.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:48.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00ca45c640 10 monclient(hunting): authenticate will time out at 1852.851591s 2024-09-16T05:22:48.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c40a5d00 0x7f00c41358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c89f9640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00c41376e0 0x7f00c413bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c40a5d00 0x7f00c41358a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33754/0 (socket says 172.21.15.114:33754) 2024-09-16T05:22:48.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c89f9640 10 monclient(hunting): get_auth_request con 0x7f00c41376e0 auth_method 0 2024-09-16T05:22:48.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c89f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c3fff640 1 -- 172.21.15.114:0/3121851298 learned_addr learned my addr 172.21.15.114:0/3121851298 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:48.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c89f9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c89f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c3fff640 10 monclient(hunting): get_auth_request con 0x7f00c40a5d00 auth_method 0 2024-09-16T05:22:48.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c89f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c89f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c89f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c89f9640 10 monclient(hunting): handle_auth_done global_id 104162 payload 824 2024-09-16T05:22:48.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.217+0000 7f00c37fe640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f00c41370a0 0x7f00c4135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c89f9640 1 -- 172.21.15.114:0/3121851298 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f00c41370a0 msgr2=0x7f00c4135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c89f9640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f00c41370a0 0x7f00c4135de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c89f9640 1 -- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c40a5d00 msgr2=0x7f00c41358a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c89f9640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c40a5d00 0x7f00c41358a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c89f9640 10 monclient: _finish_hunting 0 2024-09-16T05:22:48.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c89f9640 1 monclient: found mon.noname-c 2024-09-16T05:22:48.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c89f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:48.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c89f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:22:48.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c89f9640 1 -- 172.21.15.114:0/3121851298 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c413e570 con 0x7f00c41376e0 2024-09-16T05:22:48.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c37fe640 10 monclient: get_auth_request con 0x7f00c41370a0 auth_method 0 2024-09-16T05:22:48.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c37fe640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f00c41370a0 0x7f00c4135de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:22:48.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c3fff640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c40a5d00 0x7f00c41358a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:22:48.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c89f9640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00c41376e0 0x7f00c413bd70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f00bc00c6e0 tx=0x7f00bc00cba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 client.?.objecter ms_handle_connect 0x7f00c41376e0 2024-09-16T05:22:48.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:22:48.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 1 -- 172.21.15.114:0/3121851298 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc018070 con 0x7f00c41376e0 2024-09-16T05:22:48.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:22:48.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 monclient: dump: 2024-09-16T05:22:48.234 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.234 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.234 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.234 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.234 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.234 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.235 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.235 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.235 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.235 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 monclient: _finish_auth 0 2024-09-16T05:22:48.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:22:48.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:48.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 1 -- 172.21.15.114:0/3121851298 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00bc004d90 con 0x7f00c41376e0 2024-09-16T05:22:48.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00ca45c640 5 monclient: authenticate success, global_id 104162 2024-09-16T05:22:48.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00ca45c640 10 monclient: _renew_subs 2024-09-16T05:22:48.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00ca45c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:48.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00ca45c640 1 -- 172.21.15.114:0/3121851298 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c4120520 con 0x7f00c41376e0 2024-09-16T05:22:48.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:48.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 1 -- 172.21.15.114:0/3121851298 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc00a880 con 0x7f00c41376e0 2024-09-16T05:22:48.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:22:48.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 monclient: dump: 2024-09-16T05:22:48.236 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.236 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.236 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.236 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.236 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.236 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.236 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.236 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.236 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.236 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c27fc640 4 set_mon_vals no callback set 2024-09-16T05:22:48.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00ca45c640 10 client.104162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:22:48.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00ca45c640 10 monclient: _renew_subs 2024-09-16T05:22:48.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00ca45c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:48.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00ca45c640 1 -- 172.21.15.114:0/3121851298 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c4120240 con 0x7f00c41376e0 2024-09-16T05:22:48.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00ca45c640 1 librados: init done 2024-09-16T05:22:48.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00ca45c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:48.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00ca45c640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c413c2b0 0x7f00c413c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00ca45c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00ca45c640 1 -- 172.21.15.114:0/3121851298 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4137ac0 con 0x7f00c413c2b0 2024-09-16T05:22:48.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 1 -- 172.21.15.114:0/3121851298 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f00bc01b070 con 0x7f00c41376e0 2024-09-16T05:22:48.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c3fff640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c413c2b0 0x7f00c413c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c3fff640 10 monclient: get_auth_request con 0x7f00c413c2b0 auth_method 0 2024-09-16T05:22:48.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 1 --2- 172.21.15.114:0/3121851298 >> v2:172.21.15.149:6812/1185531135 conn(0x7f00b0037810 0x7f00b0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 1 -- 172.21.15.114:0/3121851298 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(615..615 src has 1..615) ==== 5592+0+0 (secure 0 0 0) 0x7f00bc0483d0 con 0x7f00c41376e0 2024-09-16T05:22:48.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 client.104162.objecter ms_dispatch 0x7f00c4000ba0 osd_map(615..615 src has 1..615) 2024-09-16T05:22:48.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 3 client.104162.objecter handle_osd_map got epochs [615,615] > 0 2024-09-16T05:22:48.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 3 client.104162.objecter handle_osd_map decoding full epoch 615 2024-09-16T05:22:48.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c3fff640 10 monclient(hunting): handle_auth_done global_id 74649 payload 824 2024-09-16T05:22:48.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c3fff640 1 -- 172.21.15.114:0/3121851298 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f00c413e2d0 con 0x7f00c413c2b0 2024-09-16T05:22:48.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 20 client.104162.objecter dump_active .. 0 homeless 2024-09-16T05:22:48.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 client.104162.objecter ms_handle_connect 0x7f00c413c2b0 2024-09-16T05:22:48.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 client.104162.objecter resend_mon_ops 2024-09-16T05:22:48.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c3fff640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c413c2b0 0x7f00c413c690 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f00b401aa10 tx=0x7f00b4019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c37fe640 1 --2- 172.21.15.114:0/3121851298 >> v2:172.21.15.149:6812/1185531135 conn(0x7f00b0037810 0x7f00b0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 1 -- 172.21.15.114:0/3121851298 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b40bb030 con 0x7f00c413c2b0 2024-09-16T05:22:48.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:48.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 monclient: dump: 2024-09-16T05:22:48.239 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.239 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.239 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.239 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.239 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.240 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.240 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.240 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.240 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.240 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c37fe640 10 monclient: get_auth_request con 0x7f00b0037810 auth_method 0 2024-09-16T05:22:48.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 1 -- 172.21.15.114:0/3121851298 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f00b40abcd0 con 0x7f00c413c2b0 2024-09-16T05:22:48.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:48.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:22:48.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 1 -- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c413c2b0 msgr2=0x7f00c413c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c413c2b0 0x7f00c413c690 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f00b401aa10 tx=0x7f00b4019100 comp rx=0 tx=0).stop 2024-09-16T05:22:48.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 1 -- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c413c2b0 msgr2=0x7f00c413c690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:22:48.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c413c2b0 0x7f00c413c690 secure :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f00b401aa10 tx=0x7f00b4019100 comp rx=0 tx=0).stop 2024-09-16T05:22:48.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c37fe640 1 --2- 172.21.15.114:0/3121851298 >> v2:172.21.15.149:6812/1185531135 conn(0x7f00b0037810 0x7f00b0039cd0 secure :-1 s=READY pgs=3582 cs=0 l=1 rev1=1 crypto rx=0x7f00c40a3b70 tx=0x7f00b8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.221+0000 7f00c0ff9640 10 client.104162.objecter ms_handle_connect 0x7f00b0037810 2024-09-16T05:22:48.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:22:48.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c413cbd0 0x7f00c4138190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 1 -- 172.21.15.114:0/3121851298 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4068820 con 0x7f00c413cbd0 2024-09-16T05:22:48.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c3fff640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c413cbd0 0x7f00c4138190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c3fff640 10 monclient: get_auth_request con 0x7f00c413cbd0 auth_method 0 2024-09-16T05:22:48.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c3fff640 10 monclient(hunting): handle_auth_done global_id 74652 payload 824 2024-09-16T05:22:48.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c3fff640 1 -- 172.21.15.114:0/3121851298 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f00c411ff60 con 0x7f00c413cbd0 2024-09-16T05:22:48.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c3fff640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c413cbd0 0x7f00c4138190 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f00b401c3a0 tx=0x7f00b4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c0ff9640 10 client.104162.objecter ms_handle_connect 0x7f00c413cbd0 2024-09-16T05:22:48.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c0ff9640 10 client.104162.objecter resend_mon_ops 2024-09-16T05:22:48.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c0ff9640 1 -- 172.21.15.114:0/3121851298 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b40bb070 con 0x7f00c413cbd0 2024-09-16T05:22:48.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:48.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c0ff9640 10 monclient: dump: 2024-09-16T05:22:48.244 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.244 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.244 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.244 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.244 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.244 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.244 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.245 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.245 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.245 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c0ff9640 1 -- 172.21.15.114:0/3121851298 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f00b4003e40 con 0x7f00c413cbd0 2024-09-16T05:22:48.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:22:48.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:22:48.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c0ff9640 1 -- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c413cbd0 msgr2=0x7f00c4138190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c0ff9640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c413cbd0 0x7f00c4138190 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f00b401c3a0 tx=0x7f00b4008040 comp rx=0 tx=0).stop 2024-09-16T05:22:48.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c0ff9640 1 -- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c413cbd0 msgr2=0x7f00c4138190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:22:48.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c0ff9640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c413cbd0 0x7f00c4138190 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.246 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:22:48.246 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:22:48.246 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:22:48.246 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:22:48.246 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 62, 2024-09-16T05:22:48.246 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 527284, 2024-09-16T05:22:48.246 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:22:48.246 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:22:48.246 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:22:48.246 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:22:48.246 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:48.246 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:22:48.247 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:22:48.247 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:22:48.247 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:22:48.247 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:48.247 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:48.247 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:48.247 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:48.247 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:48.247 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:48.247 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:48.247 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:48.247 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:48.247 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:48.248 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:48.248 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:48.248 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:22:48.248 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:22:48.248 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:48.248 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:48.248 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:48.248 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:48.248 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:48.248 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:48.248 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:48.248 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:48.248 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:48.249 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:48.249 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:48.249 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.249 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:22:48.249 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:22:48.249 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:22:48.249 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:22:48.249 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:22:48.249 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:22:48.249 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:22:48.249 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:22:48.249 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:22:48.249 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:22:48.250 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:22:48.250 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:22:48.250 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:22:48.250 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:22:48.250 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:22:48.250 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:22:48.250 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:22:48.250 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:48.250 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:48.250 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:48.250 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:48.250 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:48.250 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:48.250 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:48.251 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:48.251 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:48.251 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:48.251 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:48.251 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:48.251 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:22:48.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.251 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:22:48.251 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.251 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:22:48.251 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:22:48.251 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:48.251 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:48.251 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.252 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:48.252 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:22:48.252 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.252 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.252 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.252 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:48.252 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:22:48.252 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.252 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.252 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.252 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.252 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:22:48.252 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:22:48.253 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:48.253 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:48.253 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:48.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.253 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.253 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:22:48.253 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:22:48.253 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:48.253 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:48.253 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.253 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:48.253 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:22:48.253 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.254 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:48.254 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:22:48.254 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.254 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.254 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.254 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:22:48.254 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:22:48.254 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:48.254 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:48.254 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:48.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.255 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:22:48.255 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:22:48.255 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:48.255 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:48.255 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.255 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:48.255 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:22:48.255 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.255 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.255 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.255 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:48.255 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:22:48.255 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.255 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.256 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.256 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.256 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:22:48.256 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:22:48.256 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:48.256 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:48.256 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:48.256 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.256 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.256 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.256 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:22:48.256 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:22:48.256 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.256 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:48.257 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:48.257 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:22:48.257 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.257 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:48.257 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:22:48.257 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.257 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:48.257 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:48.257 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:22:48.257 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.257 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:48.257 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:22:48.257 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.257 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:48.258 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:48.258 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:22:48.258 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.258 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.258 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.258 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:22:48.258 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:22:48.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 10 librados: watch_flush enter 2024-09-16T05:22:48.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 10 librados: watch_flush exit 2024-09-16T05:22:48.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 20 client.104162.objecter shutdown clearing up homeless session... 2024-09-16T05:22:48.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 10 client.104162.objecter successfully canceled tick 2024-09-16T05:22:48.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 1 -- 172.21.15.114:0/3121851298 >> v2:172.21.15.149:6812/1185531135 conn(0x7f00b0037810 msgr2=0x7f00b0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 1 --2- 172.21.15.114:0/3121851298 >> v2:172.21.15.149:6812/1185531135 conn(0x7f00b0037810 0x7f00b0039cd0 secure :-1 s=READY pgs=3582 cs=0 l=1 rev1=1 crypto rx=0x7f00c40a3b70 tx=0x7f00b8008040 comp rx=0 tx=0).stop 2024-09-16T05:22:48.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 10 monclient: shutdown 2024-09-16T05:22:48.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:48.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 1 -- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00c41376e0 msgr2=0x7f00c413bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00c41376e0 0x7f00c413bd70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f00bc00c6e0 tx=0x7f00bc00cba0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00c89f9640 1 -- 172.21.15.114:0/3121851298 reap_dead start 2024-09-16T05:22:48.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 1 -- 172.21.15.114:0/3121851298 shutdown_connections 2024-09-16T05:22:48.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 1 --2- 172.21.15.114:0/3121851298 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00c41376e0 0x7f00c413bd70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 1 -- 172.21.15.114:0/3121851298 >> 172.21.15.114:0/3121851298 conn(0x7f00c40a9a40 msgr2=0x7f00c40a7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:48.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 1 -- 172.21.15.114:0/3121851298 shutdown_connections 2024-09-16T05:22:48.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 1 -- 172.21.15.114:0/3121851298 wait complete. 2024-09-16T05:22:48.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.229+0000 7f00ca45c640 1 librados: shutdown 2024-09-16T05:22:48.261 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:22:48.261 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:22:48.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:22:48.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.377+0000 7f198348d640 10 monclient: get_monmap_and_config 2024-09-16T05:22:48.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.377+0000 7f198348d640 10 monclient: build_initial_monmap 2024-09-16T05:22:48.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.377+0000 7f198348d640 10 monclient: monmap: 2024-09-16T05:22:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:22:48.384225+0000 2024-09-16T05:22:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:22:48.384225+0000 2024-09-16T05:22:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:22:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:22:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:22:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:22:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:22:48.379 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 1 Processor -- start 2024-09-16T05:22:48.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 1 -- start start 2024-09-16T05:22:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 10 monclient: init 2024-09-16T05:22:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-16T05:22:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c137730 0x7f197c137b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 10 monclient(hunting): picked mon.noname-b con 0x7f197c137730 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f197c0a5d00 0x7f197c0a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 10 monclient(hunting): picked mon.noname-a con 0x7f197c0a5d00 addr v2:172.21.15.114:3300/0 2024-09-16T05:22:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f197c0a47b0 addr v1:172.21.15.114:6789/0 2024-09-16T05:22:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f197c0a55e0 con 0x7f197c0a47b0 2024-09-16T05:22:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f197c09c450 con 0x7f197c0a5d00 2024-09-16T05:22:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f197c0a3710 con 0x7f197c137730 2024-09-16T05:22:48.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:48.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f198348d640 10 monclient(hunting): authenticate will time out at 1853.013408s 2024-09-16T05:22:48.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c137730 0x7f197c137b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c137730 0x7f197c137b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:36406/0 (socket says 172.21.15.114:36406) 2024-09-16T05:22:48.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 1 -- 172.21.15.114:0/2138158463 learned_addr learned my addr 172.21.15.114:0/2138158463 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:48.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 10 monclient(hunting): get_auth_request con 0x7f197c137730 auth_method 0 2024-09-16T05:22:48.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1980a28640 1 --2- 172.21.15.114:0/2138158463 >> v2:172.21.15.114:3300/0 conn(0x7f197c0a5d00 0x7f197c0a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1980a28640 10 monclient(hunting): get_auth_request con 0x7f197c0a5d00 auth_method 0 2024-09-16T05:22:48.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1980a28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1980a28640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1980a28640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1980a28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1980a28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1980a28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 10 monclient(hunting): handle_auth_done global_id 104165 payload 274 2024-09-16T05:22:48.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 1 -- 172.21.15.114:0/2138158463 >> v2:172.21.15.114:3300/0 conn(0x7f197c0a5d00 msgr2=0x7f197c0a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:22:48.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 1 --2- 172.21.15.114:0/2138158463 >> v2:172.21.15.114:3300/0 conn(0x7f197c0a5d00 0x7f197c0a4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 1 -- 172.21.15.114:0/2138158463 >> v1:172.21.15.114:6789/0 conn(0x7f197c0a47b0 legacy=0x7f197c13e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 10 monclient: _finish_hunting 0 2024-09-16T05:22:48.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 1 monclient: found mon.noname-b 2024-09-16T05:22:48.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:48.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:22:48.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.381+0000 7f1981229640 1 -- 172.21.15.114:0/2138158463 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197c09be00 con 0x7f197c137730 2024-09-16T05:22:48.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981229640 1 --2- 172.21.15.114:0/2138158463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c137730 0x7f197c137b10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1978009870 tx=0x7f197801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dbecbe63b7e444b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1973fff640 1 -- 172.21.15.114:0/2138158463 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19780b0070 con 0x7f197c137730 2024-09-16T05:22:48.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1973fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1973fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:22:48.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1973fff640 10 monclient: dump: 2024-09-16T05:22:48.392 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.392 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.392 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.392 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.392 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.392 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.392 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.392 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.392 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.393 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1973fff640 10 monclient: _finish_auth 0 2024-09-16T05:22:48.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1973fff640 10 monclient: _check_auth_tickets 2024-09-16T05:22:48.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1973fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:48.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1973fff640 1 -- 172.21.15.114:0/2138158463 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19780ab600 con 0x7f197c137730 2024-09-16T05:22:48.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 5 monclient: authenticate success, global_id 104165 2024-09-16T05:22:48.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:22:48.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1973fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:48.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1973fff640 1 -- 172.21.15.114:0/2138158463 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19780ab9d0 con 0x7f197c137730 2024-09-16T05:22:48.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1973fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1973fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:22:48.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1973fff640 10 monclient: dump: 2024-09-16T05:22:48.394 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.394 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.394 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.394 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.394 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.394 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.394 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.394 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.394 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.394 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient: get_monmap_and_config success 2024-09-16T05:22:48.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 4 set_mon_vals no callback set 2024-09-16T05:22:48.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient: shutdown 2024-09-16T05:22:48.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:48.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 -- 172.21.15.114:0/2138158463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c137730 msgr2=0x7f197c137b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 --2- 172.21.15.114:0/2138158463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c137730 0x7f197c137b10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1978009870 tx=0x7f197801c480 comp rx=0 tx=0).stop 2024-09-16T05:22:48.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 -- 172.21.15.114:0/2138158463 shutdown_connections 2024-09-16T05:22:48.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 --2- 172.21.15.114:0/2138158463 >> v2:172.21.15.114:3300/0 conn(0x7f197c0a5d00 0x7f197c0a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 --2- 172.21.15.114:0/2138158463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c137730 0x7f197c137b10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 -- 172.21.15.114:0/2138158463 >> 172.21.15.114:0/2138158463 conn(0x7f197c0a9c70 msgr2=0x7f197c0aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:48.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 -- 172.21.15.114:0/2138158463 shutdown_connections 2024-09-16T05:22:48.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 -- 172.21.15.114:0/2138158463 wait complete. 2024-09-16T05:22:48.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient: build_initial_monmap 2024-09-16T05:22:48.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient: monmap: 2024-09-16T05:22:48.395 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:48.395 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:22:48.396 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:22:48.396 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:22:48.396 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:48.396 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.396 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:22:48.396 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:22:48.396 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:22:48.396 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 librados: starting msgr at 2024-09-16T05:22:48.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 librados: starting objecter 2024-09-16T05:22:48.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 Processor -- start 2024-09-16T05:22:48.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 -- start start 2024-09-16T05:22:48.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 librados: setting wanted keys 2024-09-16T05:22:48.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 librados: calling monclient init 2024-09-16T05:22:48.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient: init 2024-09-16T05:22:48.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:48.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:22:48.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f197c0a47b0 0x7f197c1cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient(hunting): picked mon.noname-a con 0x7f197c0a47b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:22:48.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f197c0a5d00 0x7f197c13e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient(hunting): picked mon.noname-b con 0x7f197c0a5d00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:22:48.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c137730 0x7f197c13eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient(hunting): picked mon.noname-c con 0x7f197c137730 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:48.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f197c072a20 con 0x7f197c0a47b0 2024-09-16T05:22:48.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f197c0728a0 con 0x7f197c0a5d00 2024-09-16T05:22:48.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f197c1371a0 con 0x7f197c137730 2024-09-16T05:22:48.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:48.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient(hunting): authenticate will time out at 1853.018342s 2024-09-16T05:22:48.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981229640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f197c0a47b0 0x7f197c1cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c137730 0x7f197c13eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981229640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f197c0a47b0 0x7f197c1cf7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33798/0 (socket says 172.21.15.114:33798) 2024-09-16T05:22:48.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981229640 1 -- 172.21.15.114:0/155340521 learned_addr learned my addr 172.21.15.114:0/155340521 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:48.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981229640 10 monclient(hunting): get_auth_request con 0x7f197c0a47b0 auth_method 0 2024-09-16T05:22:48.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981229640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981229640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981229640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 10 monclient(hunting): get_auth_request con 0x7f197c137730 auth_method 0 2024-09-16T05:22:48.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981229640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981229640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981229640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 10 monclient(hunting): handle_auth_done global_id 104168 payload 824 2024-09-16T05:22:48.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1980a28640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f197c0a5d00 0x7f197c13e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 1 -- 172.21.15.114:0/155340521 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f197c0a5d00 msgr2=0x7f197c13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f197c0a5d00 0x7f197c13e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 1 -- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f197c0a47b0 msgr2=0x7f197c1cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f197c0a47b0 0x7f197c1cf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 10 monclient: _finish_hunting 0 2024-09-16T05:22:48.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 1 monclient: found mon.noname-c 2024-09-16T05:22:48.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:48.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:22:48.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 1 -- 172.21.15.114:0/155340521 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197c073840 con 0x7f197c137730 2024-09-16T05:22:48.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981229640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f197c0a47b0 0x7f197c1cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:22:48.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1981a2a640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c137730 0x7f197c13eb50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f197400c730 tx=0x7f197400cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 10 client.?.objecter ms_handle_connect 0x7f197c137730 2024-09-16T05:22:48.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:22:48.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 1 -- 172.21.15.114:0/155340521 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974016020 con 0x7f197c137730 2024-09-16T05:22:48.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:22:48.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 10 monclient: dump: 2024-09-16T05:22:48.401 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.401 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.401 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.401 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.401 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.401 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.401 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.401 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.401 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.401 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 10 monclient: _finish_auth 0 2024-09-16T05:22:48.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:22:48.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:48.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 1 -- 172.21.15.114:0/155340521 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1974006d60 con 0x7f197c137730 2024-09-16T05:22:48.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 5 monclient: authenticate success, global_id 104168 2024-09-16T05:22:48.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient: _renew_subs 2024-09-16T05:22:48.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:48.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 -- 172.21.15.114:0/155340521 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f197c13cb20 con 0x7f197c137730 2024-09-16T05:22:48.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:48.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 1 -- 172.21.15.114:0/155340521 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974003bc0 con 0x7f197c137730 2024-09-16T05:22:48.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:22:48.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 10 monclient: dump: 2024-09-16T05:22:48.402 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.402 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.402 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.403 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.403 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.403 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.403 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.403 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.403 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.403 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1972ffd640 4 set_mon_vals no callback set 2024-09-16T05:22:48.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 client.104168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:22:48.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient: _renew_subs 2024-09-16T05:22:48.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:48.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 -- 172.21.15.114:0/155340521 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f197c13f8b0 con 0x7f197c137730 2024-09-16T05:22:48.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f198348d640 1 librados: init done 2024-09-16T05:22:48.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.385+0000 7f1971ffb640 1 -- 172.21.15.114:0/155340521 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f19740050d0 con 0x7f197c137730 2024-09-16T05:22:48.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f198348d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:48.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f198348d640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c13cd60 0x7f197c139150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f198348d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f198348d640 1 -- 172.21.15.114:0/155340521 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f197c068f10 con 0x7f197c13cd60 2024-09-16T05:22:48.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 1 --2- 172.21.15.114:0/155340521 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1960037810 0x7f1960039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 1 -- 172.21.15.114:0/155340521 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(615..615 src has 1..615) ==== 5592+0+0 (secure 0 0 0) 0x7f1974048bd0 con 0x7f197c137730 2024-09-16T05:22:48.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 10 client.104168.objecter ms_dispatch 0x7f197c000ba0 osd_map(615..615 src has 1..615) 2024-09-16T05:22:48.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 3 client.104168.objecter handle_osd_map got epochs [615,615] > 0 2024-09-16T05:22:48.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 3 client.104168.objecter handle_osd_map decoding full epoch 615 2024-09-16T05:22:48.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1981229640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c13cd60 0x7f197c139150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1981229640 10 monclient: get_auth_request con 0x7f197c13cd60 auth_method 0 2024-09-16T05:22:48.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1981229640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1981229640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1981229640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 20 client.104168.objecter dump_active .. 0 homeless 2024-09-16T05:22:48.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1981229640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1981229640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1981229640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1981229640 10 monclient(hunting): handle_auth_done global_id 104171 payload 824 2024-09-16T05:22:48.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1981229640 1 -- 172.21.15.114:0/155340521 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f197c0a4270 con 0x7f197c13cd60 2024-09-16T05:22:48.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1980a28640 1 --2- 172.21.15.114:0/155340521 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1960037810 0x7f1960039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1980a28640 10 monclient: get_auth_request con 0x7f1960037810 auth_method 0 2024-09-16T05:22:48.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 10 client.104168.objecter ms_handle_connect 0x7f197c13cd60 2024-09-16T05:22:48.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 10 client.104168.objecter resend_mon_ops 2024-09-16T05:22:48.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1981229640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c13cd60 0x7f197c139150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f197801a0b0 tx=0x7f197801acc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 1 -- 172.21.15.114:0/155340521 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19780b0050 con 0x7f197c13cd60 2024-09-16T05:22:48.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:22:48.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 10 monclient: dump: 2024-09-16T05:22:48.406 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.406 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.406 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.406 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.406 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.406 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.406 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.406 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.406 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.406 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 1 -- 172.21.15.114:0/155340521 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f19780b1040 con 0x7f197c13cd60 2024-09-16T05:22:48.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:48.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:22:48.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 1 -- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c13cd60 msgr2=0x7f197c139150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c13cd60 0x7f197c139150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f197801a0b0 tx=0x7f197801acc0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 1 -- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c13cd60 msgr2=0x7f197c139150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:22:48.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c13cd60 0x7f197c139150 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f197801a0b0 tx=0x7f197801acc0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1971ffb640 10 client.104168.objecter ms_handle_connect 0x7f1960037810 2024-09-16T05:22:48.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.389+0000 7f1980a28640 1 --2- 172.21.15.114:0/155340521 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1960037810 0x7f1960039cd0 secure :-1 s=READY pgs=3583 cs=0 l=1 rev1=1 crypto rx=0x7f197c13a9f0 tx=0x7f196c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f198348d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:22:48.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f198348d640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c139690 0x7f197c139aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f198348d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f198348d640 1 -- 172.21.15.114:0/155340521 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f197c139fe0 con 0x7f197c139690 2024-09-16T05:22:48.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1981229640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c139690 0x7f197c139aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1981229640 10 monclient: get_auth_request con 0x7f197c139690 auth_method 0 2024-09-16T05:22:48.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1981229640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1981229640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1981229640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1981229640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1981229640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1981229640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1981229640 10 monclient(hunting): handle_auth_done global_id 104174 payload 824 2024-09-16T05:22:48.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1981229640 1 -- 172.21.15.114:0/155340521 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f197c13b6d0 con 0x7f197c139690 2024-09-16T05:22:48.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1981229640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c139690 0x7f197c139aa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1978009870 tx=0x7f1978019220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1971ffb640 10 client.104168.objecter ms_handle_connect 0x7f197c139690 2024-09-16T05:22:48.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1971ffb640 10 client.104168.objecter resend_mon_ops 2024-09-16T05:22:48.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1971ffb640 1 -- 172.21.15.114:0/155340521 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19780b7b90 con 0x7f197c139690 2024-09-16T05:22:48.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1971ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1971ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:22:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1971ffb640 10 monclient: dump: 2024-09-16T05:22:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.411 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1971ffb640 1 -- 172.21.15.114:0/155340521 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f19780bf720 con 0x7f197c139690 2024-09-16T05:22:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1971ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:22:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1971ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:22:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1971ffb640 1 -- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c139690 msgr2=0x7f197c139aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1971ffb640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c139690 0x7f197c139aa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1978009870 tx=0x7f1978019220 comp rx=0 tx=0).stop 2024-09-16T05:22:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1971ffb640 1 -- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c139690 msgr2=0x7f197c139aa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:22:48.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.393+0000 7f1971ffb640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c139690 0x7f197c139aa0 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1978009870 tx=0x7f1978019220 comp rx=0 tx=0).stop 2024-09-16T05:22:48.412 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:22:48.412 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:22:48.412 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:22:48.413 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:22:48.413 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 62, 2024-09-16T05:22:48.413 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 144557, 2024-09-16T05:22:48.413 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:22:48.413 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:22:48.413 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:22:48.413 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:22:48.413 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:48.413 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:22:48.413 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:22:48.413 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:22:48.413 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:22:48.413 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:48.414 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:48.414 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:48.414 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:48.414 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:48.414 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:48.414 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:48.414 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:48.414 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:48.414 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:48.414 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:48.414 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:48.414 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:22:48.414 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:22:48.414 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:48.415 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:48.415 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:48.415 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:48.415 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:48.415 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:48.415 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:48.415 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:48.415 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:48.415 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:48.415 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:48.415 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.415 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:22:48.416 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:22:48.416 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:22:48.416 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:22:48.416 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:22:48.416 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:22:48.416 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:22:48.416 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:22:48.416 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:22:48.416 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:22:48.416 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:22:48.416 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:22:48.416 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:22:48.416 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:22:48.416 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:22:48.417 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:22:48.417 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:22:48.417 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:48.417 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:48.417 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:48.417 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:48.417 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:48.417 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:48.417 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:48.417 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:48.417 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:48.417 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:48.417 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:48.418 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:48.418 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:22:48.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.418 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:22:48.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.418 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:22:48.418 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:22:48.418 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:48.418 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:48.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.418 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:48.418 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:22:48.418 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.419 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:48.419 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:22:48.419 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.419 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.419 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.419 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:22:48.419 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:22:48.419 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:48.419 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:48.419 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:48.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.420 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:22:48.420 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:22:48.420 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:48.420 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:48.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.420 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:48.420 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:22:48.420 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.420 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:48.421 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:22:48.421 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.421 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.421 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.421 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:22:48.421 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:22:48.421 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:48.421 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:48.421 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:48.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.421 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:22:48.422 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:22:48.422 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:48.422 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:48.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.422 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:48.422 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:22:48.422 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.422 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:48.422 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:22:48.422 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.422 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.423 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.423 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:22:48.423 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:22:48.423 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:48.423 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:48.423 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:48.423 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.423 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.423 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:22:48.423 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:22:48.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.423 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:48.424 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:48.424 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:22:48.424 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.424 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:48.424 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:22:48.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.424 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:48.424 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:48.424 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:22:48.424 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.424 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:48.425 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:22:48.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.425 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:48.425 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:48.425 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:22:48.425 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.425 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:48.425 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:22:48.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.425 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:48.425 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:48.425 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:22:48.425 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.425 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.426 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:22:48.426 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:22:48.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f198348d640 10 librados: watch_flush enter 2024-09-16T05:22:48.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f198348d640 10 librados: watch_flush exit 2024-09-16T05:22:48.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f198348d640 20 client.104168.objecter shutdown clearing up homeless session... 2024-09-16T05:22:48.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f198348d640 10 client.104168.objecter successfully canceled tick 2024-09-16T05:22:48.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f198348d640 1 -- 172.21.15.114:0/155340521 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1960037810 msgr2=0x7f1960039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f198348d640 1 --2- 172.21.15.114:0/155340521 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1960037810 0x7f1960039cd0 secure :-1 s=READY pgs=3583 cs=0 l=1 rev1=1 crypto rx=0x7f197c13a9f0 tx=0x7f196c008040 comp rx=0 tx=0).stop 2024-09-16T05:22:48.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f198348d640 10 monclient: shutdown 2024-09-16T05:22:48.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f198348d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:48.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f198348d640 1 -- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c137730 msgr2=0x7f197c13eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f198348d640 1 --2- 172.21.15.114:0/155340521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f197c137730 0x7f197c13eb50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f197400c730 tx=0x7f197400cbf0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f1981a2a640 1 -- 172.21.15.114:0/155340521 reap_dead start 2024-09-16T05:22:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f1981a2a640 1 -- 172.21.15.114:0/155340521 reap_dead start 2024-09-16T05:22:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f198348d640 1 -- 172.21.15.114:0/155340521 shutdown_connections 2024-09-16T05:22:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f198348d640 1 -- 172.21.15.114:0/155340521 >> 172.21.15.114:0/155340521 conn(0x7f197c0a9c70 msgr2=0x7f197c136220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f198348d640 1 -- 172.21.15.114:0/155340521 shutdown_connections 2024-09-16T05:22:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f198348d640 1 -- 172.21.15.114:0/155340521 wait complete. 2024-09-16T05:22:48.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.397+0000 7f198348d640 1 librados: shutdown 2024-09-16T05:22:48.429 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:22:48.429 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:22:48.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:22:48.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.541+0000 7f3c66aa0640 10 monclient: get_monmap_and_config 2024-09-16T05:22:48.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.541+0000 7f3c66aa0640 10 monclient: build_initial_monmap 2024-09-16T05:22:48.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.541+0000 7f3c66aa0640 10 monclient: monmap: 2024-09-16T05:22:48.553 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:48.553 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.553 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:22:48.546933+0000 2024-09-16T05:22:48.553 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:22:48.546933+0000 2024-09-16T05:22:48.553 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:48.553 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.553 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:22:48.553 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:22:48.554 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:22:48.554 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:22:48.554 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:22:48.554 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.541+0000 7f3c66aa0640 1 Processor -- start 2024-09-16T05:22:48.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.541+0000 7f3c66aa0640 1 -- start start 2024-09-16T05:22:48.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.541+0000 7f3c66aa0640 10 monclient: init 2024-09-16T05:22:48.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.541+0000 7f3c66aa0640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:48.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.541+0000 7f3c66aa0640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T05:22:48.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f3c6009b6c0 0x7f3c600a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 10 monclient(hunting): picked mon.noname-c con 0x7f3c6009b6c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:22:48.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3c600a1400 addr v1:172.21.15.114:6789/0 2024-09-16T05:22:48.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3c600a1810 0x7f3c6013cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 10 monclient(hunting): picked mon.noname-a con 0x7f3c600a1810 addr v2:172.21.15.114:3300/0 2024-09-16T05:22:48.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c6009d4e0 con 0x7f3c600a1400 2024-09-16T05:22:48.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f3c6009e390 con 0x7f3c600a1810 2024-09-16T05:22:48.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f3c6009ece0 con 0x7f3c6009b6c0 2024-09-16T05:22:48.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:48.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 10 monclient(hunting): authenticate will time out at 1853.176096s 2024-09-16T05:22:48.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3c600a1810 0x7f3c6013cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c5ffff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f3c600a1400 0x7f3c60139640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34672/0 (socket says 172.21.15.114:34672) 2024-09-16T05:22:48.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 10 monclient(hunting): get_auth_request con 0x7f3c600a1810 auth_method 0 2024-09-16T05:22:48.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c5ffff640 1 -- 172.21.15.114:0/206896464 learned_addr learned my addr 172.21.15.114:0/206896464 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:22:48.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6483c640 1 --2- 172.21.15.114:0/206896464 >> v2:172.21.15.149:3300/0 conn(0x7f3c6009b6c0 0x7f3c600a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6483c640 10 monclient(hunting): get_auth_request con 0x7f3c6009b6c0 auth_method 0 2024-09-16T05:22:48.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6483c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6483c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6483c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 10 monclient(hunting): handle_auth_done global_id 74661 payload 274 2024-09-16T05:22:48.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 1 -- 172.21.15.114:0/206896464 >> v2:172.21.15.149:3300/0 conn(0x7f3c6009b6c0 msgr2=0x7f3c600a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:22:48.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 1 --2- 172.21.15.114:0/206896464 >> v2:172.21.15.149:3300/0 conn(0x7f3c6009b6c0 0x7f3c600a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 1 -- 172.21.15.114:0/206896464 >> v1:172.21.15.114:6789/0 conn(0x7f3c600a1400 legacy=0x7f3c60139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 10 monclient: _finish_hunting 0 2024-09-16T05:22:48.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 1 monclient: found mon.noname-a 2024-09-16T05:22:48.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:48.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:22:48.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 1 -- 172.21.15.114:0/206896464 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6009f430 con 0x7f3c600a1810 2024-09-16T05:22:48.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c6503d640 1 --2- 172.21.15.114:0/206896464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c600a1810 0x7f3c6013cd70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3c600a2f70 tx=0x7f3c5801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5260a1eb443c2d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c5f7fe640 1 -- 172.21.15.114:0/206896464 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c580b2070 con 0x7f3c600a1810 2024-09-16T05:22:48.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c5f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:22:48.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c5f7fe640 10 monclient: dump: 2024-09-16T05:22:48.558 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.558 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.558 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.558 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.558 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.558 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.558 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.558 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.559 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.559 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c5f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:22:48.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c5f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:22:48.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:48.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c5f7fe640 1 -- 172.21.15.114:0/206896464 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c580a57a0 con 0x7f3c600a1810 2024-09-16T05:22:48.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 5 monclient: authenticate success, global_id 74661 2024-09-16T05:22:48.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:22:48.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c5f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:48.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 10 monclient: get_monmap_and_config success 2024-09-16T05:22:48.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c5f7fe640 1 -- 172.21.15.114:0/206896464 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c580a5b70 con 0x7f3c600a1810 2024-09-16T05:22:48.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 4 set_mon_vals no callback set 2024-09-16T05:22:48.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:48.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c5f7fe640 10 monclient: dump: 2024-09-16T05:22:48.560 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.560 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.560 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.560 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.560 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.560 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.560 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.560 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.560 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.560 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 10 monclient: shutdown 2024-09-16T05:22:48.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:48.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 1 -- 172.21.15.114:0/206896464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c600a1810 msgr2=0x7f3c6013cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 1 --2- 172.21.15.114:0/206896464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c600a1810 0x7f3c6013cd70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3c600a2f70 tx=0x7f3c5801c7f0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 1 -- 172.21.15.114:0/206896464 shutdown_connections 2024-09-16T05:22:48.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 1 --2- 172.21.15.114:0/206896464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c600a1810 0x7f3c6013cd70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 1 --2- 172.21.15.114:0/206896464 >> v2:172.21.15.149:3300/0 conn(0x7f3c6009b6c0 0x7f3c600a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 1 -- 172.21.15.114:0/206896464 >> 172.21.15.114:0/206896464 conn(0x7f3c600a1fb0 msgr2=0x7f3c600a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:48.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 1 -- 172.21.15.114:0/206896464 shutdown_connections 2024-09-16T05:22:48.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 1 -- 172.21.15.114:0/206896464 wait complete. 2024-09-16T05:22:48.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 10 monclient: build_initial_monmap 2024-09-16T05:22:48.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.545+0000 7f3c66aa0640 10 monclient: monmap: 2024-09-16T05:22:48.561 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:48.561 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:22:48.561 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:22:48.561 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:22:48.562 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:48.562 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.562 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:22:48.562 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:22:48.562 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:22:48.562 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 librados: starting msgr at 2024-09-16T05:22:48.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 librados: starting objecter 2024-09-16T05:22:48.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 Processor -- start 2024-09-16T05:22:48.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 -- start start 2024-09-16T05:22:48.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 librados: setting wanted keys 2024-09-16T05:22:48.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 librados: calling monclient init 2024-09-16T05:22:48.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient: init 2024-09-16T05:22:48.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:48.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:22:48.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c6009b6c0 0x7f3c6013cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient(hunting): picked mon.noname-c con 0x7f3c6009b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:48.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c600a1400 0x7f3c600994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient(hunting): picked mon.noname-b con 0x7f3c600a1400 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:22:48.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c600a1810 0x7f3c60099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient(hunting): picked mon.noname-a con 0x7f3c600a1810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:22:48.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6009cad0 con 0x7f3c600a1810 2024-09-16T05:22:48.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6009db20 con 0x7f3c600a1400 2024-09-16T05:22:48.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f3c6013fad0 con 0x7f3c6009b6c0 2024-09-16T05:22:48.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:48.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient(hunting): authenticate will time out at 1853.180807s 2024-09-16T05:22:48.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c6009b6c0 0x7f3c6013cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6503d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c600a1810 0x7f3c60099a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c6009b6c0 0x7f3c6013cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:36440/0 (socket says 172.21.15.114:36440) 2024-09-16T05:22:48.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6503d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c600a1810 0x7f3c60099a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33816/0 (socket says 172.21.15.114:33816) 2024-09-16T05:22:48.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 1 -- 172.21.15.114:0/3931017027 learned_addr learned my addr 172.21.15.114:0/3931017027 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:48.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 10 monclient(hunting): get_auth_request con 0x7f3c6009b6c0 auth_method 0 2024-09-16T05:22:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6503d640 10 monclient(hunting): get_auth_request con 0x7f3c600a1810 auth_method 0 2024-09-16T05:22:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6503d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6503d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6503d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6503d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6503d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6503d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5ffff640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c600a1400 0x7f3c600994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 10 monclient(hunting): handle_auth_done global_id 104177 payload 824 2024-09-16T05:22:48.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 1 -- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c600a1400 msgr2=0x7f3c600994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c600a1400 0x7f3c600994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 1 -- 172.21.15.114:0/3931017027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c600a1810 msgr2=0x7f3c60099a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c600a1810 0x7f3c60099a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 10 monclient: _finish_hunting 0 2024-09-16T05:22:48.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 1 monclient: found mon.noname-c 2024-09-16T05:22:48.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:48.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:22:48.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 1 -- 172.21.15.114:0/3931017027 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6009b180 con 0x7f3c6009b6c0 2024-09-16T05:22:48.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5ffff640 10 monclient: get_auth_request con 0x7f3c600a1400 auth_method 0 2024-09-16T05:22:48.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5ffff640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c600a1400 0x7f3c600994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:22:48.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6503d640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c600a1810 0x7f3c60099a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:22:48.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6483c640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c6009b6c0 0x7f3c6013cac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400d5f0 tx=0x7f3c5400dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 10 client.?.objecter ms_handle_connect 0x7f3c6009b6c0 2024-09-16T05:22:48.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:22:48.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 1 -- 172.21.15.114:0/3931017027 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c540125f0 con 0x7f3c6009b6c0 2024-09-16T05:22:48.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:22:48.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 10 monclient: dump: 2024-09-16T05:22:48.567 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.567 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.567 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.567 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.567 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.567 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.567 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.567 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.568 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.568 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 10 monclient: _finish_auth 0 2024-09-16T05:22:48.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:22:48.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:48.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 5 monclient: authenticate success, global_id 104177 2024-09-16T05:22:48.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 1 -- 172.21.15.114:0/3931017027 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c54019070 con 0x7f3c6009b6c0 2024-09-16T05:22:48.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:48.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 1 -- 172.21.15.114:0/3931017027 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c540129b0 con 0x7f3c6009b6c0 2024-09-16T05:22:48.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient: _renew_subs 2024-09-16T05:22:48.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:48.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 -- 172.21.15.114:0/3931017027 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c6009c6a0 con 0x7f3c6009b6c0 2024-09-16T05:22:48.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5effd640 4 set_mon_vals no callback set 2024-09-16T05:22:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:22:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 10 monclient: dump: 2024-09-16T05:22:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.569 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 client.104177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:22:48.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient: _renew_subs 2024-09-16T05:22:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 -- 172.21.15.114:0/3931017027 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c60097d00 con 0x7f3c6009b6c0 2024-09-16T05:22:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 librados: init done 2024-09-16T05:22:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 1 -- 172.21.15.114:0/3931017027 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3c54012b50 con 0x7f3c6009b6c0 2024-09-16T05:22:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c60099f60 0x7f3c6008e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c66aa0640 1 -- 172.21.15.114:0/3931017027 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f3c60069390 con 0x7f3c60099f60 2024-09-16T05:22:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 1 --2- 172.21.15.114:0/3931017027 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3c44037840 0x7f3c44039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 1 -- 172.21.15.114:0/3931017027 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(615..615 src has 1..615) ==== 5592+0+0 (secure 0 0 0) 0x7f3c54049230 con 0x7f3c6009b6c0 2024-09-16T05:22:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 10 client.104177.objecter ms_dispatch 0x7f3c60000ba0 osd_map(615..615 src has 1..615) 2024-09-16T05:22:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 3 client.104177.objecter handle_osd_map got epochs [615,615] > 0 2024-09-16T05:22:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 3 client.104177.objecter handle_osd_map decoding full epoch 615 2024-09-16T05:22:48.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 20 client.104177.objecter dump_active .. 0 homeless 2024-09-16T05:22:48.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5ffff640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c60099f60 0x7f3c6008e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5ffff640 10 monclient: get_auth_request con 0x7f3c60099f60 auth_method 0 2024-09-16T05:22:48.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6503d640 1 --2- 172.21.15.114:0/3931017027 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3c44037840 0x7f3c44039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5ffff640 1 -- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c60099f60 msgr2=0x7f3c6008e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:22:48.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5ffff640 1 -- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c60099f60 msgr2=0x7f3c6008e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:22:48.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5ffff640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c60099f60 0x7f3c6008e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:22:48.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6503d640 10 monclient: get_auth_request con 0x7f3c44037840 auth_method 0 2024-09-16T05:22:48.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5ffff640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c60099f60 0x7f3c6008e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:22:48.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c5d7fa640 10 client.104177.objecter ms_handle_connect 0x7f3c44037840 2024-09-16T05:22:48.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.549+0000 7f3c6503d640 1 --2- 172.21.15.114:0/3931017027 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3c44037840 0x7f3c44039d00 secure :-1 s=READY pgs=3584 cs=0 l=1 rev1=1 crypto rx=0x7f3c580af000 tx=0x7f3c580a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.753+0000 7f3c5ffff640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c60099f60 0x7f3c6008e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.753+0000 7f3c5ffff640 10 monclient: get_auth_request con 0x7f3c60099f60 auth_method 0 2024-09-16T05:22:48.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.753+0000 7f3c5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.753+0000 7f3c5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.753+0000 7f3c5ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:22:48.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.753+0000 7f3c5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.753+0000 7f3c5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.753+0000 7f3c5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.753+0000 7f3c5ffff640 10 monclient(hunting): handle_auth_done global_id 114106 payload 824 2024-09-16T05:22:48.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.753+0000 7f3c5ffff640 1 -- 172.21.15.114:0/3931017027 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c6009a340 con 0x7f3c60099f60 2024-09-16T05:22:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.753+0000 7f3c5ffff640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c60099f60 0x7f3c6008e010 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3c6009ad80 tx=0x7f3c4c00f9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.753+0000 7f3c5d7fa640 10 client.104177.objecter ms_handle_connect 0x7f3c60099f60 2024-09-16T05:22:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.753+0000 7f3c5d7fa640 10 client.104177.objecter resend_mon_ops 2024-09-16T05:22:48.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.757+0000 7f3c5d7fa640 1 -- 172.21.15.114:0/3931017027 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c017020 con 0x7f3c60099f60 2024-09-16T05:22:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.757+0000 7f3c5d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.757+0000 7f3c5d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:22:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.757+0000 7f3c5d7fa640 10 monclient: dump: 2024-09-16T05:22:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.757 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.757+0000 7f3c5d7fa640 1 -- 172.21.15.114:0/3931017027 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3c4c0072e0 con 0x7f3c60099f60 2024-09-16T05:22:48.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.757+0000 7f3c5d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:48.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.757+0000 7f3c5d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:22:48.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.757+0000 7f3c5d7fa640 1 -- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c60099f60 msgr2=0x7f3c6008e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.757+0000 7f3c5d7fa640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c60099f60 0x7f3c6008e010 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3c6009ad80 tx=0x7f3c4c00f9b0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.757+0000 7f3c5d7fa640 1 -- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c60099f60 msgr2=0x7f3c6008e010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:22:48.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.757+0000 7f3c5d7fa640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c60099f60 0x7f3c6008e010 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3c6009ad80 tx=0x7f3c4c00f9b0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.769+0000 7f3c66aa0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:22:48.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.769+0000 7f3c66aa0640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c6008e550 0x7f3c6008e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.769+0000 7f3c66aa0640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.769+0000 7f3c66aa0640 1 -- 172.21.15.114:0/3931017027 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f3c600975f0 con 0x7f3c6008e550 2024-09-16T05:22:48.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.769+0000 7f3c5ffff640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c6008e550 0x7f3c6008e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.769+0000 7f3c5ffff640 10 monclient: get_auth_request con 0x7f3c6008e550 auth_method 0 2024-09-16T05:22:48.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.769+0000 7f3c5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.769+0000 7f3c5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.769+0000 7f3c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.769+0000 7f3c5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.769+0000 7f3c5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.769+0000 7f3c5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5ffff640 10 monclient(hunting): handle_auth_done global_id 114109 payload 824 2024-09-16T05:22:48.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5ffff640 1 -- 172.21.15.114:0/3931017027 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3c600a0ec0 con 0x7f3c6008e550 2024-09-16T05:22:48.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5ffff640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c6008e550 0x7f3c6008e960 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c018000 tx=0x7f3c4c000bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5d7fa640 10 client.104177.objecter ms_handle_connect 0x7f3c6008e550 2024-09-16T05:22:48.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5d7fa640 10 client.104177.objecter resend_mon_ops 2024-09-16T05:22:48.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5d7fa640 1 -- 172.21.15.114:0/3931017027 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c017070 con 0x7f3c6008e550 2024-09-16T05:22:48.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:22:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5d7fa640 10 monclient: dump: 2024-09-16T05:22:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.772 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.772 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5d7fa640 1 -- 172.21.15.114:0/3931017027 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f3c4c005b00 con 0x7f3c6008e550 2024-09-16T05:22:48.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:22:48.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:22:48.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5d7fa640 1 -- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c6008e550 msgr2=0x7f3c6008e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5d7fa640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c6008e550 0x7f3c6008e960 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c018000 tx=0x7f3c4c000bd0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5d7fa640 1 -- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c6008e550 msgr2=0x7f3c6008e960 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:22:48.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c5d7fa640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c6008e550 0x7f3c6008e960 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c018000 tx=0x7f3c4c000bd0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.774 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:22:48.774 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:22:48.774 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:22:48.774 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:22:48.774 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 62, 2024-09-16T05:22:48.774 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 15995, 2024-09-16T05:22:48.774 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:22:48.774 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:22:48.774 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:22:48.774 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:22:48.774 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:48.775 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 1, 2024-09-16T05:22:48.775 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:22:48.775 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:22:48.775 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:22:48.775 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:48.775 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:48.775 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:48.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:48.775 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:48.775 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:48.775 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:48.776 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:48.776 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:48.776 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:48.776 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:48.776 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:48.776 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:22:48.776 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:22:48.776 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:48.776 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:48.776 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:48.776 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:48.776 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:48.776 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:48.777 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:48.777 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:48.777 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:48.777 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:48.777 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:48.777 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.777 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.777 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:22:48.777 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:22:48.777 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:22:48.777 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:22:48.777 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:22:48.778 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:22:48.778 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:22:48.778 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:22:48.778 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:22:48.778 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:22:48.778 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:22:48.778 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:22:48.778 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:22:48.778 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:22:48.778 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:22:48.778 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:22:48.778 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:22:48.778 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:48.779 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:48.779 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:48.779 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:48.779 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:48.779 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:48.779 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:48.779 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:48.779 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:48.779 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:48.779 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:48.779 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:48.779 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:22:48.780 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.780 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:22:48.780 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.780 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:22:48.780 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:22:48.780 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:48.780 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:48.780 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.780 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:48.780 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:22:48.780 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.780 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.781 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.781 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:48.781 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:22:48.781 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.781 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.781 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.781 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.781 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:22:48.781 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:22:48.781 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:48.781 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:48.781 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:48.781 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.782 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.782 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:22:48.782 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:22:48.782 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:48.782 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:48.782 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.782 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:48.782 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:22:48.782 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.782 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.782 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.782 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:48.782 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:22:48.783 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.783 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.783 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.783 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.783 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:22:48.783 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:22:48.783 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:48.783 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:48.783 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:48.783 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.783 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.783 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:22:48.783 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:22:48.783 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:48.784 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:48.784 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.784 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:48.784 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:22:48.784 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.784 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.784 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.784 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:48.784 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:22:48.784 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:48.784 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.784 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.785 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.785 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:22:48.785 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:22:48.785 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:48.785 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:48.785 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:48.785 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.785 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.785 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.785 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:22:48.785 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:22:48.785 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.785 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:48.785 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:48.786 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:22:48.786 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.786 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:48.786 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:22:48.786 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:48.786 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:48.786 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:48.786 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:22:48.786 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:48.786 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:48.786 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:48.786 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:22:48.786 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:22:48.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c66aa0640 10 librados: watch_flush enter 2024-09-16T05:22:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c66aa0640 10 librados: watch_flush exit 2024-09-16T05:22:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c66aa0640 20 client.104177.objecter shutdown clearing up homeless session... 2024-09-16T05:22:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c66aa0640 10 client.104177.objecter successfully canceled tick 2024-09-16T05:22:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c66aa0640 1 -- 172.21.15.114:0/3931017027 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3c44037840 msgr2=0x7f3c44039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c66aa0640 1 --2- 172.21.15.114:0/3931017027 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3c44037840 0x7f3c44039d00 secure :-1 s=READY pgs=3584 cs=0 l=1 rev1=1 crypto rx=0x7f3c580af000 tx=0x7f3c580a7070 comp rx=0 tx=0).stop 2024-09-16T05:22:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c66aa0640 10 monclient: shutdown 2024-09-16T05:22:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c66aa0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c66aa0640 1 -- 172.21.15.114:0/3931017027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c6009b6c0 msgr2=0x7f3c6013cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c66aa0640 1 --2- 172.21.15.114:0/3931017027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c6009b6c0 0x7f3c6013cac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400d5f0 tx=0x7f3c5400dab0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c6503d640 1 -- 172.21.15.114:0/3931017027 reap_dead start 2024-09-16T05:22:48.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c6503d640 1 -- 172.21.15.114:0/3931017027 reap_dead start 2024-09-16T05:22:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c66aa0640 1 -- 172.21.15.114:0/3931017027 shutdown_connections 2024-09-16T05:22:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c66aa0640 1 -- 172.21.15.114:0/3931017027 >> 172.21.15.114:0/3931017027 conn(0x7f3c600a1fb0 msgr2=0x7f3c60071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c66aa0640 1 -- 172.21.15.114:0/3931017027 shutdown_connections 2024-09-16T05:22:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c66aa0640 1 -- 172.21.15.114:0/3931017027 wait complete. 2024-09-16T05:22:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.773+0000 7f3c66aa0640 1 librados: shutdown 2024-09-16T05:22:48.790 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:22:48.791 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:22:48.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:22:48.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.901+0000 7f3287e97640 10 monclient: get_monmap_and_config 2024-09-16T05:22:48.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.901+0000 7f3287e97640 10 monclient: build_initial_monmap 2024-09-16T05:22:48.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.901+0000 7f3287e97640 10 monclient: monmap: 2024-09-16T05:22:48.904 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:48.905 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.905 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:22:48.907129+0000 2024-09-16T05:22:48.905 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:22:48.907129+0000 2024-09-16T05:22:48.905 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:48.905 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.905 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:22:48.905 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:22:48.905 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:22:48.905 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:22:48.905 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:22:48.905 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.901+0000 7f3287e97640 1 Processor -- start 2024-09-16T05:22:48.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.901+0000 7f3287e97640 1 -- start start 2024-09-16T05:22:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.901+0000 7f3287e97640 10 monclient: init 2024-09-16T05:22:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-16T05:22:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3280137730 addr v1:172.21.15.149:6789/0 2024-09-16T05:22:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32800a5d00 0x7f328013c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 10 monclient(hunting): picked mon.noname-b con 0x7f32800a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f328013ca00 0x7f328013edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 10 monclient(hunting): picked mon.noname-a con 0x7f328013ca00 addr v2:172.21.15.114:3300/0 2024-09-16T05:22:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32800a55e0 con 0x7f3280137730 2024-09-16T05:22:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f328009c450 con 0x7f328013ca00 2024-09-16T05:22:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f32800a3710 con 0x7f32800a5d00 2024-09-16T05:22:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 10 monclient(hunting): authenticate will time out at 1853.536285s 2024-09-16T05:22:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3285432640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32800a5d00 0x7f328013c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f328013ca00 0x7f328013edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3285432640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32800a5d00 0x7f328013c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:36446/0 (socket says 172.21.15.114:36446) 2024-09-16T05:22:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3285432640 1 -- 172.21.15.114:0/2649018851 learned_addr learned my addr 172.21.15.114:0/2649018851 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3285432640 10 monclient(hunting): get_auth_request con 0x7f32800a5d00 auth_method 0 2024-09-16T05:22:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3285432640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3285432640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3285432640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 10 monclient(hunting): get_auth_request con 0x7f328013ca00 auth_method 0 2024-09-16T05:22:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3285432640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3285432640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3285432640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 10 monclient(hunting): handle_auth_done global_id 74670 payload 274 2024-09-16T05:22:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 1 -- 172.21.15.114:0/2649018851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32800a5d00 msgr2=0x7f328013c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:22:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 1 --2- 172.21.15.114:0/2649018851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32800a5d00 0x7f328013c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 1 -- 172.21.15.114:0/2649018851 >> v1:172.21.15.149:6789/0 conn(0x7f3280137730 legacy=0x7f3280137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:22:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 10 monclient: _finish_hunting 0 2024-09-16T05:22:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 1 monclient: found mon.noname-a 2024-09-16T05:22:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:22:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 1 -- 172.21.15.114:0/2649018851 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328009be00 con 0x7f328013ca00 2024-09-16T05:22:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3285432640 1 --2- 172.21.15.114:0/2649018851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32800a5d00 0x7f328013c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:22:48.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3286434640 1 --2- 172.21.15.114:0/2649018851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f328013ca00 0x7f328013edf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3278009d90 tx=0x7f327801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5159a5dc63cb2148 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3284c31640 1 -- 172.21.15.114:0/2649018851 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32780b2070 con 0x7f328013ca00 2024-09-16T05:22:48.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3284c31640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3284c31640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:22:48.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3284c31640 10 monclient: dump: 2024-09-16T05:22:48.918 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.918 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.918 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.918 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.918 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.918 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.918 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.918 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.918 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.918 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3284c31640 10 monclient: _finish_auth 0 2024-09-16T05:22:48.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3284c31640 10 monclient: _check_auth_tickets 2024-09-16T05:22:48.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3284c31640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:48.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3284c31640 1 -- 172.21.15.114:0/2649018851 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32780a5780 con 0x7f328013ca00 2024-09-16T05:22:48.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 5 monclient: authenticate success, global_id 74670 2024-09-16T05:22:48.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:22:48.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3284c31640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:48.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 10 monclient: get_monmap_and_config success 2024-09-16T05:22:48.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3284c31640 1 -- 172.21.15.114:0/2649018851 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32780a5b50 con 0x7f328013ca00 2024-09-16T05:22:48.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3284c31640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 4 set_mon_vals no callback set 2024-09-16T05:22:48.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3284c31640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:48.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3284c31640 10 monclient: dump: 2024-09-16T05:22:48.919 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.920 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 10 monclient: shutdown 2024-09-16T05:22:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 1 -- 172.21.15.114:0/2649018851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f328013ca00 msgr2=0x7f328013edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 1 --2- 172.21.15.114:0/2649018851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f328013ca00 0x7f328013edf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3278009d90 tx=0x7f327801c7f0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 1 -- 172.21.15.114:0/2649018851 shutdown_connections 2024-09-16T05:22:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 1 --2- 172.21.15.114:0/2649018851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f328013ca00 0x7f328013edf0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 1 --2- 172.21.15.114:0/2649018851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32800a5d00 0x7f328013c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 1 -- 172.21.15.114:0/2649018851 >> 172.21.15.114:0/2649018851 conn(0x7f32800a9c70 msgr2=0x7f32800aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 1 -- 172.21.15.114:0/2649018851 shutdown_connections 2024-09-16T05:22:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.905+0000 7f3287e97640 1 -- 172.21.15.114:0/2649018851 wait complete. 2024-09-16T05:22:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient: build_initial_monmap 2024-09-16T05:22:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient: monmap: 2024-09-16T05:22:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:22:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:22:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:22:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:22:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:22:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:22:48.922 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 librados: starting msgr at 2024-09-16T05:22:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 librados: starting objecter 2024-09-16T05:22:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 Processor -- start 2024-09-16T05:22:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 -- start start 2024-09-16T05:22:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 librados: setting wanted keys 2024-09-16T05:22:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 librados: calling monclient init 2024-09-16T05:22:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient: init 2024-09-16T05:22:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:22:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32800a5d00 0x7f3280136230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient(hunting): picked mon.noname-c con 0x7f32800a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:48.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3280137730 0x7f3280136770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient(hunting): picked mon.noname-b con 0x7f3280137730 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:22:48.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f328013ca00 0x7f328009fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient(hunting): picked mon.noname-a con 0x7f328013ca00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:22:48.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3280141cd0 con 0x7f328013ca00 2024-09-16T05:22:48.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f3280141b50 con 0x7f3280137730 2024-09-16T05:22:48.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:48.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f3280141e50 con 0x7f32800a5d00 2024-09-16T05:22:48.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:48.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient(hunting): authenticate will time out at 1853.541257s 2024-09-16T05:22:48.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32800a5d00 0x7f3280136230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3286434640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f328013ca00 0x7f328009fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32800a5d00 0x7f3280136230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:36460/0 (socket says 172.21.15.114:36460) 2024-09-16T05:22:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 1 -- 172.21.15.114:0/1883213857 learned_addr learned my addr 172.21.15.114:0/1883213857 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 10 monclient(hunting): get_auth_request con 0x7f32800a5d00 auth_method 0 2024-09-16T05:22:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3286434640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f328013ca00 0x7f328009fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33842/0 (socket says 172.21.15.114:33842) 2024-09-16T05:22:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3286434640 10 monclient(hunting): get_auth_request con 0x7f328013ca00 auth_method 0 2024-09-16T05:22:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3286434640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3286434640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3286434640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3286434640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3286434640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3286434640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285432640 1 --2- 172.21.15.114:0/1883213857 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3280137730 0x7f3280136770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285432640 10 monclient(hunting): get_auth_request con 0x7f3280137730 auth_method 0 2024-09-16T05:22:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285432640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285432640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285432640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 10 monclient(hunting): handle_auth_done global_id 104183 payload 824 2024-09-16T05:22:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 1 -- 172.21.15.114:0/1883213857 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3280137730 msgr2=0x7f3280136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 1 --2- 172.21.15.114:0/1883213857 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3280137730 0x7f3280136770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 1 -- 172.21.15.114:0/1883213857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f328013ca00 msgr2=0x7f328009fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 1 --2- 172.21.15.114:0/1883213857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f328013ca00 0x7f328009fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:48.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 10 monclient: _finish_hunting 0 2024-09-16T05:22:48.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 1 monclient: found mon.noname-c 2024-09-16T05:22:48.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:48.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:22:48.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 1 -- 172.21.15.114:0/1883213857 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3280120ac0 con 0x7f32800a5d00 2024-09-16T05:22:48.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3286434640 1 --2- 172.21.15.114:0/1883213857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f328013ca00 0x7f328009fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:22:48.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285c33640 1 --2- 172.21.15.114:0/1883213857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32800a5d00 0x7f3280136230 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f327000c450 tx=0x7f327000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 10 client.?.objecter ms_handle_connect 0x7f32800a5d00 2024-09-16T05:22:48.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:22:48.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 1 -- 172.21.15.114:0/1883213857 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270012830 con 0x7f32800a5d00 2024-09-16T05:22:48.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:22:48.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 10 monclient: dump: 2024-09-16T05:22:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.927 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 10 monclient: _finish_auth 0 2024-09-16T05:22:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 10 monclient: _check_auth_tickets 2024-09-16T05:22:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 1 -- 172.21.15.114:0/1883213857 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3270017070 con 0x7f32800a5d00 2024-09-16T05:22:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 5 monclient: authenticate success, global_id 104183 2024-09-16T05:22:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient: _renew_subs 2024-09-16T05:22:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 -- 172.21.15.114:0/1883213857 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3280120da0 con 0x7f32800a5d00 2024-09-16T05:22:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 1 -- 172.21.15.114:0/1883213857 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270012c30 con 0x7f32800a5d00 2024-09-16T05:22:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:22:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 10 monclient: dump: 2024-09-16T05:22:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:48.929 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3277fff640 4 set_mon_vals no callback set 2024-09-16T05:22:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 client.104183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:22:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient: _renew_subs 2024-09-16T05:22:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 -- 172.21.15.114:0/1883213857 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f32800a9220 con 0x7f32800a5d00 2024-09-16T05:22:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 librados: init done 2024-09-16T05:22:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:48.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 1 -- 172.21.15.114:0/1883213857 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f327001c460 con 0x7f32800a5d00 2024-09-16T05:22:48.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3287e97640 1 -- 172.21.15.114:0/1883213857 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3280137b10 con 0x7f32800a5d00 2024-09-16T05:22:48.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 1 --2- 172.21.15.114:0/1883213857 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3264037840 0x7f3264039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:48.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 1 -- 172.21.15.114:0/1883213857 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(616..616 src has 1..616) ==== 5965+0+0 (secure 0 0 0) 0x7f3270048cc0 con 0x7f32800a5d00 2024-09-16T05:22:48.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 10 client.104183.objecter ms_dispatch 0x7f3280000ba0 osd_map(616..616 src has 1..616) 2024-09-16T05:22:48.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 3 client.104183.objecter handle_osd_map got epochs [616,616] > 0 2024-09-16T05:22:48.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 3 client.104183.objecter handle_osd_map decoding full epoch 616 2024-09-16T05:22:48.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f32767fc640 20 client.104183.objecter dump_active .. 0 homeless 2024-09-16T05:22:48.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285432640 1 --2- 172.21.15.114:0/1883213857 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3264037840 0x7f3264039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:48.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.909+0000 7f3285432640 10 monclient: get_auth_request con 0x7f3264037840 auth_method 0 2024-09-16T05:22:48.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.913+0000 7f32767fc640 10 client.104183.objecter ms_handle_connect 0x7f3264037840 2024-09-16T05:22:48.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.913+0000 7f3285432640 1 --2- 172.21.15.114:0/1883213857 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3264037840 0x7f3264039d00 secure :-1 s=READY pgs=3585 cs=0 l=1 rev1=1 crypto rx=0x7f327c009630 tx=0x7f327c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:48.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.913+0000 7f32767fc640 1 -- 172.21.15.114:0/1883213857 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f3270012dd0 con 0x7f32800a5d00 2024-09-16T05:22:48.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.913+0000 7f32767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:48.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:48.913+0000 7f32767fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:22:49.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.121+0000 7f3287e97640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:22:49.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.121+0000 7f3287e97640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:22:49.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.121+0000 7f3287e97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:49.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.121+0000 7f3287e97640 1 -- 172.21.15.114:0/1883213857 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f32800a21a0 con 0x7f32800a5d00 2024-09-16T05:22:49.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.121+0000 7f32767fc640 1 -- 172.21.15.114:0/1883213857 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f327001a760 con 0x7f32800a5d00 2024-09-16T05:22:49.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.121+0000 7f32767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:22:49.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.121+0000 7f32767fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:22:49.122 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.123 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=11,kv=12,logm=56} crc {auth=1013073745,config=2586702516,health=719061908,kv=2757161437,logm=4239634946}) 2024-09-16T05:22:49.123 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=10,kv=11,logm=58} crc {auth=2543129159,config=2586702516,health=3616884070,kv=1951643036,logm=2055339420}) 2024-09-16T05:22:49.123 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.123 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=11,kv=12,logm=56} crc {auth=1013073745,config=2586702516,health=719061908,kv=2757161437,logm=4239634946}) 2024-09-16T05:22:49.123 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=11,kv=12,logm=55} crc {auth=2752672558,config=2586702516,health=1527692481,kv=2757161437,logm=2423334953}) 2024-09-16T05:22:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 10 librados: watch_flush enter 2024-09-16T05:22:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 10 librados: watch_flush exit 2024-09-16T05:22:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 20 client.104183.objecter shutdown clearing up homeless session... 2024-09-16T05:22:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 10 client.104183.objecter successfully canceled tick 2024-09-16T05:22:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 1 -- 172.21.15.114:0/1883213857 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3264037840 msgr2=0x7f3264039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 1 --2- 172.21.15.114:0/1883213857 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3264037840 0x7f3264039d00 secure :-1 s=READY pgs=3585 cs=0 l=1 rev1=1 crypto rx=0x7f327c009630 tx=0x7f327c008040 comp rx=0 tx=0).stop 2024-09-16T05:22:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 10 monclient: shutdown 2024-09-16T05:22:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 1 -- 172.21.15.114:0/1883213857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32800a5d00 msgr2=0x7f3280136230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 1 --2- 172.21.15.114:0/1883213857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32800a5d00 0x7f3280136230 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f327000c450 tx=0x7f327000c910 comp rx=0 tx=0).stop 2024-09-16T05:22:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 1 -- 172.21.15.114:0/1883213857 shutdown_connections 2024-09-16T05:22:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 1 --2- 172.21.15.114:0/1883213857 >> v2:172.21.15.149:6812/1185531135 conn(0x7f3264037840 0x7f3264039d00 unknown :-1 s=CLOSED pgs=3585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 1 --2- 172.21.15.114:0/1883213857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f328013ca00 0x7f328009fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 1 --2- 172.21.15.114:0/1883213857 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3280137730 0x7f3280136770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 1 --2- 172.21.15.114:0/1883213857 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32800a5d00 0x7f3280136230 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 1 -- 172.21.15.114:0/1883213857 >> 172.21.15.114:0/1883213857 conn(0x7f32800a9c70 msgr2=0x7f32800a76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 1 -- 172.21.15.114:0/1883213857 shutdown_connections 2024-09-16T05:22:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 1 -- 172.21.15.114:0/1883213857 wait complete. 2024-09-16T05:22:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f3287e97640 1 librados: shutdown 2024-09-16T05:22:49.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=837646469}) 2024-09-16T05:22:49.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2894386515}) 2024-09-16T05:22:49.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=837646469}) 2024-09-16T05:22:49.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.125+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3060011029}) 2024-09-16T05:22:49.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.129+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.129+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2723565309}) 2024-09-16T05:22:49.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.129+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=193252939}) 2024-09-16T05:22:49.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.129+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.129+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2723565309}) 2024-09-16T05:22:49.128 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.129+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4023234740}) 2024-09-16T05:22:49.130 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.129+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.130 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.129+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=427764987}) 2024-09-16T05:22:49.130 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.129+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1403118985}) 2024-09-16T05:22:49.130 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.129+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.130 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.129+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=427764987}) 2024-09-16T05:22:49.130 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.129+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1852617254}) 2024-09-16T05:22:49.132 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.133+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.133 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.133+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621385894}) 2024-09-16T05:22:49.133 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.133+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1011561535}) 2024-09-16T05:22:49.133 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.133+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.133 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.133+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621385894}) 2024-09-16T05:22:49.133 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.133+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1231960323}) 2024-09-16T05:22:49.135 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.137+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.135 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.137+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1640366094}) 2024-09-16T05:22:49.135 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.137+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1102639133}) 2024-09-16T05:22:49.135 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.137+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.135 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.137+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1640366094}) 2024-09-16T05:22:49.135 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.137+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=484827720}) 2024-09-16T05:22:49.137 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.137+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.137 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.137+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738278302}) 2024-09-16T05:22:49.137 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.137+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829533744}) 2024-09-16T05:22:49.137 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.137+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.137 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.137+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738278302}) 2024-09-16T05:22:49.137 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.137+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228646360}) 2024-09-16T05:22:49.139 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.141+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.139 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.141+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3461524693}) 2024-09-16T05:22:49.139 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.141+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2074670286}) 2024-09-16T05:22:49.139 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.141+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.139 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.141+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3461524693}) 2024-09-16T05:22:49.139 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.141+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=777408205}) 2024-09-16T05:22:49.141 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:22:49.142 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.141+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.142 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.141+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4200184889}) 2024-09-16T05:22:49.142 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.141+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4029898505}) 2024-09-16T05:22:49.142 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.141+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.142 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.141+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4200184889}) 2024-09-16T05:22:49.142 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.141+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1579323575}) 2024-09-16T05:22:49.143 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.145+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.143 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.145+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1298046915}) 2024-09-16T05:22:49.143 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.145+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3182673630}) 2024-09-16T05:22:49.143 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.145+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.143 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.145+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1298046915}) 2024-09-16T05:22:49.143 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.145+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=42259771}) 2024-09-16T05:22:49.145 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.145+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.145 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.145+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196217221}) 2024-09-16T05:22:49.145 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.145+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3339471056}) 2024-09-16T05:22:49.145 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.145+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.145 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.145+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196217221}) 2024-09-16T05:22:49.145 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.145+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3912827277}) 2024-09-16T05:22:49.146 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.147 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425516974}) 2024-09-16T05:22:49.147 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=6662789}) 2024-09-16T05:22:49.147 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.147 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425516974}) 2024-09-16T05:22:49.147 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1899575557}) 2024-09-16T05:22:49.148 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.148 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136083433}) 2024-09-16T05:22:49.148 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2069833513}) 2024-09-16T05:22:49.149 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.149 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136083433}) 2024-09-16T05:22:49.149 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1103434391}) 2024-09-16T05:22:49.150 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.150 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=599508641}) 2024-09-16T05:22:49.150 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2507183873}) 2024-09-16T05:22:49.150 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.149+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.150 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.153+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=599508641}) 2024-09-16T05:22:49.150 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.153+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2608011208}) 2024-09-16T05:22:49.152 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.153+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.152 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.153+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3203559561}) 2024-09-16T05:22:49.152 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.153+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3568591871}) 2024-09-16T05:22:49.153 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.153+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.153 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.153+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3203559561}) 2024-09-16T05:22:49.153 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.153+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=616106257}) 2024-09-16T05:22:49.156 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.153+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.156 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.153+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1687036552}) 2024-09-16T05:22:49.156 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.153+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983287928}) 2024-09-16T05:22:49.156 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.153+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.157 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.153+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1687036552}) 2024-09-16T05:22:49.157 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.153+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3444715564}) 2024-09-16T05:22:49.158 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.157+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.158 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.157+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720192581}) 2024-09-16T05:22:49.158 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.157+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4173995276}) 2024-09-16T05:22:49.158 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.157+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.158 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.157+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720192581}) 2024-09-16T05:22:49.158 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.157+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1525736081}) 2024-09-16T05:22:49.158 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.157+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.158 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.157+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=224004333}) 2024-09-16T05:22:49.158 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.157+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2812396449}) 2024-09-16T05:22:49.158 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.157+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.158 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.157+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=224004333}) 2024-09-16T05:22:49.158 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.157+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4213824621}) 2024-09-16T05:22:49.159 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.161+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.159 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.161+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434459767}) 2024-09-16T05:22:49.159 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.161+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=671761860}) 2024-09-16T05:22:49.159 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.161+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.160 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.161+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434459767}) 2024-09-16T05:22:49.160 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.161+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=805121927}) 2024-09-16T05:22:49.161 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.161+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.161 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.161+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860100134}) 2024-09-16T05:22:49.161 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.161+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3743614380}) 2024-09-16T05:22:49.161 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.161+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.161 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.161+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860100134}) 2024-09-16T05:22:49.161 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.161+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=593067143}) 2024-09-16T05:22:49.163 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.165+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.163 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.165+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3027082618}) 2024-09-16T05:22:49.163 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.165+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3442292596}) 2024-09-16T05:22:49.163 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.165+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.163 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.165+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3027082618}) 2024-09-16T05:22:49.163 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.165+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309730651}) 2024-09-16T05:22:49.164 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.165+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.165 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.165+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3319025417}) 2024-09-16T05:22:49.165 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.165+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1768174343}) 2024-09-16T05:22:49.165 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.165+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.165 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.165+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3319025417}) 2024-09-16T05:22:49.165 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.165+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2290892247}) 2024-09-16T05:22:49.166 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.169+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.166 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.169+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141362437}) 2024-09-16T05:22:49.167 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.169+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4127440703}) 2024-09-16T05:22:49.167 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.169+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.167 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.169+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141362437}) 2024-09-16T05:22:49.167 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.169+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1074306732}) 2024-09-16T05:22:49.168 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.169+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.168 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.169+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=989295370}) 2024-09-16T05:22:49.168 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.169+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1145143981}) 2024-09-16T05:22:49.168 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.169+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.168 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.169+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=989295370}) 2024-09-16T05:22:49.168 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.169+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=579717856}) 2024-09-16T05:22:49.170 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.173+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.170 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.173+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4059563767}) 2024-09-16T05:22:49.170 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.173+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1022624688}) 2024-09-16T05:22:49.170 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.173+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.171 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.173+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4059563767}) 2024-09-16T05:22:49.171 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.173+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=40668153}) 2024-09-16T05:22:49.172 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.173+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.172 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.173+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2475919117}) 2024-09-16T05:22:49.172 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.173+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3032620921}) 2024-09-16T05:22:49.172 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.173+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.172 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.173+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2475919117}) 2024-09-16T05:22:49.173 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.173+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1766629618}) 2024-09-16T05:22:49.174 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.177+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.175 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.177+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4131700576}) 2024-09-16T05:22:49.175 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.177+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2013005008}) 2024-09-16T05:22:49.175 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.177+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.175 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.177+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4131700576}) 2024-09-16T05:22:49.175 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.177+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=125971691}) 2024-09-16T05:22:49.176 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.177+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.176 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.177+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360165354}) 2024-09-16T05:22:49.177 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.177+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3891436849}) 2024-09-16T05:22:49.177 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.177+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.177 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.177+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360165354}) 2024-09-16T05:22:49.177 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.177+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2520379936}) 2024-09-16T05:22:49.178 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.181+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.178 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.181+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711205585}) 2024-09-16T05:22:49.179 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.181+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3769533576}) 2024-09-16T05:22:49.179 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.181+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.179 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.181+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711205585}) 2024-09-16T05:22:49.179 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.181+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3700782440}) 2024-09-16T05:22:49.180 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.181+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.181 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.181+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=142716719}) 2024-09-16T05:22:49.181 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.181+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3629976128}) 2024-09-16T05:22:49.181 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.181+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.181 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.181+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=142716719}) 2024-09-16T05:22:49.181 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.181+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=844616346}) 2024-09-16T05:22:49.183 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.185+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.183 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.185+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156182351}) 2024-09-16T05:22:49.183 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.185+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1007544156}) 2024-09-16T05:22:49.183 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.185+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.183 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.185+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156182351}) 2024-09-16T05:22:49.183 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.185+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1069618392}) 2024-09-16T05:22:49.184 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.185+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.185 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.185+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949607441}) 2024-09-16T05:22:49.185 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.185+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3089943927}) 2024-09-16T05:22:49.185 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.185+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.185 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.185+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949607441}) 2024-09-16T05:22:49.185 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.185+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=305795893}) 2024-09-16T05:22:49.187 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.189+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.187 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.189+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3863191995}) 2024-09-16T05:22:49.187 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.189+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1580075509}) 2024-09-16T05:22:49.187 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.189+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.187 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.189+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3863191995}) 2024-09-16T05:22:49.187 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.189+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2473122654}) 2024-09-16T05:22:49.189 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.189+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.189 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.189+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=663692494}) 2024-09-16T05:22:49.189 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.189+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4166758480}) 2024-09-16T05:22:49.189 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.189+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.189 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.189+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=663692494}) 2024-09-16T05:22:49.190 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.189+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=677160750}) 2024-09-16T05:22:49.191 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.193+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.191 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.193+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=375598735}) 2024-09-16T05:22:49.191 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.193+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3879666366}) 2024-09-16T05:22:49.191 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.193+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.191 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.193+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=375598735}) 2024-09-16T05:22:49.192 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.193+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1764634499}) 2024-09-16T05:22:49.193 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.193+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.193 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.193+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3295718043}) 2024-09-16T05:22:49.193 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.193+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3184837262}) 2024-09-16T05:22:49.194 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.193+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.194 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.193+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3295718043}) 2024-09-16T05:22:49.194 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.193+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=615832108}) 2024-09-16T05:22:49.196 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.197+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.196 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.197+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2446099275}) 2024-09-16T05:22:49.196 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.197+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1619632176}) 2024-09-16T05:22:49.196 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.197+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.196 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.197+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2446099275}) 2024-09-16T05:22:49.196 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.197+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2330008269}) 2024-09-16T05:22:49.198 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.197+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.198 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.197+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150084164}) 2024-09-16T05:22:49.198 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.197+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=295853880}) 2024-09-16T05:22:49.198 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.197+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.198 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.197+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150084164}) 2024-09-16T05:22:49.198 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.197+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1270930673}) 2024-09-16T05:22:49.200 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.201+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.200 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.201+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3161093753}) 2024-09-16T05:22:49.200 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.201+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4045065547}) 2024-09-16T05:22:49.200 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.201+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.201 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.201+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3161093753}) 2024-09-16T05:22:49.201 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.201+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1091026456}) 2024-09-16T05:22:49.202 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.205+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.202 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.205+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=668424641}) 2024-09-16T05:22:49.202 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.205+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1030766104}) 2024-09-16T05:22:49.203 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.205+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.203 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.205+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=668424641}) 2024-09-16T05:22:49.203 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.205+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=611956598}) 2024-09-16T05:22:49.205 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.205+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.205 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.205+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1724565009}) 2024-09-16T05:22:49.205 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.205+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=187153225}) 2024-09-16T05:22:49.205 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.205+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.205 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.205+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1724565009}) 2024-09-16T05:22:49.205 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.205+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1721106948}) 2024-09-16T05:22:49.207 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.209+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.207 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.209+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291572111}) 2024-09-16T05:22:49.207 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.209+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3906933253}) 2024-09-16T05:22:49.207 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.209+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.207 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.209+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291572111}) 2024-09-16T05:22:49.207 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.209+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3566717199}) 2024-09-16T05:22:49.209 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.209+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.209 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.209+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=887210913}) 2024-09-16T05:22:49.209 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.209+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1764216611}) 2024-09-16T05:22:49.209 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.209+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.209 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.209+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=887210913}) 2024-09-16T05:22:49.209 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.209+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1431257600}) 2024-09-16T05:22:49.211 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.213+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.211 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.213+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428365644}) 2024-09-16T05:22:49.211 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.213+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294943142}) 2024-09-16T05:22:49.212 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.213+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.212 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.213+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428365644}) 2024-09-16T05:22:49.212 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.213+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2415939048}) 2024-09-16T05:22:49.213 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.213+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.213 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.213+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2618448725}) 2024-09-16T05:22:49.214 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.213+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=878491167}) 2024-09-16T05:22:49.214 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.213+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.214 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.213+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2618448725}) 2024-09-16T05:22:49.214 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.213+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1190634675}) 2024-09-16T05:22:49.216 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.217+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.216 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.217+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2628231558}) 2024-09-16T05:22:49.216 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.217+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3708908401}) 2024-09-16T05:22:49.216 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.217+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.216 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.217+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2628231558}) 2024-09-16T05:22:49.216 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.217+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3682363764}) 2024-09-16T05:22:49.218 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.217+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.218 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.217+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023144925}) 2024-09-16T05:22:49.218 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.217+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=434481130}) 2024-09-16T05:22:49.218 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.217+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.218 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.217+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023144925}) 2024-09-16T05:22:49.218 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.221+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428394275}) 2024-09-16T05:22:49.220 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.221+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.220 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.221+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=392940993}) 2024-09-16T05:22:49.220 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.221+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4282588958}) 2024-09-16T05:22:49.220 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.221+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.220 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.221+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=392940993}) 2024-09-16T05:22:49.221 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.221+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2082362094}) 2024-09-16T05:22:49.223 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.225+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.223 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.225+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=398623738}) 2024-09-16T05:22:49.223 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.225+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3968588272}) 2024-09-16T05:22:49.223 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.225+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.223 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.225+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=398623738}) 2024-09-16T05:22:49.223 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.225+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2833549577}) 2024-09-16T05:22:49.225 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.225+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.225 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.225+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3319609675}) 2024-09-16T05:22:49.225 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.225+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3354884664}) 2024-09-16T05:22:49.225 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.225+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.225 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.225+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3319609675}) 2024-09-16T05:22:49.225 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.225+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2505920906}) 2024-09-16T05:22:49.227 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.229+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.227 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.229+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4282350171}) 2024-09-16T05:22:49.227 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.229+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335936445}) 2024-09-16T05:22:49.227 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.229+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.227 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.229+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4282350171}) 2024-09-16T05:22:49.228 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.229+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2505587603}) 2024-09-16T05:22:49.229 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.229+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.229 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.229+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3686332614}) 2024-09-16T05:22:49.229 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.229+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=852042933}) 2024-09-16T05:22:49.229 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.229+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.229 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.229+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3686332614}) 2024-09-16T05:22:49.229 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.229+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2857399082}) 2024-09-16T05:22:49.231 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.233+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.231 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.233+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2102095020}) 2024-09-16T05:22:49.231 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.233+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=152512454}) 2024-09-16T05:22:49.231 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.233+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.231 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.233+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2102095020}) 2024-09-16T05:22:49.232 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.233+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2641452145}) 2024-09-16T05:22:49.233 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.233+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.233 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.233+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2852501992}) 2024-09-16T05:22:49.233 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.233+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=388130354}) 2024-09-16T05:22:49.234 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.233+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.234 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.233+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2852501992}) 2024-09-16T05:22:49.234 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.233+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=71192823}) 2024-09-16T05:22:49.235 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.237+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.235 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.237+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103478313}) 2024-09-16T05:22:49.236 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.237+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2207327925}) 2024-09-16T05:22:49.236 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.237+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.236 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.237+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103478313}) 2024-09-16T05:22:49.236 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.237+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1474727849}) 2024-09-16T05:22:49.237 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.237+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.237 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.237+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3205274184}) 2024-09-16T05:22:49.237 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.237+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1230856473}) 2024-09-16T05:22:49.238 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.237+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.238 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.237+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3205274184}) 2024-09-16T05:22:49.238 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.237+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1702315488}) 2024-09-16T05:22:49.239 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.241+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.239 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.241+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062390178}) 2024-09-16T05:22:49.239 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.241+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3651622360}) 2024-09-16T05:22:49.240 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.241+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.240 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.241+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062390178}) 2024-09-16T05:22:49.240 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.241+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=976366683}) 2024-09-16T05:22:49.241 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.241+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.241 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.241+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553926794}) 2024-09-16T05:22:49.242 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.241+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2395769021}) 2024-09-16T05:22:49.242 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.241+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.242 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.241+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553926794}) 2024-09-16T05:22:49.242 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.241+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1192848456}) 2024-09-16T05:22:49.243 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.245+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.244 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.245+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094453870}) 2024-09-16T05:22:49.244 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.245+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1168122674}) 2024-09-16T05:22:49.244 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.245+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.244 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.245+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094453870}) 2024-09-16T05:22:49.244 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.245+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1341298634}) 2024-09-16T05:22:49.246 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.245+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.246 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.245+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2696857530}) 2024-09-16T05:22:49.246 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.245+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1940704332}) 2024-09-16T05:22:49.246 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.245+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.246 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.245+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2696857530}) 2024-09-16T05:22:49.246 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.245+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3714418529}) 2024-09-16T05:22:49.248 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.249+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.248 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.249+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2572927760}) 2024-09-16T05:22:49.249 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.249+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=891187598}) 2024-09-16T05:22:49.249 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.249+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.249 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.249+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2572927760}) 2024-09-16T05:22:49.249 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.249+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3359459501}) 2024-09-16T05:22:49.251 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.253+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.251 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.253+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=66,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=8} crc {logm=1555765098,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=2122625551,mon_config_key=2060602074}) 2024-09-16T05:22:49.251 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.253+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=67,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=7} crc {logm=442191982,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=2122625551,mon_config_key=1845606068}) 2024-09-16T05:22:49.251 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.253+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.252 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.253+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=66,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=8} crc {logm=1555765098,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=2122625551,mon_config_key=2060602074}) 2024-09-16T05:22:49.252 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.253+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=66,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11} crc {logm=1918736439,mdsmap=1078762840,mgr=919972836,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=1255921478,mon_config_key=3554792676}) 2024-09-16T05:22:49.253 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.253+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.253 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.253+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=38,osdmap=48} crc {mon_config_key=1470596779,monmap=3371862038,osd_metadata=4163472531,osd_pg_creating=2997021486,osd_snap=4291354106,osdmap=987861170}) 2024-09-16T05:22:49.254 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.253+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=7,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=40,osdmap=45} crc {mon_config_key=1226134842,monmap=3045472550,osd_metadata=1189497465,osd_pg_creating=2997021486,osd_snap=3323924521,osdmap=1303638374}) 2024-09-16T05:22:49.254 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.253+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.254 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.253+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=38,osdmap=48} crc {mon_config_key=1470596779,monmap=3371862038,osd_metadata=4163472531,osd_pg_creating=2997021486,osd_snap=4291354106,osdmap=987861170}) 2024-09-16T05:22:49.254 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.253+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=38,osdmap=44} crc {mon_config_key=2035627754,monmap=3045472550,osd_metadata=487805116,osd_pg_creating=2997021486,osd_snap=1157375193,osdmap=2932528880}) 2024-09-16T05:22:49.256 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.257+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.256 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.257+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=718094446}) 2024-09-16T05:22:49.257 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.257+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2640787841}) 2024-09-16T05:22:49.257 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.257+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.257 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.257+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=718094446}) 2024-09-16T05:22:49.257 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.257+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3004754381}) 2024-09-16T05:22:49.258 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.257+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.258 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.257+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4200882559}) 2024-09-16T05:22:49.258 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.257+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3031033370}) 2024-09-16T05:22:49.258 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.257+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.258 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.257+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4200882559}) 2024-09-16T05:22:49.258 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.257+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4173817935}) 2024-09-16T05:22:49.260 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.261+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.260 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.261+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1965905885}) 2024-09-16T05:22:49.260 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.261+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3120072932}) 2024-09-16T05:22:49.260 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.261+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.260 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.261+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1965905885}) 2024-09-16T05:22:49.260 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.261+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3300478125}) 2024-09-16T05:22:49.262 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.265+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.263 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.265+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4256133249}) 2024-09-16T05:22:49.263 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.265+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3769737081}) 2024-09-16T05:22:49.263 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.265+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.263 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.265+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4256133249}) 2024-09-16T05:22:49.263 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.265+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3693622704}) 2024-09-16T05:22:49.265 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.265+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.265 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.265+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2495098003}) 2024-09-16T05:22:49.265 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.265+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2412395714}) 2024-09-16T05:22:49.265 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.265+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.265 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.265+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2495098003}) 2024-09-16T05:22:49.265 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.265+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4081958705}) 2024-09-16T05:22:49.268 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.269+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.268 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.269+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1832572946}) 2024-09-16T05:22:49.268 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.269+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1002722445}) 2024-09-16T05:22:49.268 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.269+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.269 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.269+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1832572946}) 2024-09-16T05:22:49.269 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.269+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=685421085}) 2024-09-16T05:22:49.272 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.273+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.272 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.273+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=680150171}) 2024-09-16T05:22:49.273 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.273+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4074152947}) 2024-09-16T05:22:49.273 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.273+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.273 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.273+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=680150171}) 2024-09-16T05:22:49.273 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.273+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1974608342}) 2024-09-16T05:22:49.276 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.277+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.277 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.277+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1774971222}) 2024-09-16T05:22:49.277 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.277+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2502542496}) 2024-09-16T05:22:49.277 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.277+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.277 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.277+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1774971222}) 2024-09-16T05:22:49.277 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.277+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1569230833}) 2024-09-16T05:22:49.279 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.281+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.279 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.281+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=48} crc {osdmap=1572829390}) 2024-09-16T05:22:49.279 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.281+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=45} crc {osdmap=356084177}) 2024-09-16T05:22:49.279 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.281+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:22:49.279 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.281+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=48} crc {osdmap=1572829390}) 2024-09-16T05:22:49.279 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:22:49.281+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=46} crc {osdmap=221197797}) 2024-09-16T05:22:50.142 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:22:50.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:22:50.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 10 monclient: get_monmap_and_config 2024-09-16T05:22:50.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 10 monclient: build_initial_monmap 2024-09-16T05:22:50.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 10 monclient: monmap: 2024-09-16T05:22:50.260 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:50.260 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.260 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:22:50.265898+0000 2024-09-16T05:22:50.260 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:22:50.265898+0000 2024-09-16T05:22:50.260 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:50.261 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.261 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:22:50.261 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:22:50.261 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:22:50.261 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:22:50.261 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:22:50.261 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 1 Processor -- start 2024-09-16T05:22:50.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 1 -- start start 2024-09-16T05:22:50.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 10 monclient: init 2024-09-16T05:22:50.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:50.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-16T05:22:50.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23d809b6c0 addr v1:172.21.15.114:6789/0 2024-09-16T05:22:50.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23d80a15d0 addr v1:172.21.15.149:6789/0 2024-09-16T05:22:50.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f23d80a19e0 0x7f23d813dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 10 monclient(hunting): picked mon.noname-c con 0x7f23d80a19e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:22:50.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23d809d4e0 con 0x7f23d809b6c0 2024-09-16T05:22:50.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23d809f430 con 0x7f23d80a15d0 2024-09-16T05:22:50.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f23d809e390 con 0x7f23d80a19e0 2024-09-16T05:22:50.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:50.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23de567640 10 monclient(hunting): authenticate will time out at 1854.895486s 2024-09-16T05:22:50.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23d7fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f23d809b6c0 0x7f23d80a0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34686/0 (socket says 172.21.15.114:34686) 2024-09-16T05:22:50.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23d7fff640 1 -- 172.21.15.114:0/32739310 learned_addr learned my addr 172.21.15.114:0/32739310 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:22:50.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23dcb04640 1 --2- 172.21.15.114:0/32739310 >> v2:172.21.15.149:3300/0 conn(0x7f23d80a19e0 0x7f23d813dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23dcb04640 10 monclient(hunting): get_auth_request con 0x7f23d80a19e0 auth_method 0 2024-09-16T05:22:50.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23dcb04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23dcb04640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.261+0000 7f23dcb04640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 -- 172.21.15.114:0/32739310 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1648798708 0 0) 0x7f23d809d4e0 con 0x7f23d809b6c0 2024-09-16T05:22:50.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:22:50.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient(hunting): my global_id is 74676 2024-09-16T05:22:50.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 -- 172.21.15.114:0/32739310 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23c4003530 con 0x7f23d809b6c0 2024-09-16T05:22:50.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23dcb04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23dcb04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23dcb04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 -- 172.21.15.114:0/32739310 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1692431927 0 0) 0x7f23c4003530 con 0x7f23d809b6c0 2024-09-16T05:22:50.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 -- 172.21.15.114:0/32739310 >> v2:172.21.15.149:3300/0 conn(0x7f23d80a19e0 msgr2=0x7f23d813dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:22:50.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 --2- 172.21.15.114:0/32739310 >> v2:172.21.15.149:3300/0 conn(0x7f23d80a19e0 0x7f23d813dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 -- 172.21.15.114:0/32739310 >> v1:172.21.15.149:6789/0 conn(0x7f23d80a15d0 legacy=0x7f23d813a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: _finish_hunting 0 2024-09-16T05:22:50.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:22:50.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:50.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:22:50.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 -- 172.21.15.114:0/32739310 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23d809c2f0 con 0x7f23d809b6c0 2024-09-16T05:22:50.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: _finish_auth 0 2024-09-16T05:22:50.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: _check_auth_tickets 2024-09-16T05:22:50.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:50.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 -- 172.21.15.114:0/32739310 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f23c8001680 con 0x7f23d809b6c0 2024-09-16T05:22:50.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:22:50.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: dump: 2024-09-16T05:22:50.271 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.271 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.271 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.271 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.271 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.271 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.271 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.271 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.271 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.271 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:22:50.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:50.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 -- 172.21.15.114:0/32739310 >> v1:172.21.15.114:6789/0 conn(0x7f23d809b6c0 legacy=0x7f23d80a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:22:50.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 --2- 172.21.15.114:0/32739310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c40039c0 0x7f23c4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient(hunting): picked mon.a con 0x7f23c40039c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:22:50.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 --2- 172.21.15.114:0/32739310 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23c4004810 0x7f23c4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient(hunting): picked mon.b con 0x7f23c4004810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:22:50.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 --2- 172.21.15.114:0/32739310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23c40051d0 0x7f23c400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient(hunting): picked mon.c con 0x7f23c40051d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:50.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 -- 172.21.15.114:0/32739310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f23d809e390 con 0x7f23c40039c0 2024-09-16T05:22:50.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 -- 172.21.15.114:0/32739310 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f23c400c170 con 0x7f23c4004810 2024-09-16T05:22:50.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 -- 172.21.15.114:0/32739310 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f23c400c370 con 0x7f23c40051d0 2024-09-16T05:22:50.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:50.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:22:50.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 1 --2- 172.21.15.114:0/32739310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c40039c0 0x7f23c4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 10 monclient(hunting): get_auth_request con 0x7f23c40039c0 auth_method 0 2024-09-16T05:22:50.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:22:50.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23dcb04640 1 --2- 172.21.15.114:0/32739310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23c40051d0 0x7f23c400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:22:50.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23dcb04640 10 monclient(hunting): get_auth_request con 0x7f23c40051d0 auth_method 0 2024-09-16T05:22:50.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23dcb04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23dcb04640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23dcb04640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:22:50.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23dcb04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23cf7fe640 1 --2- 172.21.15.114:0/32739310 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23c4004810 0x7f23c4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23dcb04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23dcb04640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:22:50.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 10 monclient(hunting): handle_auth_done global_id 74676 payload 293 2024-09-16T05:22:50.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 1 -- 172.21.15.114:0/32739310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23c40051d0 msgr2=0x7f23c400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:22:50.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 1 --2- 172.21.15.114:0/32739310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23c40051d0 0x7f23c400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 1 -- 172.21.15.114:0/32739310 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23c4004810 msgr2=0x7f23c4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:22:50.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 1 --2- 172.21.15.114:0/32739310 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23c4004810 0x7f23c4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:22:50.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 1 monclient: found mon.a 2024-09-16T05:22:50.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:50.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:22:50.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 1 -- 172.21.15.114:0/32739310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c4003530 con 0x7f23c40039c0 2024-09-16T05:22:50.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 10 monclient: _finish_auth 0 2024-09-16T05:22:50.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:22:50.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:50.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23dcb04640 1 -- 172.21.15.114:0/32739310 reap_dead start 2024-09-16T05:22:50.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23de567640 5 monclient: authenticate success, global_id 74676 2024-09-16T05:22:50.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23de567640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:22:50.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23cf7fe640 10 monclient: get_auth_request con 0x7f23c4004810 auth_method 0 2024-09-16T05:22:50.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23cf7fe640 1 --2- 172.21.15.114:0/32739310 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23c4004810 0x7f23c4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:22:50.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d7fff640 1 --2- 172.21.15.114:0/32739310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c40039c0 0x7f23c4003de0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f23c800c210 tx=0x7f23c801cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c6c0458bd7cc96 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:50.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 -- 172.21.15.114:0/32739310 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c8007d40 con 0x7f23c40039c0 2024-09-16T05:22:50.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:50.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: dump: 2024-09-16T05:22:50.276 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.276 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.276 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.276 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.276 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.276 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.276 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.276 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.276 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.276 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23de567640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:22:50.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 -- 172.21.15.114:0/32739310 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23c80a9070 con 0x7f23c40039c0 2024-09-16T05:22:50.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:50.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23de567640 10 monclient: get_monmap_and_config success 2024-09-16T05:22:50.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23de567640 4 set_mon_vals no callback set 2024-09-16T05:22:50.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 1 -- 172.21.15.114:0/32739310 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c8005d80 con 0x7f23c40039c0 2024-09-16T05:22:50.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:50.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23d77fe640 10 monclient: dump: 2024-09-16T05:22:50.277 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.277 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.277 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.277 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.277 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.277 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.277 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.278 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.278 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.278 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23de567640 10 monclient: shutdown 2024-09-16T05:22:50.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23de567640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:50.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23de567640 1 -- 172.21.15.114:0/32739310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c40039c0 msgr2=0x7f23c4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23de567640 1 --2- 172.21.15.114:0/32739310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c40039c0 0x7f23c4003de0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f23c800c210 tx=0x7f23c801cc00 comp rx=0 tx=0).stop 2024-09-16T05:22:50.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23de567640 1 -- 172.21.15.114:0/32739310 shutdown_connections 2024-09-16T05:22:50.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23de567640 1 --2- 172.21.15.114:0/32739310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c40039c0 0x7f23c4003de0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23de567640 1 -- 172.21.15.114:0/32739310 >> 172.21.15.114:0/32739310 conn(0x7f23d80a1fb0 msgr2=0x7f23d80a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:50.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23de567640 1 -- 172.21.15.114:0/32739310 shutdown_connections 2024-09-16T05:22:50.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.265+0000 7f23de567640 1 -- 172.21.15.114:0/32739310 wait complete. 2024-09-16T05:22:50.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient: build_initial_monmap 2024-09-16T05:22:50.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient: monmap: 2024-09-16T05:22:50.279 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:50.279 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:22:50.279 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:22:50.279 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:22:50.279 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:50.279 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.279 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:22:50.279 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:22:50.279 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:22:50.279 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 librados: starting msgr at 2024-09-16T05:22:50.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 librados: starting objecter 2024-09-16T05:22:50.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 Processor -- start 2024-09-16T05:22:50.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 -- start start 2024-09-16T05:22:50.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 librados: setting wanted keys 2024-09-16T05:22:50.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 librados: calling monclient init 2024-09-16T05:22:50.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient: init 2024-09-16T05:22:50.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:50.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:22:50.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c40039c0 0x7f23d8134890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient(hunting): picked mon.noname-a con 0x7f23c40039c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:22:50.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23d81cd4b0 0x7f23d8134dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient(hunting): picked mon.noname-c con 0x7f23d81cd4b0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:50.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23d812f4f0 0x7f23d812f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient(hunting): picked mon.noname-b con 0x7f23d812f4f0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:22:50.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f23d80a1450 con 0x7f23c40039c0 2024-09-16T05:22:50.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f23d80a12d0 con 0x7f23d812f4f0 2024-09-16T05:22:50.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f23d8137e40 con 0x7f23d81cd4b0 2024-09-16T05:22:50.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:50.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient(hunting): authenticate will time out at 1854.901430s 2024-09-16T05:22:50.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23d81cd4b0 0x7f23d8134dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23d81cd4b0 0x7f23d8134dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:36478/0 (socket says 172.21.15.114:36478) 2024-09-16T05:22:50.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 1 -- 172.21.15.114:0/221118796 learned_addr learned my addr 172.21.15.114:0/221118796 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:50.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 10 monclient(hunting): get_auth_request con 0x7f23d81cd4b0 auth_method 0 2024-09-16T05:22:50.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d7fff640 1 --2- 172.21.15.114:0/221118796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c40039c0 0x7f23d8134890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d7fff640 10 monclient(hunting): get_auth_request con 0x7f23c40039c0 auth_method 0 2024-09-16T05:22:50.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23dcb04640 1 --2- 172.21.15.114:0/221118796 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23d812f4f0 0x7f23d812f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23dcb04640 10 monclient(hunting): get_auth_request con 0x7f23d812f4f0 auth_method 0 2024-09-16T05:22:50.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23dcb04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23dcb04640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23dcb04640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 10 monclient(hunting): handle_auth_done global_id 104186 payload 824 2024-09-16T05:22:50.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 1 -- 172.21.15.114:0/221118796 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23d812f4f0 msgr2=0x7f23d812f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 1 --2- 172.21.15.114:0/221118796 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23d812f4f0 0x7f23d812f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 1 -- 172.21.15.114:0/221118796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c40039c0 msgr2=0x7f23d8134890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 1 --2- 172.21.15.114:0/221118796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c40039c0 0x7f23d8134890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:22:50.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 1 monclient: found mon.noname-c 2024-09-16T05:22:50.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:50.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:22:50.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 1 -- 172.21.15.114:0/221118796 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23d81330a0 con 0x7f23d81cd4b0 2024-09-16T05:22:50.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23cf7fe640 1 --2- 172.21.15.114:0/221118796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23d81cd4b0 0x7f23d8134dd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f23c000d430 tx=0x7f23c000d8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:50.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 10 client.?.objecter ms_handle_connect 0x7f23d81cd4b0 2024-09-16T05:22:50.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:22:50.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 1 -- 172.21.15.114:0/221118796 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c0016020 con 0x7f23d81cd4b0 2024-09-16T05:22:50.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:22:50.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 10 monclient: dump: 2024-09-16T05:22:50.288 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.288 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.288 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.288 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.288 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.288 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.288 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.288 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.289 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 10 monclient: _finish_auth 0 2024-09-16T05:22:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 10 monclient: _check_auth_tickets 2024-09-16T05:22:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 1 -- 172.21.15.114:0/221118796 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23c0012800 con 0x7f23d81cd4b0 2024-09-16T05:22:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 5 monclient: authenticate success, global_id 104186 2024-09-16T05:22:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient: _renew_subs 2024-09-16T05:22:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 -- 172.21.15.114:0/221118796 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23d8131530 con 0x7f23d81cd4b0 2024-09-16T05:22:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 1 -- 172.21.15.114:0/221118796 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c0012bf0 con 0x7f23d81cd4b0 2024-09-16T05:22:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 10 monclient: dump: 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d67fc640 4 set_mon_vals no callback set 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 client.104186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient: _renew_subs 2024-09-16T05:22:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 -- 172.21.15.114:0/221118796 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f23d8073520 con 0x7f23d81cd4b0 2024-09-16T05:22:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 librados: init done 2024-09-16T05:22:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 1 -- 172.21.15.114:0/221118796 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f23c0019020 con 0x7f23d81cd4b0 2024-09-16T05:22:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23de567640 1 -- 172.21.15.114:0/221118796 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23d809c6a0 con 0x7f23d81cd4b0 2024-09-16T05:22:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 1 --2- 172.21.15.114:0/221118796 >> v2:172.21.15.149:6812/1185531135 conn(0x7f23b8037840 0x7f23b8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 1 -- 172.21.15.114:0/221118796 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(617..617 src has 1..617) ==== 5978+0+0 (secure 0 0 0) 0x7f23c0048c40 con 0x7f23d81cd4b0 2024-09-16T05:22:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 10 client.104186.objecter ms_dispatch 0x7f23d8000ba0 osd_map(617..617 src has 1..617) 2024-09-16T05:22:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 3 client.104186.objecter handle_osd_map got epochs [617,617] > 0 2024-09-16T05:22:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 3 client.104186.objecter handle_osd_map decoding full epoch 617 2024-09-16T05:22:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d57fa640 20 client.104186.objecter dump_active .. 0 homeless 2024-09-16T05:22:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d7fff640 1 --2- 172.21.15.114:0/221118796 >> v2:172.21.15.149:6812/1185531135 conn(0x7f23b8037840 0x7f23b8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.269+0000 7f23d7fff640 10 monclient: get_auth_request con 0x7f23b8037840 auth_method 0 2024-09-16T05:22:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.273+0000 7f23d57fa640 10 client.104186.objecter ms_handle_connect 0x7f23b8037840 2024-09-16T05:22:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.273+0000 7f23d7fff640 1 --2- 172.21.15.114:0/221118796 >> v2:172.21.15.149:6812/1185531135 conn(0x7f23b8037840 0x7f23b8039d00 secure :-1 s=READY pgs=3586 cs=0 l=1 rev1=1 crypto rx=0x7f23c80ab000 tx=0x7f23c80a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.277+0000 7f23d57fa640 1 -- 172.21.15.114:0/221118796 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f23c0020030 con 0x7f23d81cd4b0 2024-09-16T05:22:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.277+0000 7f23d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.277+0000 7f23d57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:22:50.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.501+0000 7f23de567640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:22:50.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.501+0000 7f23de567640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:22:50.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.501+0000 7f23de567640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:50.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.501+0000 7f23de567640 1 -- 172.21.15.114:0/221118796 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f23d8138050 con 0x7f23d81cd4b0 2024-09-16T05:22:50.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.501+0000 7f23d57fa640 1 -- 172.21.15.114:0/221118796 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f23c000f2f0 con 0x7f23d81cd4b0 2024-09-16T05:22:50.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.501+0000 7f23d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:22:50.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.501+0000 7f23d57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:22:50.500 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:22:50.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.501+0000 7f23de567640 10 librados: watch_flush enter 2024-09-16T05:22:50.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.501+0000 7f23de567640 10 librados: watch_flush exit 2024-09-16T05:22:50.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.501+0000 7f23de567640 20 client.104186.objecter shutdown clearing up homeless session... 2024-09-16T05:22:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.501+0000 7f23de567640 10 client.104186.objecter successfully canceled tick 2024-09-16T05:22:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.505+0000 7f23de567640 1 -- 172.21.15.114:0/221118796 >> v2:172.21.15.149:6812/1185531135 conn(0x7f23b8037840 msgr2=0x7f23b8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.505+0000 7f23de567640 1 --2- 172.21.15.114:0/221118796 >> v2:172.21.15.149:6812/1185531135 conn(0x7f23b8037840 0x7f23b8039d00 secure :-1 s=READY pgs=3586 cs=0 l=1 rev1=1 crypto rx=0x7f23c80ab000 tx=0x7f23c80a7070 comp rx=0 tx=0).stop 2024-09-16T05:22:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.505+0000 7f23de567640 10 monclient: shutdown 2024-09-16T05:22:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.505+0000 7f23de567640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.505+0000 7f23de567640 1 -- 172.21.15.114:0/221118796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23d81cd4b0 msgr2=0x7f23d8134dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.505+0000 7f23de567640 1 --2- 172.21.15.114:0/221118796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23d81cd4b0 0x7f23d8134dd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f23c000d430 tx=0x7f23c000d8f0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.505+0000 7f23de567640 1 -- 172.21.15.114:0/221118796 shutdown_connections 2024-09-16T05:22:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.505+0000 7f23de567640 1 --2- 172.21.15.114:0/221118796 >> v2:172.21.15.149:6812/1185531135 conn(0x7f23b8037840 0x7f23b8039d00 unknown :-1 s=CLOSED pgs=3586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.505+0000 7f23de567640 1 --2- 172.21.15.114:0/221118796 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23d812f4f0 0x7f23d812f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.505+0000 7f23de567640 1 --2- 172.21.15.114:0/221118796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23d81cd4b0 0x7f23d8134dd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.505+0000 7f23de567640 1 --2- 172.21.15.114:0/221118796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c40039c0 0x7f23d8134890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.505+0000 7f23de567640 1 -- 172.21.15.114:0/221118796 >> 172.21.15.114:0/221118796 conn(0x7f23d80a1fb0 msgr2=0x7f23d80a4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:50.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.505+0000 7f23de567640 1 -- 172.21.15.114:0/221118796 shutdown_connections 2024-09-16T05:22:50.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.505+0000 7f23de567640 1 -- 172.21.15.114:0/221118796 wait complete. 2024-09-16T05:22:50.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.505+0000 7f23de567640 1 librados: shutdown 2024-09-16T05:22:50.518 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:22:50.519 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:22:50.519 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:22:50.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:22:50.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 10 monclient: get_monmap_and_config 2024-09-16T05:22:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 10 monclient: build_initial_monmap 2024-09-16T05:22:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 10 monclient: monmap: 2024-09-16T05:22:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:22:50.640712+0000 2024-09-16T05:22:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:22:50.640712+0000 2024-09-16T05:22:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:22:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:22:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:22:50.636 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:22:50.636 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:22:50.636 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 1 Processor -- start 2024-09-16T05:22:50.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 1 -- start start 2024-09-16T05:22:50.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 10 monclient: init 2024-09-16T05:22:50.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:50.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T05:22:50.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7efd8809b6c0 0x7efd880a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 10 monclient(hunting): picked mon.noname-c con 0x7efd8809b6c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:22:50.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd880a1400 addr v1:172.21.15.114:6789/0 2024-09-16T05:22:50.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efd880a1810 0x7efd8813cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 10 monclient(hunting): picked mon.noname-a con 0x7efd880a1810 addr v2:172.21.15.114:3300/0 2024-09-16T05:22:50.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd8809d4e0 con 0x7efd880a1400 2024-09-16T05:22:50.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7efd8809e390 con 0x7efd880a1810 2024-09-16T05:22:50.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7efd8809ece0 con 0x7efd8809b6c0 2024-09-16T05:22:50.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:50.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8f30d640 10 monclient(hunting): authenticate will time out at 1855.270056s 2024-09-16T05:22:50.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efd880a1810 0x7efd8813cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8c8a8640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7efd880a1400 0x7efd88139640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34688/0 (socket says 172.21.15.114:34688) 2024-09-16T05:22:50.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8c8a8640 1 -- 172.21.15.114:0/2770751936 learned_addr learned my addr 172.21.15.114:0/2770751936 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:22:50.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 10 monclient(hunting): get_auth_request con 0x7efd880a1810 auth_method 0 2024-09-16T05:22:50.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d0a9640 1 --2- 172.21.15.114:0/2770751936 >> v2:172.21.15.149:3300/0 conn(0x7efd8809b6c0 0x7efd880a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d0a9640 10 monclient(hunting): get_auth_request con 0x7efd8809b6c0 auth_method 0 2024-09-16T05:22:50.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d0a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d0a9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d0a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 10 monclient(hunting): handle_auth_done global_id 74682 payload 274 2024-09-16T05:22:50.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 1 -- 172.21.15.114:0/2770751936 >> v2:172.21.15.149:3300/0 conn(0x7efd8809b6c0 msgr2=0x7efd880a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:22:50.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 1 --2- 172.21.15.114:0/2770751936 >> v2:172.21.15.149:3300/0 conn(0x7efd8809b6c0 0x7efd880a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 1 -- 172.21.15.114:0/2770751936 >> v1:172.21.15.114:6789/0 conn(0x7efd880a1400 legacy=0x7efd88139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 10 monclient: _finish_hunting 0 2024-09-16T05:22:50.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 1 monclient: found mon.noname-a 2024-09-16T05:22:50.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:50.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:22:50.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd8d8aa640 1 -- 172.21.15.114:0/2770751936 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8809f430 con 0x7efd880a1810 2024-09-16T05:22:50.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd7ffff640 1 -- 172.21.15.114:0/2770751936 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1103524094 0 0) 0x7efd8809d4e0 con 0x7efd880a1400 2024-09-16T05:22:50.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.637+0000 7efd7ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:22:50.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d8aa640 1 --2- 172.21.15.114:0/2770751936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd880a1810 0x7efd8813cd70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7efd880a2f70 tx=0x7efd8001c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a5b8632745bf9f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:50.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7ffff640 1 -- 172.21.15.114:0/2770751936 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd800b2070 con 0x7efd880a1810 2024-09-16T05:22:50.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:22:50.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7ffff640 10 monclient: dump: 2024-09-16T05:22:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.652 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7ffff640 10 monclient: _finish_auth 0 2024-09-16T05:22:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:22:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 5 monclient: authenticate success, global_id 74682 2024-09-16T05:22:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7ffff640 1 -- 172.21.15.114:0/2770751936 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd800a57a0 con 0x7efd880a1810 2024-09-16T05:22:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:22:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient: get_monmap_and_config success 2024-09-16T05:22:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 4 set_mon_vals no callback set 2024-09-16T05:22:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7ffff640 1 -- 172.21.15.114:0/2770751936 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd800a5b70 con 0x7efd880a1810 2024-09-16T05:22:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7ffff640 10 monclient: dump: 2024-09-16T05:22:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.654 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient: shutdown 2024-09-16T05:22:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 -- 172.21.15.114:0/2770751936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd880a1810 msgr2=0x7efd8813cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 --2- 172.21.15.114:0/2770751936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd880a1810 0x7efd8813cd70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7efd880a2f70 tx=0x7efd8001c7f0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 -- 172.21.15.114:0/2770751936 shutdown_connections 2024-09-16T05:22:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 --2- 172.21.15.114:0/2770751936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd880a1810 0x7efd8813cd70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 --2- 172.21.15.114:0/2770751936 >> v2:172.21.15.149:3300/0 conn(0x7efd8809b6c0 0x7efd880a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 -- 172.21.15.114:0/2770751936 >> 172.21.15.114:0/2770751936 conn(0x7efd880a1fb0 msgr2=0x7efd880a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 -- 172.21.15.114:0/2770751936 shutdown_connections 2024-09-16T05:22:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 -- 172.21.15.114:0/2770751936 wait complete. 2024-09-16T05:22:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient: build_initial_monmap 2024-09-16T05:22:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient: monmap: 2024-09-16T05:22:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:22:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:22:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:22:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:22:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:22:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:22:50.656 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 librados: starting msgr at 2024-09-16T05:22:50.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 librados: starting objecter 2024-09-16T05:22:50.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 Processor -- start 2024-09-16T05:22:50.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 -- start start 2024-09-16T05:22:50.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 librados: setting wanted keys 2024-09-16T05:22:50.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 librados: calling monclient init 2024-09-16T05:22:50.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient: init 2024-09-16T05:22:50.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:50.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:22:50.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd8809b6c0 0x7efd88098570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient(hunting): picked mon.noname-c con 0x7efd8809b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:50.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd880a1400 0x7efd880994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient(hunting): picked mon.noname-b con 0x7efd880a1400 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:22:50.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd880a1810 0x7efd88099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient(hunting): picked mon.noname-a con 0x7efd880a1810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:22:50.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd8809cad0 con 0x7efd880a1810 2024-09-16T05:22:50.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7efd8809db20 con 0x7efd880a1400 2024-09-16T05:22:50.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7efd8813fad0 con 0x7efd8809b6c0 2024-09-16T05:22:50.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:50.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient(hunting): authenticate will time out at 1855.274946s 2024-09-16T05:22:50.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd8809b6c0 0x7efd88098570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd8809b6c0 0x7efd88098570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:36490/0 (socket says 172.21.15.114:36490) 2024-09-16T05:22:50.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d8aa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd880a1810 0x7efd88099a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 1 -- 172.21.15.114:0/2173540239 learned_addr learned my addr 172.21.15.114:0/2173540239 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:50.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 10 monclient(hunting): get_auth_request con 0x7efd8809b6c0 auth_method 0 2024-09-16T05:22:50.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d8aa640 10 monclient(hunting): get_auth_request con 0x7efd880a1810 auth_method 0 2024-09-16T05:22:50.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d8aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d8aa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d8aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d8aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d8aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d8aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8c8a8640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd880a1400 0x7efd880994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 10 monclient(hunting): handle_auth_done global_id 104189 payload 824 2024-09-16T05:22:50.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 1 -- 172.21.15.114:0/2173540239 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd880a1400 msgr2=0x7efd880994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd880a1400 0x7efd880994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 1 -- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd880a1810 msgr2=0x7efd88099a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd880a1810 0x7efd88099a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 10 monclient: _finish_hunting 0 2024-09-16T05:22:50.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 1 monclient: found mon.noname-c 2024-09-16T05:22:50.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:50.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:22:50.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 1 -- 172.21.15.114:0/2173540239 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8809b0e0 con 0x7efd8809b6c0 2024-09-16T05:22:50.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d8aa640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd880a1810 0x7efd88099a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:22:50.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8d0a9640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd8809b6c0 0x7efd88098570 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efd8400d620 tx=0x7efd8400dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:50.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7dffb640 10 client.?.objecter ms_handle_connect 0x7efd8809b6c0 2024-09-16T05:22:50.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:22:50.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7dffb640 1 -- 172.21.15.114:0/2173540239 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8400b620 con 0x7efd8809b6c0 2024-09-16T05:22:50.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:22:50.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7dffb640 10 monclient: dump: 2024-09-16T05:22:50.662 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.662 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.662 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.663 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.663 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.663 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.663 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.663 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.663 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.663 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7dffb640 10 monclient: _finish_auth 0 2024-09-16T05:22:50.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:22:50.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:50.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 5 monclient: authenticate success, global_id 104189 2024-09-16T05:22:50.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7dffb640 1 -- 172.21.15.114:0/2173540239 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd84019070 con 0x7efd8809b6c0 2024-09-16T05:22:50.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient: _renew_subs 2024-09-16T05:22:50.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:50.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 -- 172.21.15.114:0/2173540239 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd88097b70 con 0x7efd8809b6c0 2024-09-16T05:22:50.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:50.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7dffb640 1 -- 172.21.15.114:0/2173540239 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8400b9e0 con 0x7efd8809b6c0 2024-09-16T05:22:50.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:22:50.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7dffb640 10 monclient: dump: 2024-09-16T05:22:50.664 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.664 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.664 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.664 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.664 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.664 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.664 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.664 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.665 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.665 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 client.104189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:22:50.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient: _renew_subs 2024-09-16T05:22:50.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:50.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 -- 172.21.15.114:0/2173540239 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efd88072b60 con 0x7efd8809b6c0 2024-09-16T05:22:50.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd8f30d640 1 librados: init done 2024-09-16T05:22:50.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.641+0000 7efd7f7fe640 4 set_mon_vals no callback set 2024-09-16T05:22:50.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8f30d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:50.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8f30d640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd8813cb00 0x7efd88099f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8f30d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8f30d640 1 -- 172.21.15.114:0/2173540239 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd88069390 con 0x7efd8813cb00 2024-09-16T05:22:50.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8c8a8640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd8813cb00 0x7efd88099f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 1 -- 172.21.15.114:0/2173540239 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7efd8400bb80 con 0x7efd8809b6c0 2024-09-16T05:22:50.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8c8a8640 10 monclient: get_auth_request con 0x7efd8813cb00 auth_method 0 2024-09-16T05:22:50.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8c8a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8c8a8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8c8a8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8c8a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8c8a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8c8a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8c8a8640 10 monclient(hunting): handle_auth_done global_id 74691 payload 824 2024-09-16T05:22:50.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8c8a8640 1 -- 172.21.15.114:0/2173540239 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd880976d0 con 0x7efd8813cb00 2024-09-16T05:22:50.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 1 --2- 172.21.15.114:0/2173540239 >> v2:172.21.15.149:6812/1185531135 conn(0x7efd6c037810 0x7efd6c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 1 -- 172.21.15.114:0/2173540239 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(617..617 src has 1..617) ==== 5978+0+0 (secure 0 0 0) 0x7efd84049480 con 0x7efd8809b6c0 2024-09-16T05:22:50.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 10 client.104189.objecter ms_dispatch 0x7efd88000ba0 osd_map(617..617 src has 1..617) 2024-09-16T05:22:50.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 3 client.104189.objecter handle_osd_map got epochs [617,617] > 0 2024-09-16T05:22:50.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 3 client.104189.objecter handle_osd_map decoding full epoch 617 2024-09-16T05:22:50.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8c8a8640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd8813cb00 0x7efd88099f60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7efd8809ab50 tx=0x7efd7800db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:50.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 20 client.104189.objecter dump_active .. 0 homeless 2024-09-16T05:22:50.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 10 client.104189.objecter ms_handle_connect 0x7efd8813cb00 2024-09-16T05:22:50.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 10 client.104189.objecter resend_mon_ops 2024-09-16T05:22:50.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 1 -- 172.21.15.114:0/2173540239 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd78007270 con 0x7efd8813cb00 2024-09-16T05:22:50.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:50.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 10 monclient: dump: 2024-09-16T05:22:50.667 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.667 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.667 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.668 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.668 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.668 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.668 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.668 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.668 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.668 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8d8aa640 1 --2- 172.21.15.114:0/2173540239 >> v2:172.21.15.149:6812/1185531135 conn(0x7efd6c037810 0x7efd6c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8d8aa640 10 monclient: get_auth_request con 0x7efd6c037810 auth_method 0 2024-09-16T05:22:50.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 1 -- 172.21.15.114:0/2173540239 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efd78017070 con 0x7efd8813cb00 2024-09-16T05:22:50.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:50.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:22:50.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 1 -- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd8813cb00 msgr2=0x7efd88099f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd8813cb00 0x7efd88099f60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7efd8809ab50 tx=0x7efd7800db80 comp rx=0 tx=0).stop 2024-09-16T05:22:50.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 1 -- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd8813cb00 msgr2=0x7efd88099f60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:22:50.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd8813cb00 0x7efd88099f60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd8d8aa640 1 --2- 172.21.15.114:0/2173540239 >> v2:172.21.15.149:6812/1185531135 conn(0x7efd6c037810 0x7efd6c039cd0 secure :-1 s=READY pgs=3587 cs=0 l=1 rev1=1 crypto rx=0x7efd800af000 tx=0x7efd800a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:50.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.645+0000 7efd7dffb640 10 client.104189.objecter ms_handle_connect 0x7efd6c037810 2024-09-16T05:22:50.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.649+0000 7efd8f30d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:22:50.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.649+0000 7efd8f30d640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd8808e010 0x7efd8808e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.649+0000 7efd8f30d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.649+0000 7efd8f30d640 1 -- 172.21.15.114:0/2173540239 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd8809c6a0 con 0x7efd8808e010 2024-09-16T05:22:50.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.649+0000 7efd8c8a8640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd8808e010 0x7efd8808e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.649+0000 7efd8c8a8640 10 monclient: get_auth_request con 0x7efd8808e010 auth_method 0 2024-09-16T05:22:50.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.649+0000 7efd8c8a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.649+0000 7efd8c8a8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.649+0000 7efd8c8a8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.649+0000 7efd8c8a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.649+0000 7efd8c8a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.649+0000 7efd8c8a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8c8a8640 10 monclient(hunting): handle_auth_done global_id 74694 payload 824 2024-09-16T05:22:50.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8c8a8640 1 -- 172.21.15.114:0/2173540239 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efd880a0e40 con 0x7efd8808e010 2024-09-16T05:22:50.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8c8a8640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd8808e010 0x7efd8808e420 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7efd78013000 tx=0x7efd78000f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:50.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd7dffb640 10 client.104189.objecter ms_handle_connect 0x7efd8808e010 2024-09-16T05:22:50.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd7dffb640 10 client.104189.objecter resend_mon_ops 2024-09-16T05:22:50.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd7dffb640 1 -- 172.21.15.114:0/2173540239 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd780059c0 con 0x7efd8808e010 2024-09-16T05:22:50.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd7dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:50.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd7dffb640 10 monclient: dump: 2024-09-16T05:22:50.671 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.671 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.671 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.671 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.671 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.671 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.671 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.671 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.671 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.671 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd7dffb640 1 -- 172.21.15.114:0/2173540239 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7efd7800b6a0 con 0x7efd8808e010 2024-09-16T05:22:50.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd7dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:22:50.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd7dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:22:50.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd7dffb640 1 -- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd8808e010 msgr2=0x7efd8808e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd7dffb640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd8808e010 0x7efd8808e420 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7efd78013000 tx=0x7efd78000f80 comp rx=0 tx=0).stop 2024-09-16T05:22:50.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd7dffb640 1 -- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd8808e010 msgr2=0x7efd8808e420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:22:50.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd7dffb640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd8808e010 0x7efd8808e420 secure :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7efd78013000 tx=0x7efd78000f80 comp rx=0 tx=0).stop 2024-09-16T05:22:50.672 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:22:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:22:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:22:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:22:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 62, 2024-09-16T05:22:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 529708, 2024-09-16T05:22:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:22:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:22:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:22:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:22:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:22:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:22:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:22:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:22:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:22:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:22:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:22:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:22:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:22:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:22:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:22:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:22:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:22:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:22:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:22:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:22:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:22:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:22:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:22:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:22:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:22:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:22:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:22:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:22:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:22:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:22:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:22:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:22:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:22:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:22:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:22:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:22:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:22:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:22:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:22:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:22:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:22:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:22:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:22:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:22:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:22:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:22:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:22:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:22:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:22:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:22:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:22:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:22:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:22:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:22:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:22:50.686 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:22:50.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8f30d640 10 librados: watch_flush enter 2024-09-16T05:22:50.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8f30d640 10 librados: watch_flush exit 2024-09-16T05:22:50.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8f30d640 20 client.104189.objecter shutdown clearing up homeless session... 2024-09-16T05:22:50.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8f30d640 10 client.104189.objecter successfully canceled tick 2024-09-16T05:22:50.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8f30d640 1 -- 172.21.15.114:0/2173540239 >> v2:172.21.15.149:6812/1185531135 conn(0x7efd6c037810 msgr2=0x7efd6c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8f30d640 1 --2- 172.21.15.114:0/2173540239 >> v2:172.21.15.149:6812/1185531135 conn(0x7efd6c037810 0x7efd6c039cd0 secure :-1 s=READY pgs=3587 cs=0 l=1 rev1=1 crypto rx=0x7efd800af000 tx=0x7efd800a7070 comp rx=0 tx=0).stop 2024-09-16T05:22:50.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8f30d640 10 monclient: shutdown 2024-09-16T05:22:50.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8f30d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:50.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8f30d640 1 -- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd8809b6c0 msgr2=0x7efd88098570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8f30d640 1 --2- 172.21.15.114:0/2173540239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd8809b6c0 0x7efd88098570 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efd8400d620 tx=0x7efd8400dae0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8d8aa640 1 -- 172.21.15.114:0/2173540239 reap_dead start 2024-09-16T05:22:50.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8d8aa640 1 -- 172.21.15.114:0/2173540239 reap_dead start 2024-09-16T05:22:50.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8f30d640 1 -- 172.21.15.114:0/2173540239 shutdown_connections 2024-09-16T05:22:50.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8f30d640 1 -- 172.21.15.114:0/2173540239 >> 172.21.15.114:0/2173540239 conn(0x7efd880a1fb0 msgr2=0x7efd88071280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:50.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8f30d640 1 -- 172.21.15.114:0/2173540239 shutdown_connections 2024-09-16T05:22:50.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8f30d640 1 -- 172.21.15.114:0/2173540239 wait complete. 2024-09-16T05:22:50.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.653+0000 7efd8f30d640 1 librados: shutdown 2024-09-16T05:22:50.689 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:22:50.689 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:22:50.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:22:50.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.805+0000 7f5624e0f640 10 monclient: get_monmap_and_config 2024-09-16T05:22:50.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.805+0000 7f5624e0f640 10 monclient: build_initial_monmap 2024-09-16T05:22:50.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.805+0000 7f5624e0f640 10 monclient: monmap: 2024-09-16T05:22:50.807 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:50.807 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.807 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:22:50.812130+0000 2024-09-16T05:22:50.807 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:22:50.812130+0000 2024-09-16T05:22:50.807 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:50.807 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.807 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:22:50.807 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:22:50.807 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:22:50.807 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:22:50.807 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:22:50.808 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 1 Processor -- start 2024-09-16T05:22:50.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 1 -- start start 2024-09-16T05:22:50.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 10 monclient: init 2024-09-16T05:22:50.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:50.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-16T05:22:50.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f562009b6c0 0x7f56200a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f562009b6c0 addr v2:172.21.15.114:3300/0 2024-09-16T05:22:50.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f56200a1400 0x7f56200a1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f56200a1400 addr v2:172.21.15.149:3300/0 2024-09-16T05:22:50.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5620139640 addr v1:172.21.15.114:6789/0 2024-09-16T05:22:50.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f562009d4e0 con 0x7f5620139640 2024-09-16T05:22:50.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f56200a1d50 con 0x7f562009b6c0 2024-09-16T05:22:50.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f562009ece0 con 0x7f56200a1400 2024-09-16T05:22:50.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:50.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 10 monclient(hunting): authenticate will time out at 1855.440986s 2024-09-16T05:22:50.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f562009b6c0 0x7f56200a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561ed9d640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5620139640 0x7f562013ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34704/0 (socket says 172.21.15.114:34704) 2024-09-16T05:22:50.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561ed9d640 1 -- 172.21.15.114:0/2842483114 learned_addr learned my addr 172.21.15.114:0/2842483114 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:22:50.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 10 monclient(hunting): get_auth_request con 0x7f562009b6c0 auth_method 0 2024-09-16T05:22:50.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5617fff640 1 --2- 172.21.15.114:0/2842483114 >> v2:172.21.15.149:3300/0 conn(0x7f56200a1400 0x7f56200a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5617fff640 10 monclient(hunting): get_auth_request con 0x7f56200a1400 auth_method 0 2024-09-16T05:22:50.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5617fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5617fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5617fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 10 monclient(hunting): handle_auth_done global_id 74697 payload 274 2024-09-16T05:22:50.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 1 -- 172.21.15.114:0/2842483114 >> v2:172.21.15.149:3300/0 conn(0x7f56200a1400 msgr2=0x7f56200a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:22:50.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 1 --2- 172.21.15.114:0/2842483114 >> v2:172.21.15.149:3300/0 conn(0x7f56200a1400 0x7f56200a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 1 -- 172.21.15.114:0/2842483114 >> v1:172.21.15.114:6789/0 conn(0x7f5620139640 legacy=0x7f562013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 10 monclient: _finish_hunting 0 2024-09-16T05:22:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 1 monclient: found mon.noname-a 2024-09-16T05:22:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:22:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 1 -- 172.21.15.114:0/2842483114 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f562009f430 con 0x7f562009b6c0 2024-09-16T05:22:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561dd9b640 1 -- 172.21.15.114:0/2842483114 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 298224494 0 0) 0x7f562009d4e0 con 0x7f5620139640 2024-09-16T05:22:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561dd9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:22:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561e59c640 1 --2- 172.21.15.114:0/2842483114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f562009b6c0 0x7f56200a0ec0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f562009c630 tx=0x7f561001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a5d343790aa3db5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561dd9b640 1 -- 172.21.15.114:0/2842483114 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56100b0070 con 0x7f562009b6c0 2024-09-16T05:22:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:22:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561dd9b640 10 monclient: dump: 2024-09-16T05:22:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.815 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561dd9b640 10 monclient: _finish_auth 0 2024-09-16T05:22:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561dd9b640 10 monclient: _check_auth_tickets 2024-09-16T05:22:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561dd9b640 1 -- 172.21.15.114:0/2842483114 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56100ab600 con 0x7f562009b6c0 2024-09-16T05:22:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 5 monclient: authenticate success, global_id 74697 2024-09-16T05:22:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:22:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 10 monclient: get_monmap_and_config success 2024-09-16T05:22:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561dd9b640 1 -- 172.21.15.114:0/2842483114 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56100a4e60 con 0x7f562009b6c0 2024-09-16T05:22:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f5624e0f640 4 set_mon_vals no callback set 2024-09-16T05:22:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.809+0000 7f561dd9b640 10 monclient: dump: 2024-09-16T05:22:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.817 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient: shutdown 2024-09-16T05:22:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 -- 172.21.15.114:0/2842483114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f562009b6c0 msgr2=0x7f56200a0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 --2- 172.21.15.114:0/2842483114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f562009b6c0 0x7f56200a0ec0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f562009c630 tx=0x7f561001c3d0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 -- 172.21.15.114:0/2842483114 shutdown_connections 2024-09-16T05:22:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 --2- 172.21.15.114:0/2842483114 >> v2:172.21.15.149:3300/0 conn(0x7f56200a1400 0x7f56200a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 --2- 172.21.15.114:0/2842483114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f562009b6c0 0x7f56200a0ec0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 -- 172.21.15.114:0/2842483114 >> 172.21.15.114:0/2842483114 conn(0x7f56200a1fb0 msgr2=0x7f56200a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 -- 172.21.15.114:0/2842483114 shutdown_connections 2024-09-16T05:22:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 -- 172.21.15.114:0/2842483114 wait complete. 2024-09-16T05:22:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient: build_initial_monmap 2024-09-16T05:22:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient: monmap: 2024-09-16T05:22:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:22:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:22:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:22:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:22:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:22:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:22:50.818 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 librados: starting msgr at 2024-09-16T05:22:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 librados: starting objecter 2024-09-16T05:22:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 Processor -- start 2024-09-16T05:22:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 -- start start 2024-09-16T05:22:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 librados: setting wanted keys 2024-09-16T05:22:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 librados: calling monclient init 2024-09-16T05:22:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient: init 2024-09-16T05:22:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:22:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f562009b6c0 0x7f5620098800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f562009b6c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:22:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f56200a1400 0x7f562008e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f56200a1400 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:22:50.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5620139640 0x7f562008e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f5620139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:50.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f562013f970 con 0x7f562009b6c0 2024-09-16T05:22:50.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f562009ddd0 con 0x7f56200a1400 2024-09-16T05:22:50.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f562013faf0 con 0x7f5620139640 2024-09-16T05:22:50.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:50.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient(hunting): authenticate will time out at 1855.446217s 2024-09-16T05:22:50.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f562009b6c0 0x7f5620098800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f562009b6c0 0x7f5620098800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33900/0 (socket says 172.21.15.114:33900) 2024-09-16T05:22:50.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 1 -- 172.21.15.114:0/3981072899 learned_addr learned my addr 172.21.15.114:0/3981072899 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:50.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 10 monclient(hunting): get_auth_request con 0x7f562009b6c0 auth_method 0 2024-09-16T05:22:50.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561ed9d640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5620139640 0x7f562008e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561ed9d640 10 monclient(hunting): get_auth_request con 0x7f5620139640 auth_method 0 2024-09-16T05:22:50.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5617fff640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f56200a1400 0x7f562008e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5617fff640 10 monclient(hunting): get_auth_request con 0x7f56200a1400 auth_method 0 2024-09-16T05:22:50.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5617fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5617fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5617fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 10 monclient(hunting): handle_auth_done global_id 74703 payload 824 2024-09-16T05:22:50.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 1 -- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5620139640 msgr2=0x7f562008e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5620139640 0x7f562008e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 1 -- 172.21.15.114:0/3981072899 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f56200a1400 msgr2=0x7f562008e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f56200a1400 0x7f562008e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 10 monclient: _finish_hunting 0 2024-09-16T05:22:50.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 1 monclient: found mon.noname-a 2024-09-16T05:22:50.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:50.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:22:50.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 1 -- 172.21.15.114:0/3981072899 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f562009c6a0 con 0x7f562009b6c0 2024-09-16T05:22:50.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561ed9d640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5620139640 0x7f562008e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:22:50.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561e59c640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f562009b6c0 0x7f5620098800 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f56100ae040 tx=0x7f5610019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:50.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5616ffd640 10 client.?.objecter ms_handle_connect 0x7f562009b6c0 2024-09-16T05:22:50.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5616ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:22:50.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5616ffd640 1 -- 172.21.15.114:0/3981072899 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56100b8020 con 0x7f562009b6c0 2024-09-16T05:22:50.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5616ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5616ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:22:50.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5616ffd640 10 monclient: dump: 2024-09-16T05:22:50.830 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.831 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.831 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.831 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.831 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.831 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.831 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.831 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.831 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.831 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5616ffd640 10 monclient: _finish_auth 0 2024-09-16T05:22:50.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5616ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:22:50.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5616ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:50.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 5 monclient: authenticate success, global_id 74703 2024-09-16T05:22:50.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5616ffd640 1 -- 172.21.15.114:0/3981072899 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56100b4410 con 0x7f562009b6c0 2024-09-16T05:22:50.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5616ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:50.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient: _renew_subs 2024-09-16T05:22:50.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5616ffd640 1 -- 172.21.15.114:0/3981072899 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56100b0040 con 0x7f562009b6c0 2024-09-16T05:22:50.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:22:50.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 -- 172.21.15.114:0/3981072899 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5620098d40 con 0x7f562009b6c0 2024-09-16T05:22:50.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f561d59a640 4 set_mon_vals no callback set 2024-09-16T05:22:50.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5616ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5616ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:50.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5616ffd640 10 monclient: dump: 2024-09-16T05:22:50.832 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.832 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.832 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.833 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.833 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.833 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.833 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.833 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.833 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.833 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 client.74703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:22:50.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient: _renew_subs 2024-09-16T05:22:50.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:22:50.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 -- 172.21.15.114:0/3981072899 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5620139d20 con 0x7f562009b6c0 2024-09-16T05:22:50.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 librados: init done 2024-09-16T05:22:50.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:50.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5620091ab0 0x7f562008ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.813+0000 7f5624e0f640 1 -- 172.21.15.114:0/3981072899 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f5620069390 con 0x7f5620091ab0 2024-09-16T05:22:50.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5617fff640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5620091ab0 0x7f562008ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5617fff640 10 monclient: get_auth_request con 0x7f5620091ab0 auth_method 0 2024-09-16T05:22:50.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5617fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5617fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5617fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 1 -- 172.21.15.114:0/3981072899 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f56100bb020 con 0x7f562009b6c0 2024-09-16T05:22:50.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5617fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5617fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5617fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5617fff640 10 monclient(hunting): handle_auth_done global_id 104195 payload 824 2024-09-16T05:22:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5617fff640 1 -- 172.21.15.114:0/3981072899 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f56200a0ec0 con 0x7f5620091ab0 2024-09-16T05:22:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 1 --2- 172.21.15.114:0/3981072899 >> v2:172.21.15.149:6812/1185531135 conn(0x7f5600037840 0x7f5600039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 1 -- 172.21.15.114:0/3981072899 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(618..618 src has 1..618) ==== 5973+0+0 (secure 0 0 0) 0x7f56100e2b00 con 0x7f562009b6c0 2024-09-16T05:22:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 10 client.74703.objecter ms_dispatch 0x7f5620000ba0 osd_map(618..618 src has 1..618) 2024-09-16T05:22:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 3 client.74703.objecter handle_osd_map got epochs [618,618] > 0 2024-09-16T05:22:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 3 client.74703.objecter handle_osd_map decoding full epoch 618 2024-09-16T05:22:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5617fff640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5620091ab0 0x7f562008ead0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5608006fd0 tx=0x7f560800f930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 20 client.74703.objecter dump_active .. 0 homeless 2024-09-16T05:22:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 10 client.74703.objecter ms_handle_connect 0x7f5620091ab0 2024-09-16T05:22:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 10 client.74703.objecter resend_mon_ops 2024-09-16T05:22:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 1 -- 172.21.15.114:0/3981072899 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5608019030 con 0x7f5620091ab0 2024-09-16T05:22:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:22:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 10 monclient: dump: 2024-09-16T05:22:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.836 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f561ed9d640 1 --2- 172.21.15.114:0/3981072899 >> v2:172.21.15.149:6812/1185531135 conn(0x7f5600037840 0x7f5600039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f561ed9d640 10 monclient: get_auth_request con 0x7f5600037840 auth_method 0 2024-09-16T05:22:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 1 -- 172.21.15.114:0/3981072899 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5608004020 con 0x7f5620091ab0 2024-09-16T05:22:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:22:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 1 -- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5620091ab0 msgr2=0x7f562008ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5620091ab0 0x7f562008ead0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5608006fd0 tx=0x7f560800f930 comp rx=0 tx=0).stop 2024-09-16T05:22:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 1 -- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5620091ab0 msgr2=0x7f562008ead0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:22:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5620091ab0 0x7f562008ead0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5608006fd0 tx=0x7f560800f930 comp rx=0 tx=0).stop 2024-09-16T05:22:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f5616ffd640 10 client.74703.objecter ms_handle_connect 0x7f5600037840 2024-09-16T05:22:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.817+0000 7f561ed9d640 1 --2- 172.21.15.114:0/3981072899 >> v2:172.21.15.149:6812/1185531135 conn(0x7f5600037840 0x7f5600039d00 secure :-1 s=READY pgs=3588 cs=0 l=1 rev1=1 crypto rx=0x7f56180012f0 tx=0x7f5618009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.821+0000 7f5624e0f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:22:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.821+0000 7f5624e0f640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56200709b0 0x7f5620070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.821+0000 7f5624e0f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.821+0000 7f5624e0f640 1 -- 172.21.15.114:0/3981072899 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f5620091e90 con 0x7f56200709b0 2024-09-16T05:22:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.821+0000 7f5617fff640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56200709b0 0x7f5620070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.821+0000 7f5617fff640 10 monclient: get_auth_request con 0x7f56200709b0 auth_method 0 2024-09-16T05:22:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.821+0000 7f5617fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.821+0000 7f5617fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.821+0000 7f5617fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.821+0000 7f5617fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.821+0000 7f5617fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.821+0000 7f5617fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5617fff640 10 monclient(hunting): handle_auth_done global_id 104198 payload 824 2024-09-16T05:22:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5617fff640 1 -- 172.21.15.114:0/3981072899 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f56200686f0 con 0x7f56200709b0 2024-09-16T05:22:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5617fff640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56200709b0 0x7f5620070dc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5608006fd0 tx=0x7f56080046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5616ffd640 10 client.74703.objecter ms_handle_connect 0x7f56200709b0 2024-09-16T05:22:50.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5616ffd640 10 client.74703.objecter resend_mon_ops 2024-09-16T05:22:50.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5616ffd640 1 -- 172.21.15.114:0/3981072899 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5608014ba0 con 0x7f56200709b0 2024-09-16T05:22:50.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5616ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5616ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:22:50.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5616ffd640 10 monclient: dump: 2024-09-16T05:22:50.839 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.839 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.839 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.839 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.839 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.839 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.839 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.839 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.839 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.840 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5616ffd640 1 -- 172.21.15.114:0/3981072899 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f5608014ba0 con 0x7f56200709b0 2024-09-16T05:22:50.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5616ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:22:50.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5616ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:22:50.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5616ffd640 1 -- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56200709b0 msgr2=0x7f5620070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5616ffd640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56200709b0 0x7f5620070dc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5608006fd0 tx=0x7f56080046d0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5616ffd640 1 -- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56200709b0 msgr2=0x7f5620070dc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:22:50.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5616ffd640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56200709b0 0x7f5620070dc0 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5608006fd0 tx=0x7f56080046d0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.840 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:22:50.841 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:22:50.841 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:22:50.841 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:22:50.841 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 62, 2024-09-16T05:22:50.841 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 146989, 2024-09-16T05:22:50.841 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:22:50.841 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:22:50.841 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:22:50.841 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:22:50.841 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:50.841 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 3, 2024-09-16T05:22:50.841 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:22:50.841 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:22:50.842 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:22:50.842 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:50.842 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:50.842 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:50.842 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:50.842 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:50.842 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:50.842 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:50.842 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:50.842 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:50.842 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:50.842 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:50.842 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:50.843 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:22:50.843 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:22:50.843 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:50.843 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:50.843 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:50.843 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:50.843 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:50.843 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:50.843 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:50.843 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:50.843 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:50.843 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:50.843 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:50.843 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:50.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.844 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:22:50.844 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:22:50.844 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:22:50.844 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:22:50.844 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:22:50.844 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:22:50.844 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:22:50.844 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:22:50.844 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:22:50.844 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:22:50.844 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:22:50.844 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:22:50.845 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:22:50.845 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:22:50.845 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:22:50.845 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:22:50.845 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:22:50.845 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:50.845 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:50.845 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:50.845 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:50.845 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:50.845 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:50.845 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:50.845 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:50.845 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:50.846 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:50.846 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:50.846 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:50.846 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:22:50.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.846 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:22:50.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.846 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:22:50.846 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:22:50.846 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:50.846 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:50.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.846 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:50.846 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:22:50.847 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:50.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.847 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.847 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:50.847 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:22:50.847 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:50.847 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:50.847 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:50.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.847 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:22:50.847 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:22:50.847 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:50.847 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:50.848 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:50.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.848 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.848 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:22:50.848 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:22:50.848 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:50.848 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:50.848 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.848 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:50.848 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:22:50.848 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:50.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.849 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.849 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:50.849 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:22:50.849 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:50.849 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:50.849 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:50.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.849 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:22:50.849 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:22:50.849 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:50.849 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:50.849 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:50.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.849 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.850 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:22:50.850 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:22:50.850 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:50.850 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:50.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.850 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:50.850 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:22:50.850 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:50.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.850 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:50.850 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:22:50.850 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:50.850 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:22:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:22:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:22:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:22:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:50.852 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:50.852 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:22:50.852 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:50.852 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:50.852 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:22:50.852 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.852 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:50.852 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:50.852 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:22:50.852 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:50.852 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:50.852 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:22:50.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.853 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:50.853 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:50.853 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 3 2024-09-16T05:22:50.853 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:50.853 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:50.853 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:22:50.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:50.853 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:50.853 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:50.853 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:22:50.853 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:50.853 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:50.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:50.854 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:22:50.854 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:22:50.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 10 librados: watch_flush enter 2024-09-16T05:22:50.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 10 librados: watch_flush exit 2024-09-16T05:22:50.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 20 client.74703.objecter shutdown clearing up homeless session... 2024-09-16T05:22:50.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 10 client.74703.objecter successfully canceled tick 2024-09-16T05:22:50.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 1 -- 172.21.15.114:0/3981072899 >> v2:172.21.15.149:6812/1185531135 conn(0x7f5600037840 msgr2=0x7f5600039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 1 --2- 172.21.15.114:0/3981072899 >> v2:172.21.15.149:6812/1185531135 conn(0x7f5600037840 0x7f5600039d00 secure :-1 s=READY pgs=3588 cs=0 l=1 rev1=1 crypto rx=0x7f56180012f0 tx=0x7f5618009040 comp rx=0 tx=0).stop 2024-09-16T05:22:50.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 10 monclient: shutdown 2024-09-16T05:22:50.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:50.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 1 -- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f562009b6c0 msgr2=0x7f5620098800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f562009b6c0 0x7f5620098800 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f56100ae040 tx=0x7f5610019170 comp rx=0 tx=0).stop 2024-09-16T05:22:50.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f561ed9d640 1 -- 172.21.15.114:0/3981072899 reap_dead start 2024-09-16T05:22:50.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 1 -- 172.21.15.114:0/3981072899 shutdown_connections 2024-09-16T05:22:50.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 1 --2- 172.21.15.114:0/3981072899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f562009b6c0 0x7f5620098800 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 1 -- 172.21.15.114:0/3981072899 >> 172.21.15.114:0/3981072899 conn(0x7f56200a1fb0 msgr2=0x7f562013b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:50.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 1 -- 172.21.15.114:0/3981072899 shutdown_connections 2024-09-16T05:22:50.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 1 -- 172.21.15.114:0/3981072899 wait complete. 2024-09-16T05:22:50.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.825+0000 7f5624e0f640 1 librados: shutdown 2024-09-16T05:22:50.857 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:22:50.857 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:22:50.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:22:50.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 10 monclient: get_monmap_and_config 2024-09-16T05:22:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 10 monclient: build_initial_monmap 2024-09-16T05:22:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 10 monclient: monmap: 2024-09-16T05:22:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:22:50.976558+0000 2024-09-16T05:22:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:22:50.976558+0000 2024-09-16T05:22:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:22:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:22:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:22:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:22:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:22:50.972 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 1 Processor -- start 2024-09-16T05:22:50.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 1 -- start start 2024-09-16T05:22:50.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 10 monclient: init 2024-09-16T05:22:50.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-16T05:22:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe9680735b0 0x7fe968136f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 10 monclient(hunting): picked mon.noname-c con 0x7fe9680735b0 addr v2:172.21.15.149:3300/0 2024-09-16T05:22:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9680a0170 addr v1:172.21.15.149:6789/0 2024-09-16T05:22:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe9680a05a0 0x7fe968075ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 10 monclient(hunting): picked mon.noname-a con 0x7fe9680a05a0 addr v2:172.21.15.114:3300/0 2024-09-16T05:22:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9680a1d90 con 0x7fe9680a0170 2024-09-16T05:22:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe968073990 con 0x7fe9680a05a0 2024-09-16T05:22:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fe968137de0 con 0x7fe9680735b0 2024-09-16T05:22:50.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:50.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96fd51640 10 monclient(hunting): authenticate will time out at 1855.605711s 2024-09-16T05:22:50.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe9680a05a0 0x7fe968075ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe9680a05a0 0x7fe968075ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33908/0 (socket says 172.21.15.114:33908) 2024-09-16T05:22:50.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 1 -- 172.21.15.114:0/1650369602 learned_addr learned my addr 172.21.15.114:0/1650369602 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:50.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 10 monclient(hunting): get_auth_request con 0x7fe9680a05a0 auth_method 0 2024-09-16T05:22:50.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 10 monclient(hunting): handle_auth_done global_id 74706 payload 274 2024-09-16T05:22:50.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 1 -- 172.21.15.114:0/1650369602 >> v2:172.21.15.149:3300/0 conn(0x7fe9680735b0 msgr2=0x7fe968136f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:22:50.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 1 --2- 172.21.15.114:0/1650369602 >> v2:172.21.15.149:3300/0 conn(0x7fe9680735b0 0x7fe968136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 1 -- 172.21.15.114:0/1650369602 >> v1:172.21.15.149:6789/0 conn(0x7fe9680a0170 legacy=0x7fe9680757d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:22:50.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 10 monclient: _finish_hunting 0 2024-09-16T05:22:50.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 1 monclient: found mon.noname-a 2024-09-16T05:22:50.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:50.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:22:50.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.973+0000 7fe96e2ee640 1 -- 172.21.15.114:0/1650369602 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe96809d950 con 0x7fe9680a05a0 2024-09-16T05:22:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96e2ee640 1 --2- 172.21.15.114:0/1650369602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9680a05a0 0x7fe968075ee0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe968076810 tx=0x7fe95801c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2430e7f3fcd590f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96caeb640 1 -- 172.21.15.114:0/1650369602 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9580b0070 con 0x7fe9680a05a0 2024-09-16T05:22:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96caeb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96caeb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:22:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96caeb640 10 monclient: dump: 2024-09-16T05:22:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.986 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96caeb640 10 monclient: _finish_auth 0 2024-09-16T05:22:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96caeb640 10 monclient: _check_auth_tickets 2024-09-16T05:22:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96caeb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96caeb640 1 -- 172.21.15.114:0/1650369602 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9580ab600 con 0x7fe9680a05a0 2024-09-16T05:22:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 5 monclient: authenticate success, global_id 74706 2024-09-16T05:22:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:22:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96caeb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 10 monclient: get_monmap_and_config success 2024-09-16T05:22:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96caeb640 1 -- 172.21.15.114:0/1650369602 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9580ab9d0 con 0x7fe9680a05a0 2024-09-16T05:22:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96caeb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 4 set_mon_vals no callback set 2024-09-16T05:22:50.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96caeb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:50.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96caeb640 10 monclient: dump: 2024-09-16T05:22:50.987 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.987 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.987 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.987 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.987 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.987 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.988 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 10 monclient: shutdown 2024-09-16T05:22:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 -- 172.21.15.114:0/1650369602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9680a05a0 msgr2=0x7fe968075ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 --2- 172.21.15.114:0/1650369602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9680a05a0 0x7fe968075ee0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe968076810 tx=0x7fe95801c370 comp rx=0 tx=0).stop 2024-09-16T05:22:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 -- 172.21.15.114:0/1650369602 shutdown_connections 2024-09-16T05:22:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 --2- 172.21.15.114:0/1650369602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9680a05a0 0x7fe968075ee0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 --2- 172.21.15.114:0/1650369602 >> v2:172.21.15.149:3300/0 conn(0x7fe9680735b0 0x7fe968136f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 -- 172.21.15.114:0/1650369602 >> 172.21.15.114:0/1650369602 conn(0x7fe9680a6680 msgr2=0x7fe9680a6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 -- 172.21.15.114:0/1650369602 shutdown_connections 2024-09-16T05:22:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 -- 172.21.15.114:0/1650369602 wait complete. 2024-09-16T05:22:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 10 monclient: build_initial_monmap 2024-09-16T05:22:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 10 monclient: monmap: 2024-09-16T05:22:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:22:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:22:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:22:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:22:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:22:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:22:50.989 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 librados: starting msgr at 2024-09-16T05:22:50.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 librados: starting objecter 2024-09-16T05:22:50.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 Processor -- start 2024-09-16T05:22:50.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 -- start start 2024-09-16T05:22:50.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 librados: setting wanted keys 2024-09-16T05:22:50.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 librados: calling monclient init 2024-09-16T05:22:50.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 10 monclient: init 2024-09-16T05:22:50.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:50.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:22:50.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9680735b0 0x7fe9681cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 10 monclient(hunting): picked mon.noname-a con 0x7fe9680735b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:22:50.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9680a0170 0x7fe968130bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 10 monclient(hunting): picked mon.noname-c con 0x7fe9680a0170 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:50.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe9680a05a0 0x7fe96812b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 10 monclient(hunting): picked mon.noname-b con 0x7fe9680a05a0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:22:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe9680765a0 con 0x7fe9680735b0 2024-09-16T05:22:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fe968076420 con 0x7fe9680a05a0 2024-09-16T05:22:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe968141a70 con 0x7fe9680a0170 2024-09-16T05:22:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96fd51640 10 monclient(hunting): authenticate will time out at 1855.611462s 2024-09-16T05:22:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96daed640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9680735b0 0x7fe9681cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96d2ec640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9680a0170 0x7fe968130bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96daed640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9680735b0 0x7fe9681cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33924/0 (socket says 172.21.15.114:33924) 2024-09-16T05:22:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96daed640 1 -- 172.21.15.114:0/749231514 learned_addr learned my addr 172.21.15.114:0/749231514 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96daed640 10 monclient(hunting): get_auth_request con 0x7fe9680735b0 auth_method 0 2024-09-16T05:22:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96daed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96daed640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96daed640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96d2ec640 10 monclient(hunting): get_auth_request con 0x7fe9680a0170 auth_method 0 2024-09-16T05:22:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96d2ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96d2ec640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96d2ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96d2ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96d2ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96d2ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96daed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96daed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96daed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96e2ee640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe9680a05a0 0x7fe96812b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96e2ee640 10 monclient(hunting): get_auth_request con 0x7fe9680a05a0 auth_method 0 2024-09-16T05:22:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96e2ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96e2ee640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96e2ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96daed640 10 monclient(hunting): handle_auth_done global_id 74709 payload 824 2024-09-16T05:22:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96daed640 1 -- 172.21.15.114:0/749231514 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9680a0170 msgr2=0x7fe968130bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96daed640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9680a0170 0x7fe968130bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96daed640 1 -- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe9680a05a0 msgr2=0x7fe96812b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.977+0000 7fe96daed640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe9680a05a0 0x7fe96812b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96daed640 10 monclient: _finish_hunting 0 2024-09-16T05:22:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96daed640 1 monclient: found mon.noname-a 2024-09-16T05:22:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96daed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96daed640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:22:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96daed640 1 -- 172.21.15.114:0/749231514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9681cfe10 con 0x7fe9680735b0 2024-09-16T05:22:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96d2ec640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9680a0170 0x7fe968130bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:22:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96daed640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9680735b0 0x7fe9681cef00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe96000ba50 tx=0x7fe96000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 client.?.objecter ms_handle_connect 0x7fe9680735b0 2024-09-16T05:22:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:22:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 1 -- 172.21.15.114:0/749231514 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe960019070 con 0x7fe9680735b0 2024-09-16T05:22:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:22:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 monclient: dump: 2024-09-16T05:22:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.995 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 monclient: _finish_auth 0 2024-09-16T05:22:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 monclient: _check_auth_tickets 2024-09-16T05:22:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 1 -- 172.21.15.114:0/749231514 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe96000cc00 con 0x7fe9680735b0 2024-09-16T05:22:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96fd51640 5 monclient: authenticate success, global_id 74709 2024-09-16T05:22:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96fd51640 10 monclient: _renew_subs 2024-09-16T05:22:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96fd51640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:22:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96fd51640 1 -- 172.21.15.114:0/749231514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9680a44e0 con 0x7fe9680735b0 2024-09-16T05:22:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 1 -- 172.21.15.114:0/749231514 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe960014440 con 0x7fe9680735b0 2024-09-16T05:22:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 monclient: dump: 2024-09-16T05:22:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:50.997 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:50.997 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:50.997 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:50.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96fd51640 10 client.74709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:22:50.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96fd51640 10 monclient: _renew_subs 2024-09-16T05:22:50.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96fd51640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:22:50.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96fd51640 1 -- 172.21.15.114:0/749231514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe96812e0a0 con 0x7fe9680735b0 2024-09-16T05:22:50.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe967fff640 4 set_mon_vals no callback set 2024-09-16T05:22:50.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96fd51640 1 librados: init done 2024-09-16T05:22:50.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96fd51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:50.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96fd51640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe968131110 0x7fe96812f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96fd51640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:50.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96fd51640 1 -- 172.21.15.114:0/749231514 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fe968137000 con 0x7fe968131110 2024-09-16T05:22:50.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 1 -- 172.21.15.114:0/749231514 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe96001c070 con 0x7fe9680735b0 2024-09-16T05:22:50.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96d2ec640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe968131110 0x7fe96812f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:50.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96d2ec640 10 monclient: get_auth_request con 0x7fe968131110 auth_method 0 2024-09-16T05:22:50.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96d2ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:50.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96d2ec640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:50.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96d2ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:50.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 1 --2- 172.21.15.114:0/749231514 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe954037840 0x7fe954039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:50.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 1 -- 172.21.15.114:0/749231514 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(618..618 src has 1..618) ==== 5973+0+0 (secure 0 0 0) 0x7fe960048d10 con 0x7fe9680735b0 2024-09-16T05:22:50.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 client.74709.objecter ms_dispatch 0x7fe968000ba0 osd_map(618..618 src has 1..618) 2024-09-16T05:22:50.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 3 client.74709.objecter handle_osd_map got epochs [618,618] > 0 2024-09-16T05:22:50.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 3 client.74709.objecter handle_osd_map decoding full epoch 618 2024-09-16T05:22:50.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96d2ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:50.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96d2ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:50.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 20 client.74709.objecter dump_active .. 0 homeless 2024-09-16T05:22:50.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96d2ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:50.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96e2ee640 1 --2- 172.21.15.114:0/749231514 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe954037840 0x7fe954039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:51.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96e2ee640 10 monclient: get_auth_request con 0x7fe954037840 auth_method 0 2024-09-16T05:22:51.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96d2ec640 10 monclient(hunting): handle_auth_done global_id 114136 payload 824 2024-09-16T05:22:51.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96d2ec640 1 -- 172.21.15.114:0/749231514 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe9680008e0 con 0x7fe968131110 2024-09-16T05:22:51.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 client.74709.objecter ms_handle_connect 0x7fe968131110 2024-09-16T05:22:51.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 client.74709.objecter resend_mon_ops 2024-09-16T05:22:51.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96d2ec640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe968131110 0x7fe96812f040 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe96812d980 tx=0x7fe95c00ff70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:51.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 1 -- 172.21.15.114:0/749231514 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe95c0052e0 con 0x7fe968131110 2024-09-16T05:22:51.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:51.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:22:51.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 monclient: dump: 2024-09-16T05:22:51.000 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:51.000 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:51.000 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:51.000 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:51.001 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:51.001 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:51.001 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:51.001 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:51.001 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:51.001 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:51.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 client.74709.objecter ms_handle_connect 0x7fe954037840 2024-09-16T05:22:51.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe96e2ee640 1 --2- 172.21.15.114:0/749231514 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe954037840 0x7fe954039d00 secure :-1 s=READY pgs=3589 cs=0 l=1 rev1=1 crypto rx=0x7fe9580ad000 tx=0x7fe9580a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:51.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 1 -- 172.21.15.114:0/749231514 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe95c018070 con 0x7fe968131110 2024-09-16T05:22:51.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:51.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:22:51.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 1 -- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe968131110 msgr2=0x7fe96812f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:51.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe968131110 0x7fe96812f040 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe96812d980 tx=0x7fe95c00ff70 comp rx=0 tx=0).stop 2024-09-16T05:22:51.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 1 -- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe968131110 msgr2=0x7fe96812f040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:22:51.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.981+0000 7fe9667fc640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe968131110 0x7fe96812f040 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe96812d980 tx=0x7fe95c00ff70 comp rx=0 tx=0).stop 2024-09-16T05:22:51.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96fd51640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:22:51.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96fd51640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe96812f580 0x7fe96812be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:51.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96fd51640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:51.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96fd51640 1 -- 172.21.15.114:0/749231514 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fe9681da250 con 0x7fe96812f580 2024-09-16T05:22:51.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96d2ec640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe96812f580 0x7fe96812be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:51.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96d2ec640 10 monclient: get_auth_request con 0x7fe96812f580 auth_method 0 2024-09-16T05:22:51.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96d2ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:51.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96d2ec640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:51.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96d2ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:51.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96d2ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:51.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96d2ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:51.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96d2ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:51.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96d2ec640 10 monclient(hunting): handle_auth_done global_id 114139 payload 824 2024-09-16T05:22:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96d2ec640 1 -- 172.21.15.114:0/749231514 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe9681da400 con 0x7fe96812f580 2024-09-16T05:22:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96d2ec640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe96812f580 0x7fe96812be50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe96812d980 tx=0x7fe95c001030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe9667fc640 10 client.74709.objecter ms_handle_connect 0x7fe96812f580 2024-09-16T05:22:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe9667fc640 10 client.74709.objecter resend_mon_ops 2024-09-16T05:22:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe9667fc640 1 -- 172.21.15.114:0/749231514 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe95c01ed30 con 0x7fe96812f580 2024-09-16T05:22:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe9667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe9667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:22:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe9667fc640 10 monclient: dump: 2024-09-16T05:22:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:51.008 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:51.008 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:51.008 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:51.008 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:51.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe9667fc640 1 -- 172.21.15.114:0/749231514 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7fe95c003600 con 0x7fe96812f580 2024-09-16T05:22:51.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe9667fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:22:51.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe9667fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:22:51.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe9667fc640 1 -- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe96812f580 msgr2=0x7fe96812be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:51.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe9667fc640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe96812f580 0x7fe96812be50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe96812d980 tx=0x7fe95c001030 comp rx=0 tx=0).stop 2024-09-16T05:22:51.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe9667fc640 1 -- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe96812f580 msgr2=0x7fe96812be50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:22:51.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe9667fc640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe96812f580 0x7fe96812be50 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe96812d980 tx=0x7fe95c001030 comp rx=0 tx=0).stop 2024-09-16T05:22:51.009 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:22:51.009 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:22:51.009 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:22:51.009 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:22:51.009 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 62, 2024-09-16T05:22:51.009 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 18211, 2024-09-16T05:22:51.009 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:22:51.009 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:22:51.009 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:22:51.009 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:22:51.009 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:51.009 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 3, 2024-09-16T05:22:51.009 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:22:51.010 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:22:51.010 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:22:51.010 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:51.010 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:51.010 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:51.010 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:51.010 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:51.010 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:51.010 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:51.010 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:51.010 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:51.010 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:51.010 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:22:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:22:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:51.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:51.012 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:22:51.012 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:22:51.012 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:22:51.012 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:22:51.012 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:22:51.012 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:22:51.012 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:22:51.012 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:22:51.012 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:22:51.012 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:22:51.012 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:22:51.012 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:22:51.013 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:22:51.013 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:22:51.013 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:22:51.013 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:22:51.013 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:22:51.013 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:22:51.013 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:22:51.013 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:22:51.013 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:22:51.013 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:22:51.013 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:22:51.013 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:22:51.013 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:22:51.013 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:22:51.014 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:22:51.014 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:22:51.014 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:51.014 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:22:51.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:51.014 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:22:51.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:51.014 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:22:51.014 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:22:51.014 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:51.014 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:51.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:51.014 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:51.014 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:22:51.014 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:51.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:51.015 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:51.015 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:51.015 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:22:51.015 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:51.015 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:51.015 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:51.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:51.015 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:22:51.015 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:22:51.015 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:51.015 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:51.015 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:51.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:51.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:51.016 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:22:51.016 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:22:51.016 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:51.016 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:51.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:51.016 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:51.016 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:22:51.016 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:51.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:51.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:51.016 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:51.016 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:22:51.016 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:51.017 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:51.017 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:51.017 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:51.017 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:22:51.017 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:22:51.017 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:51.017 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:51.017 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:51.017 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:51.017 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:51.017 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:22:51.017 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:22:51.017 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:22:51.017 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:22:51.018 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:51.018 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:22:51.018 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:22:51.018 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:51.018 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:51.018 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:51.018 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:22:51.018 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:22:51.018 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:22:51.018 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:51.018 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:51.018 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:51.018 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:22:51.018 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:22:51.019 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:22:51.019 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:22:51.019 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:22:51.019 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:51.019 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:51.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:51.019 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:22:51.019 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:22:51.019 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:51.019 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:51.019 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:51.019 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:22:51.019 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:51.019 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:22:51.020 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:22:51.020 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:22:51.020 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:22:51.020 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:22:51.020 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:22:51.020 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:22:51.020 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:22:51.020 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:22:51.020 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:22:51.020 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:22:51.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96fd51640 10 librados: watch_flush enter 2024-09-16T05:22:51.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96fd51640 10 librados: watch_flush exit 2024-09-16T05:22:51.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96fd51640 20 client.74709.objecter shutdown clearing up homeless session... 2024-09-16T05:22:51.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96fd51640 10 client.74709.objecter successfully canceled tick 2024-09-16T05:22:51.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96fd51640 1 -- 172.21.15.114:0/749231514 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe954037840 msgr2=0x7fe954039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:51.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96fd51640 1 --2- 172.21.15.114:0/749231514 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe954037840 0x7fe954039d00 secure :-1 s=READY pgs=3589 cs=0 l=1 rev1=1 crypto rx=0x7fe9580ad000 tx=0x7fe9580a6000 comp rx=0 tx=0).stop 2024-09-16T05:22:51.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96fd51640 10 monclient: shutdown 2024-09-16T05:22:51.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96fd51640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:51.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96fd51640 1 -- 172.21.15.114:0/749231514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9680735b0 msgr2=0x7fe9681cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:51.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.989+0000 7fe96fd51640 1 --2- 172.21.15.114:0/749231514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9680735b0 0x7fe9681cef00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe96000ba50 tx=0x7fe96000bf10 comp rx=0 tx=0).stop 2024-09-16T05:22:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.993+0000 7fe96e2ee640 1 -- 172.21.15.114:0/749231514 reap_dead start 2024-09-16T05:22:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.993+0000 7fe96e2ee640 1 -- 172.21.15.114:0/749231514 reap_dead start 2024-09-16T05:22:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.993+0000 7fe96fd51640 1 -- 172.21.15.114:0/749231514 shutdown_connections 2024-09-16T05:22:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.993+0000 7fe96fd51640 1 -- 172.21.15.114:0/749231514 >> 172.21.15.114:0/749231514 conn(0x7fe9680a6680 msgr2=0x7fe96809a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.993+0000 7fe96fd51640 1 -- 172.21.15.114:0/749231514 shutdown_connections 2024-09-16T05:22:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.993+0000 7fe96fd51640 1 -- 172.21.15.114:0/749231514 wait complete. 2024-09-16T05:22:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:50.993+0000 7fe96fd51640 1 librados: shutdown 2024-09-16T05:22:51.023 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:22:51.024 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T05:22:51.024 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T05:22:51.024 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T05:22:51.024 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:22:51.024 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T05:22:51.024 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:22:51.043 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T05:22:51.043 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:22:51.043 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:22:51.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 10 monclient: get_monmap_and_config 2024-09-16T05:22:51.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 10 monclient: build_initial_monmap 2024-09-16T05:22:51.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 10 monclient: monmap: 2024-09-16T05:22:51.155 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:51.156 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:51.156 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:22:51.161455+0000 2024-09-16T05:22:51.156 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:22:51.161455+0000 2024-09-16T05:22:51.156 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:51.156 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:51.156 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:22:51.156 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:22:51.156 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:22:51.156 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:22:51.156 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:22:51.156 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:51.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 1 Processor -- start 2024-09-16T05:22:51.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 1 -- start start 2024-09-16T05:22:51.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 10 monclient: init 2024-09-16T05:22:51.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:51.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-16T05:22:51.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d10137730 addr v1:172.21.15.114:6789/0 2024-09-16T05:22:51.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7d100a5d00 0x7f7d1013c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:51.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 10 monclient(hunting): picked mon.noname-a con 0x7f7d100a5d00 addr v2:172.21.15.114:3300/0 2024-09-16T05:22:51.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d1013ca00 0x7f7d1013edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:51.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 10 monclient(hunting): picked mon.noname-b con 0x7f7d1013ca00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:51.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d100a55e0 con 0x7f7d10137730 2024-09-16T05:22:51.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:51.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f7d1009c450 con 0x7f7d100a5d00 2024-09-16T05:22:51.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:51.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7d100a3710 con 0x7f7d1013ca00 2024-09-16T05:22:51.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:51.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d172f8640 10 monclient(hunting): authenticate will time out at 1855.790661s 2024-09-16T05:22:51.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7d100a5d00 0x7f7d1013c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:51.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d15094640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7d10137730 0x7f7d10137b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34714/0 (socket says 172.21.15.114:34714) 2024-09-16T05:22:51.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7d100a5d00 0x7f7d1013c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33934/0 (socket says 172.21.15.114:33934) 2024-09-16T05:22:51.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d15094640 1 -- 172.21.15.114:0/2495395973 learned_addr learned my addr 172.21.15.114:0/2495395973 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:22:51.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 10 monclient(hunting): get_auth_request con 0x7f7d100a5d00 auth_method 0 2024-09-16T05:22:51.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:51.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:51.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:51.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d15895640 1 --2- 172.21.15.114:0/2495395973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d1013ca00 0x7f7d1013edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:51.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d15895640 10 monclient(hunting): get_auth_request con 0x7f7d1013ca00 auth_method 0 2024-09-16T05:22:51.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d15895640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:51.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d15895640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:51.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d15895640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:51.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:51.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:51.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:51.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d15895640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:51.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d15895640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:51.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d15895640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:51.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 10 monclient(hunting): handle_auth_done global_id 74712 payload 274 2024-09-16T05:22:51.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 1 -- 172.21.15.114:0/2495395973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d1013ca00 msgr2=0x7f7d1013edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:22:51.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 1 --2- 172.21.15.114:0/2495395973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d1013ca00 0x7f7d1013edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:51.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 1 -- 172.21.15.114:0/2495395973 >> v1:172.21.15.114:6789/0 conn(0x7f7d10137730 legacy=0x7f7d10137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:51.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 10 monclient: _finish_hunting 0 2024-09-16T05:22:51.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 1 monclient: found mon.noname-a 2024-09-16T05:22:51.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:51.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:22:51.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d07fff640 1 -- 172.21.15.114:0/2495395973 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1295922651 0 0) 0x7f7d100a55e0 con 0x7f7d10137730 2024-09-16T05:22:51.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d14893640 1 -- 172.21.15.114:0/2495395973 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1009be00 con 0x7f7d100a5d00 2024-09-16T05:22:51.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.157+0000 7f7d07fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:22:51.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d14893640 1 --2- 172.21.15.114:0/2495395973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d100a5d00 0x7f7d1013c4c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0097c0 tx=0x7f7d0c01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8469f4ea7fb1cf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:51.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d07fff640 1 -- 172.21.15.114:0/2495395973 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c0b0070 con 0x7f7d100a5d00 2024-09-16T05:22:51.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:51.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d07fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:22:51.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d07fff640 10 monclient: dump: 2024-09-16T05:22:51.169 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:51.169 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:51.169 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:51.169 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:51.169 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:51.169 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:51.169 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:51.169 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:51.169 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:51.169 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:51.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d07fff640 10 monclient: _finish_auth 0 2024-09-16T05:22:51.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d07fff640 10 monclient: _check_auth_tickets 2024-09-16T05:22:51.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:51.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d07fff640 1 -- 172.21.15.114:0/2495395973 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d0c0ab600 con 0x7f7d100a5d00 2024-09-16T05:22:51.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d07fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:51.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d07fff640 1 -- 172.21.15.114:0/2495395973 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c0ab9d0 con 0x7f7d100a5d00 2024-09-16T05:22:51.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 5 monclient: authenticate success, global_id 74712 2024-09-16T05:22:51.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 10 monclient: get_monmap_and_config success 2024-09-16T05:22:51.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 4 set_mon_vals no callback set 2024-09-16T05:22:51.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:51.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d07fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:51.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d07fff640 10 monclient: dump: 2024-09-16T05:22:51.170 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:51.170 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:51.170 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:51.170 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:51.170 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:51.171 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:51.171 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:51.171 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:51.171 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:51.171 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:51.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 10 monclient: shutdown 2024-09-16T05:22:51.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:51.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 -- 172.21.15.114:0/2495395973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d100a5d00 msgr2=0x7f7d1013c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:51.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 --2- 172.21.15.114:0/2495395973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d100a5d00 0x7f7d1013c4c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0097c0 tx=0x7f7d0c01c360 comp rx=0 tx=0).stop 2024-09-16T05:22:51.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 -- 172.21.15.114:0/2495395973 shutdown_connections 2024-09-16T05:22:51.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 --2- 172.21.15.114:0/2495395973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d1013ca00 0x7f7d1013edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:51.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 --2- 172.21.15.114:0/2495395973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d100a5d00 0x7f7d1013c4c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:51.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 -- 172.21.15.114:0/2495395973 >> 172.21.15.114:0/2495395973 conn(0x7f7d100a9c70 msgr2=0x7f7d100aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:51.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 -- 172.21.15.114:0/2495395973 shutdown_connections 2024-09-16T05:22:51.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 -- 172.21.15.114:0/2495395973 wait complete. 2024-09-16T05:22:51.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 10 monclient: build_initial_monmap 2024-09-16T05:22:51.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 10 monclient: monmap: 2024-09-16T05:22:51.172 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:51.172 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:22:51.172 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:22:51.172 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:22:51.172 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:51.172 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:51.172 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:22:51.172 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:22:51.172 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:22:51.172 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:51.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 librados: starting msgr at 2024-09-16T05:22:51.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 librados: starting objecter 2024-09-16T05:22:51.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 Processor -- start 2024-09-16T05:22:51.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 -- start start 2024-09-16T05:22:51.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 librados: setting wanted keys 2024-09-16T05:22:51.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 librados: calling monclient init 2024-09-16T05:22:51.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 10 monclient: init 2024-09-16T05:22:51.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:51.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:22:51.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d100a5d00 0x7f7d101cf080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:51.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 10 monclient(hunting): picked mon.noname-b con 0x7f7d100a5d00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:22:51.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d10137730 0x7f7d10136380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:51.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 10 monclient(hunting): picked mon.noname-a con 0x7f7d10137730 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:22:51.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d1013ca00 0x7f7d1009fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:51.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 10 monclient(hunting): picked mon.noname-c con 0x7f7d1013ca00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:51.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:51.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7d10141cd0 con 0x7f7d10137730 2024-09-16T05:22:51.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:51.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f7d10141b50 con 0x7f7d100a5d00 2024-09-16T05:22:51.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:51.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7d10141e50 con 0x7f7d1013ca00 2024-09-16T05:22:51.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:51.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d172f8640 10 monclient(hunting): authenticate will time out at 1855.795800s 2024-09-16T05:22:51.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d15895640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d1013ca00 0x7f7d1009fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:51.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d15094640 1 -- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d100a5d00 msgr2=0x7f7d101cf080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:22:51.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d15895640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d1013ca00 0x7f7d1009fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:36530/0 (socket says 172.21.15.114:36530) 2024-09-16T05:22:51.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d14893640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d10137730 0x7f7d10136380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:51.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d15094640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d100a5d00 0x7f7d101cf080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:22:51.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d15895640 1 -- 172.21.15.114:0/1919026137 learned_addr learned my addr 172.21.15.114:0/1919026137 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:51.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d15895640 10 monclient(hunting): get_auth_request con 0x7f7d1013ca00 auth_method 0 2024-09-16T05:22:51.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d15895640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:51.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d15895640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:51.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d15895640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:51.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d14893640 10 monclient(hunting): get_auth_request con 0x7f7d10137730 auth_method 0 2024-09-16T05:22:51.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d14893640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:51.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d14893640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:51.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.161+0000 7f7d14893640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d15895640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d15895640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d15895640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d14893640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d14893640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d14893640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d14893640 10 monclient(hunting): handle_auth_done global_id 74718 payload 824 2024-09-16T05:22:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d14893640 1 -- 172.21.15.114:0/1919026137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d1013ca00 msgr2=0x7f7d1009fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d14893640 1 --2- 172.21.15.114:0/1919026137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d1013ca00 0x7f7d1009fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d14893640 1 -- 172.21.15.114:0/1919026137 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d100a5d00 msgr2=0x7f7d101cf080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:22:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d14893640 1 --2- 172.21.15.114:0/1919026137 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d100a5d00 0x7f7d101cf080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d14893640 10 monclient: _finish_hunting 0 2024-09-16T05:22:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d14893640 1 monclient: found mon.noname-a 2024-09-16T05:22:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d14893640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:51.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d14893640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:22:51.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d14893640 1 -- 172.21.15.114:0/1919026137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1013e6a0 con 0x7f7d10137730 2024-09-16T05:22:51.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d15895640 1 --2- 172.21.15.114:0/1919026137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d1013ca00 0x7f7d1009fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:22:51.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d14893640 1 --2- 172.21.15.114:0/1919026137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d10137730 0x7f7d10136380 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0ab000 tx=0x7f7d0c019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:51.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 10 client.?.objecter ms_handle_connect 0x7f7d10137730 2024-09-16T05:22:51.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:22:51.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 1 -- 172.21.15.114:0/1919026137 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c0b8020 con 0x7f7d10137730 2024-09-16T05:22:51.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:51.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:22:51.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 10 monclient: dump: 2024-09-16T05:22:51.177 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:51.177 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:51.177 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:51.178 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 10 monclient: _finish_auth 0 2024-09-16T05:22:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:22:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 1 -- 172.21.15.114:0/1919026137 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d0c0b4410 con 0x7f7d10137730 2024-09-16T05:22:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d172f8640 5 monclient: authenticate success, global_id 74718 2024-09-16T05:22:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d172f8640 10 monclient: _renew_subs 2024-09-16T05:22:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d172f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:22:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d172f8640 1 -- 172.21.15.114:0/1919026137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d10120a30 con 0x7f7d10137730 2024-09-16T05:22:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 1 -- 172.21.15.114:0/1919026137 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c0b0040 con 0x7f7d10137730 2024-09-16T05:22:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 10 monclient: dump: 2024-09-16T05:22:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:51.180 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d077fe640 4 set_mon_vals no callback set 2024-09-16T05:22:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d172f8640 10 client.74718.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:22:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d172f8640 10 monclient: _renew_subs 2024-09-16T05:22:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d172f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:22:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d172f8640 1 -- 172.21.15.114:0/1919026137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d1011ff50 con 0x7f7d10137730 2024-09-16T05:22:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d172f8640 1 librados: init done 2024-09-16T05:22:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d172f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d172f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d172f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:22:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d172f8640 1 -- 172.21.15.114:0/1919026137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d10137b10 con 0x7f7d10137730 2024-09-16T05:22:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 1 -- 172.21.15.114:0/1919026137 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7d0c0bb020 con 0x7f7d10137730 2024-09-16T05:22:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 1 --2- 172.21.15.114:0/1919026137 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7cf40377c0 0x7f7cf4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 1 -- 172.21.15.114:0/1919026137 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(618..618 src has 1..618) ==== 5973+0+0 (secure 0 0 0) 0x7f7d0c0e2ad0 con 0x7f7d10137730 2024-09-16T05:22:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 10 client.74718.objecter ms_dispatch 0x7f7d10000ba0 osd_map(618..618 src has 1..618) 2024-09-16T05:22:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 3 client.74718.objecter handle_osd_map got epochs [618,618] > 0 2024-09-16T05:22:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 3 client.74718.objecter handle_osd_map decoding full epoch 618 2024-09-16T05:22:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 20 client.74718.objecter dump_active .. 0 homeless 2024-09-16T05:22:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d15094640 1 --2- 172.21.15.114:0/1919026137 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7cf40377c0 0x7f7cf4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d15094640 10 monclient: get_auth_request con 0x7f7cf40377c0 auth_method 0 2024-09-16T05:22:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d05ffb640 10 client.74718.objecter ms_handle_connect 0x7f7cf40377c0 2024-09-16T05:22:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.165+0000 7f7d15094640 1 --2- 172.21.15.114:0/1919026137 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7cf40377c0 0x7f7cf4039c80 secure :-1 s=READY pgs=3590 cs=0 l=1 rev1=1 crypto rx=0x7f7d00004820 tx=0x7f7d00007ad0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:51.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.169+0000 7f7d05ffb640 1 -- 172.21.15.114:0/1919026137 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7d0c0c2030 con 0x7f7d10137730 2024-09-16T05:22:51.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.169+0000 7f7d05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:51.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.169+0000 7f7d05ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 1 -- 172.21.15.114:0/1919026137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7d10136f80 con 0x7f7d10137730 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d05ffb640 1 -- 172.21.15.114:0/1919026137 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f7d0c016070 con 0x7f7d10137730 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d05ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 10 librados: watch_flush enter 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 10 librados: watch_flush exit 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 20 client.74718.objecter shutdown clearing up homeless session... 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 10 client.74718.objecter successfully canceled tick 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 1 -- 172.21.15.114:0/1919026137 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7cf40377c0 msgr2=0x7f7cf4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 1 --2- 172.21.15.114:0/1919026137 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7cf40377c0 0x7f7cf4039c80 secure :-1 s=READY pgs=3590 cs=0 l=1 rev1=1 crypto rx=0x7f7d00004820 tx=0x7f7d00007ad0 comp rx=0 tx=0).stop 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 10 monclient: shutdown 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 1 -- 172.21.15.114:0/1919026137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d10137730 msgr2=0x7f7d10136380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 1 --2- 172.21.15.114:0/1919026137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d10137730 0x7f7d10136380 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0ab000 tx=0x7f7d0c019100 comp rx=0 tx=0).stop 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 1 -- 172.21.15.114:0/1919026137 shutdown_connections 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 1 --2- 172.21.15.114:0/1919026137 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7cf40377c0 0x7f7cf4039c80 unknown :-1 s=CLOSED pgs=3590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 1 --2- 172.21.15.114:0/1919026137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d10137730 0x7f7d10136380 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 1 --2- 172.21.15.114:0/1919026137 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d1013ca00 0x7f7d1009fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 1 --2- 172.21.15.114:0/1919026137 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d100a5d00 0x7f7d101cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 1 -- 172.21.15.114:0/1919026137 >> 172.21.15.114:0/1919026137 conn(0x7f7d100a9c70 msgr2=0x7f7d1013df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 1 -- 172.21.15.114:0/1919026137 shutdown_connections 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 1 -- 172.21.15.114:0/1919026137 wait complete. 2024-09-16T05:22:51.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:51.385+0000 7f7d172f8640 1 librados: shutdown 2024-09-16T05:22:54.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:22:54.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 10 monclient: get_monmap_and_config 2024-09-16T05:22:54.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 10 monclient: build_initial_monmap 2024-09-16T05:22:54.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 10 monclient: monmap: 2024-09-16T05:22:54.525 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:54.525 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:54.525 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:22:54.530411+0000 2024-09-16T05:22:54.525 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:22:54.530411+0000 2024-09-16T05:22:54.525 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:54.525 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:54.525 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:22:54.525 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:22:54.525 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:22:54.525 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:22:54.525 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:22:54.525 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:54.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 1 Processor -- start 2024-09-16T05:22:54.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 1 -- start start 2024-09-16T05:22:54.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 10 monclient: init 2024-09-16T05:22:54.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-16T05:22:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f724409b6c0 0x7f72440a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 10 monclient(hunting): picked mon.noname-c con 0x7f724409b6c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:22:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f72440a1400 0x7f72440a1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 10 monclient(hunting): picked mon.noname-a con 0x7f72440a1400 addr v2:172.21.15.114:3300/0 2024-09-16T05:22:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7244139640 addr v1:172.21.15.114:6789/0 2024-09-16T05:22:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f724409d4e0 con 0x7f7244139640 2024-09-16T05:22:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f72440a1d50 con 0x7f72440a1400 2024-09-16T05:22:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f724409ece0 con 0x7f724409b6c0 2024-09-16T05:22:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:54.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7249928640 10 monclient(hunting): authenticate will time out at 1859.159554s 2024-09-16T05:22:54.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f72427fc640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f72440a1400 0x7f72440a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:54.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f72437fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7244139640 0x7f724413ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34718/0 (socket says 172.21.15.114:34718) 2024-09-16T05:22:54.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f72427fc640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f72440a1400 0x7f72440a1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33954/0 (socket says 172.21.15.114:33954) 2024-09-16T05:22:54.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f72437fe640 1 -- 172.21.15.114:0/150234992 learned_addr learned my addr 172.21.15.114:0/150234992 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:22:54.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f72427fc640 10 monclient(hunting): get_auth_request con 0x7f72440a1400 auth_method 0 2024-09-16T05:22:54.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f72427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:54.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f72427fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:54.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f72427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:54.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7242ffd640 1 -- >> v2:172.21.15.149:3300/0 conn(0x7f724409b6c0 msgr2=0x7f72440a0ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:22:54.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f7242ffd640 1 --2- 172.21.15.114:0/150234992 >> v2:172.21.15.149:3300/0 conn(0x7f724409b6c0 0x7f72440a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:22:54.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f72427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:54.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f72427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:54.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.525+0000 7f72427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:54.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.529+0000 7f72427fc640 10 monclient(hunting): handle_auth_done global_id 74721 payload 274 2024-09-16T05:22:54.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.529+0000 7f72427fc640 1 -- 172.21.15.114:0/150234992 >> v2:172.21.15.149:3300/0 conn(0x7f724409b6c0 msgr2=0x7f72440a0ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:22:54.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.529+0000 7f72427fc640 1 --2- 172.21.15.114:0/150234992 >> v2:172.21.15.149:3300/0 conn(0x7f724409b6c0 0x7f72440a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:54.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.529+0000 7f7241ffb640 1 -- 172.21.15.114:0/150234992 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2233793342 0 0) 0x7f724409d4e0 con 0x7f7244139640 2024-09-16T05:22:54.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.529+0000 7f72427fc640 1 -- 172.21.15.114:0/150234992 >> v1:172.21.15.114:6789/0 conn(0x7f7244139640 legacy=0x7f724413ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:54.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.529+0000 7f72427fc640 10 monclient: _finish_hunting 0 2024-09-16T05:22:54.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.529+0000 7f72427fc640 1 monclient: found mon.noname-a 2024-09-16T05:22:54.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.529+0000 7f72427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:54.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.529+0000 7f72427fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:22:54.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.529+0000 7f72427fc640 1 -- 172.21.15.114:0/150234992 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f724409f430 con 0x7f72440a1400 2024-09-16T05:22:54.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.529+0000 7f7241ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:22:54.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f72427fc640 1 --2- 172.21.15.114:0/150234992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72440a1400 0x7f72440a1810 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f72440a2f70 tx=0x7f723401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5de217efd5d1793 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:54.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7241ffb640 1 -- 172.21.15.114:0/150234992 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72340b0070 con 0x7f72440a1400 2024-09-16T05:22:54.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7241ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:54.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7241ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:22:54.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7241ffb640 10 monclient: dump: 2024-09-16T05:22:54.533 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:54.533 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:54.533 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:54.533 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:54.533 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:54.533 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:54.534 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:54.534 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:54.534 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:54.534 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:54.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7241ffb640 10 monclient: _finish_auth 0 2024-09-16T05:22:54.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7241ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:22:54.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7241ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:54.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 5 monclient: authenticate success, global_id 74721 2024-09-16T05:22:54.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:22:54.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7241ffb640 1 -- 172.21.15.114:0/150234992 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72340ab600 con 0x7f72440a1400 2024-09-16T05:22:54.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7241ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:54.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 10 monclient: get_monmap_and_config success 2024-09-16T05:22:54.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 4 set_mon_vals no callback set 2024-09-16T05:22:54.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7241ffb640 1 -- 172.21.15.114:0/150234992 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72340a4e60 con 0x7f72440a1400 2024-09-16T05:22:54.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7241ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:54.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7241ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:54.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7241ffb640 10 monclient: dump: 2024-09-16T05:22:54.535 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:54.536 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:54.536 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:54.536 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:54.536 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:54.536 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:54.536 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:54.536 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:54.536 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:54.536 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:54.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 10 monclient: shutdown 2024-09-16T05:22:54.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:54.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 -- 172.21.15.114:0/150234992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72440a1400 msgr2=0x7f72440a1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:54.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 --2- 172.21.15.114:0/150234992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72440a1400 0x7f72440a1810 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f72440a2f70 tx=0x7f723401c3d0 comp rx=0 tx=0).stop 2024-09-16T05:22:54.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 -- 172.21.15.114:0/150234992 shutdown_connections 2024-09-16T05:22:54.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 --2- 172.21.15.114:0/150234992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72440a1400 0x7f72440a1810 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:54.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 --2- 172.21.15.114:0/150234992 >> v2:172.21.15.149:3300/0 conn(0x7f724409b6c0 0x7f72440a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:54.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 -- 172.21.15.114:0/150234992 >> 172.21.15.114:0/150234992 conn(0x7f72440a1fb0 msgr2=0x7f72440a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:54.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 -- 172.21.15.114:0/150234992 shutdown_connections 2024-09-16T05:22:54.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 -- 172.21.15.114:0/150234992 wait complete. 2024-09-16T05:22:54.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 10 monclient: build_initial_monmap 2024-09-16T05:22:54.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 10 monclient: monmap: 2024-09-16T05:22:54.537 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:54.537 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:22:54.537 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:22:54.537 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:22:54.537 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:54.537 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:54.538 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:22:54.538 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:22:54.538 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:22:54.538 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:54.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 librados: starting msgr at 2024-09-16T05:22:54.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 librados: starting objecter 2024-09-16T05:22:54.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 Processor -- start 2024-09-16T05:22:54.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 -- start start 2024-09-16T05:22:54.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 librados: setting wanted keys 2024-09-16T05:22:54.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 librados: calling monclient init 2024-09-16T05:22:54.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 10 monclient: init 2024-09-16T05:22:54.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:54.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:22:54.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f724409b6c0 0x7f7244098600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:54.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 10 monclient(hunting): picked mon.noname-a con 0x7f724409b6c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:22:54.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72440a1400 0x7f72440994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:54.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 10 monclient(hunting): picked mon.noname-c con 0x7f72440a1400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:54.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7244139640 0x7f7244099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:54.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 10 monclient(hunting): picked mon.noname-b con 0x7f7244139640 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:22:54.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:54.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f724413f970 con 0x7f724409b6c0 2024-09-16T05:22:54.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:54.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f724409ddd0 con 0x7f7244139640 2024-09-16T05:22:54.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:54.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f724413faf0 con 0x7f72440a1400 2024-09-16T05:22:54.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:54.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7249928640 10 monclient(hunting): authenticate will time out at 1859.167271s 2024-09-16T05:22:54.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f724409b6c0 0x7f7244098600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:54.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f72427fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72440a1400 0x7f72440994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:54.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f724409b6c0 0x7f7244098600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33960/0 (socket says 172.21.15.114:33960) 2024-09-16T05:22:54.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f72427fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72440a1400 0x7f72440994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:36544/0 (socket says 172.21.15.114:36544) 2024-09-16T05:22:54.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 1 -- 172.21.15.114:0/1256159546 learned_addr learned my addr 172.21.15.114:0/1256159546 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:54.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 10 monclient(hunting): get_auth_request con 0x7f724409b6c0 auth_method 0 2024-09-16T05:22:54.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:54.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:54.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:54.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f72427fc640 10 monclient(hunting): get_auth_request con 0x7f72440a1400 auth_method 0 2024-09-16T05:22:54.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f72427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:54.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f72427fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:54.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f72427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:54.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f72437fe640 1 -- 172.21.15.114:0/1256159546 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7244139640 msgr2=0x7f7244099a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:22:54.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f72437fe640 1 --2- 172.21.15.114:0/1256159546 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7244139640 0x7f7244099a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:22:54.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f72427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:54.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f72427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:54.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f72427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:54.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:54.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:54.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:54.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 10 monclient(hunting): handle_auth_done global_id 74727 payload 824 2024-09-16T05:22:54.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 1 -- 172.21.15.114:0/1256159546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72440a1400 msgr2=0x7f72440994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:54.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 1 --2- 172.21.15.114:0/1256159546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72440a1400 0x7f72440994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:54.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 1 -- 172.21.15.114:0/1256159546 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7244139640 msgr2=0x7f7244099a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:22:54.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 1 --2- 172.21.15.114:0/1256159546 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7244139640 0x7f7244099a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:54.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:22:54.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 1 monclient: found mon.noname-a 2024-09-16T05:22:54.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:54.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:22:54.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 1 -- 172.21.15.114:0/1256159546 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7244098b40 con 0x7f724409b6c0 2024-09-16T05:22:54.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f72427fc640 1 --2- 172.21.15.114:0/1256159546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72440a1400 0x7f72440994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:22:54.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f7242ffd640 1 --2- 172.21.15.114:0/1256159546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f724409b6c0 0x7f7244098600 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f723800b560 tx=0x7f723800ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:54.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f722bfff640 10 client.?.objecter ms_handle_connect 0x7f724409b6c0 2024-09-16T05:22:54.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.533+0000 7f722bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:22:54.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 1 -- 172.21.15.114:0/1256159546 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238013020 con 0x7f724409b6c0 2024-09-16T05:22:54.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:54.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:22:54.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 10 monclient: dump: 2024-09-16T05:22:54.547 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:54.547 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:54.547 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:54.548 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:54.548 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:54.548 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:54.548 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:54.548 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:54.548 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:54.548 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:54.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 10 monclient: _finish_auth 0 2024-09-16T05:22:54.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:22:54.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:54.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 1 -- 172.21.15.114:0/1256159546 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7238003df0 con 0x7f724409b6c0 2024-09-16T05:22:54.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:54.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 1 -- 172.21.15.114:0/1256159546 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72380052b0 con 0x7f724409b6c0 2024-09-16T05:22:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:22:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 10 monclient: dump: 2024-09-16T05:22:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:54.549 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f72417fa640 4 set_mon_vals no callback set 2024-09-16T05:22:54.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f7249928640 5 monclient: authenticate success, global_id 74727 2024-09-16T05:22:54.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f7249928640 10 monclient: _renew_subs 2024-09-16T05:22:54.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f7249928640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:22:54.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f7249928640 1 -- 172.21.15.114:0/1256159546 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72440935d0 con 0x7f724409b6c0 2024-09-16T05:22:54.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f7249928640 10 client.74727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:22:54.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f7249928640 10 monclient: _renew_subs 2024-09-16T05:22:54.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f7249928640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:22:54.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f7249928640 1 -- 172.21.15.114:0/1256159546 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f724413c770 con 0x7f724409b6c0 2024-09-16T05:22:54.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f7249928640 1 librados: init done 2024-09-16T05:22:54.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f7249928640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:54.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f7249928640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:54.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f7249928640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:22:54.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f7249928640 1 -- 172.21.15.114:0/1256159546 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f724409c6a0 con 0x7f724409b6c0 2024-09-16T05:22:54.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 1 -- 172.21.15.114:0/1256159546 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7238016020 con 0x7f724409b6c0 2024-09-16T05:22:54.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 1 --2- 172.21.15.114:0/1256159546 >> v2:172.21.15.149:6812/1185531135 conn(0x7f72300377c0 0x7f7230039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:54.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 1 -- 172.21.15.114:0/1256159546 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(618..618 src has 1..618) ==== 5973+0+0 (secure 0 0 0) 0x7f7238048cf0 con 0x7f724409b6c0 2024-09-16T05:22:54.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 10 client.74727.objecter ms_dispatch 0x7f7244000ba0 osd_map(618..618 src has 1..618) 2024-09-16T05:22:54.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 3 client.74727.objecter handle_osd_map got epochs [618,618] > 0 2024-09-16T05:22:54.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 3 client.74727.objecter handle_osd_map decoding full epoch 618 2024-09-16T05:22:54.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 20 client.74727.objecter dump_active .. 0 homeless 2024-09-16T05:22:54.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f72427fc640 1 --2- 172.21.15.114:0/1256159546 >> v2:172.21.15.149:6812/1185531135 conn(0x7f72300377c0 0x7f7230039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:54.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f72427fc640 10 monclient: get_auth_request con 0x7f72300377c0 auth_method 0 2024-09-16T05:22:54.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f722bfff640 10 client.74727.objecter ms_handle_connect 0x7f72300377c0 2024-09-16T05:22:54.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.537+0000 7f72427fc640 1 --2- 172.21.15.114:0/1256159546 >> v2:172.21.15.149:6812/1185531135 conn(0x7f72300377c0 0x7f7230039c80 secure :-1 s=READY pgs=3591 cs=0 l=1 rev1=1 crypto rx=0x7f72340ad000 tx=0x7f72340a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:54.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.541+0000 7f722bfff640 1 -- 172.21.15.114:0/1256159546 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f723801d020 con 0x7f724409b6c0 2024-09-16T05:22:54.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.541+0000 7f722bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:54.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.541+0000 7f722bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:22:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:22:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:22:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:22:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 1 -- 172.21.15.114:0/1256159546 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f72440a0e40 con 0x7f724409b6c0 2024-09-16T05:22:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f722bfff640 1 -- 172.21.15.114:0/1256159546 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f7244098b40 con 0x7f724409b6c0 2024-09-16T05:22:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f722bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:22:54.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f722bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:22:54.752 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:22:54.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 10 librados: watch_flush enter 2024-09-16T05:22:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 10 librados: watch_flush exit 2024-09-16T05:22:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 20 client.74727.objecter shutdown clearing up homeless session... 2024-09-16T05:22:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 10 client.74727.objecter successfully canceled tick 2024-09-16T05:22:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 1 -- 172.21.15.114:0/1256159546 >> v2:172.21.15.149:6812/1185531135 conn(0x7f72300377c0 msgr2=0x7f7230039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 1 --2- 172.21.15.114:0/1256159546 >> v2:172.21.15.149:6812/1185531135 conn(0x7f72300377c0 0x7f7230039c80 secure :-1 s=READY pgs=3591 cs=0 l=1 rev1=1 crypto rx=0x7f72340ad000 tx=0x7f72340a6000 comp rx=0 tx=0).stop 2024-09-16T05:22:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 10 monclient: shutdown 2024-09-16T05:22:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 1 -- 172.21.15.114:0/1256159546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f724409b6c0 msgr2=0x7f7244098600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 1 --2- 172.21.15.114:0/1256159546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f724409b6c0 0x7f7244098600 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f723800b560 tx=0x7f723800ba20 comp rx=0 tx=0).stop 2024-09-16T05:22:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 1 -- 172.21.15.114:0/1256159546 shutdown_connections 2024-09-16T05:22:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 1 --2- 172.21.15.114:0/1256159546 >> v2:172.21.15.149:6812/1185531135 conn(0x7f72300377c0 0x7f7230039c80 unknown :-1 s=CLOSED pgs=3591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 1 --2- 172.21.15.114:0/1256159546 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7244139640 0x7f7244099a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:54.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 1 --2- 172.21.15.114:0/1256159546 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72440a1400 0x7f72440994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:54.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 1 --2- 172.21.15.114:0/1256159546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f724409b6c0 0x7f7244098600 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:54.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 1 -- 172.21.15.114:0/1256159546 >> 172.21.15.114:0/1256159546 conn(0x7f72440a1fb0 msgr2=0x7f724413b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:54.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 1 -- 172.21.15.114:0/1256159546 shutdown_connections 2024-09-16T05:22:54.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 1 -- 172.21.15.114:0/1256159546 wait complete. 2024-09-16T05:22:54.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:54.753+0000 7f7249928640 1 librados: shutdown 2024-09-16T05:22:57.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:22:57.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 10 monclient: get_monmap_and_config 2024-09-16T05:22:57.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 10 monclient: build_initial_monmap 2024-09-16T05:22:57.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 10 monclient: monmap: 2024-09-16T05:22:57.900 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:57.900 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:57.900 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:22:57.905773+0000 2024-09-16T05:22:57.900 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:22:57.905773+0000 2024-09-16T05:22:57.900 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:57.900 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:57.900 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:22:57.900 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:22:57.900 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:22:57.901 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:22:57.901 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:22:57.901 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 1 Processor -- start 2024-09-16T05:22:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 1 -- start start 2024-09-16T05:22:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 10 monclient: init 2024-09-16T05:22:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-16T05:22:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd868137730 0x7fd868137b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 10 monclient(hunting): picked mon.noname-b con 0x7fd868137730 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd8680a5d00 0x7fd8680a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 10 monclient(hunting): picked mon.noname-c con 0x7fd8680a5d00 addr v2:172.21.15.149:3300/0 2024-09-16T05:22:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd8680a47b0 0x7fd8680a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 10 monclient(hunting): picked mon.noname-a con 0x7fd8680a47b0 addr v2:172.21.15.114:3300/0 2024-09-16T05:22:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd86809c450 con 0x7fd8680a47b0 2024-09-16T05:22:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fd8680a55e0 con 0x7fd8680a5d00 2024-09-16T05:22:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd8680a3710 con 0x7fd868137730 2024-09-16T05:22:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86f49d640 10 monclient(hunting): authenticate will time out at 1862.534967s 2024-09-16T05:22:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd868137730 0x7fd868137b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86da3a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd8680a47b0 0x7fd8680a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86da3a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd8680a47b0 0x7fd8680a4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54180/0 (socket says 172.21.15.114:54180) 2024-09-16T05:22:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd868137730 0x7fd868137b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:32938/0 (socket says 172.21.15.114:32938) 2024-09-16T05:22:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86da3a640 1 -- 172.21.15.114:0/1638966176 learned_addr learned my addr 172.21.15.114:0/1638966176 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86da3a640 10 monclient(hunting): get_auth_request con 0x7fd8680a47b0 auth_method 0 2024-09-16T05:22:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86da3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86da3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86da3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 10 monclient(hunting): get_auth_request con 0x7fd868137730 auth_method 0 2024-09-16T05:22:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86ca38640 1 -- 172.21.15.114:0/1638966176 >> v2:172.21.15.149:3300/0 conn(0x7fd8680a5d00 msgr2=0x7fd8680a4270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:22:57.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86ca38640 1 --2- 172.21.15.114:0/1638966176 >> v2:172.21.15.149:3300/0 conn(0x7fd8680a5d00 0x7fd8680a4270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:22:57.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:57.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:57.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:57.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86da3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:57.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86da3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:57.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86da3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:57.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 10 monclient(hunting): handle_auth_done global_id 104213 payload 274 2024-09-16T05:22:57.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 1 -- 172.21.15.114:0/1638966176 >> v2:172.21.15.149:3300/0 conn(0x7fd8680a5d00 msgr2=0x7fd8680a4270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:22:57.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 1 --2- 172.21.15.114:0/1638966176 >> v2:172.21.15.149:3300/0 conn(0x7fd8680a5d00 0x7fd8680a4270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:57.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 1 -- 172.21.15.114:0/1638966176 >> v2:172.21.15.114:3300/0 conn(0x7fd8680a47b0 msgr2=0x7fd8680a4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:22:57.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 1 --2- 172.21.15.114:0/1638966176 >> v2:172.21.15.114:3300/0 conn(0x7fd8680a47b0 0x7fd8680a4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:57.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 10 monclient: _finish_hunting 0 2024-09-16T05:22:57.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 1 monclient: found mon.noname-b 2024-09-16T05:22:57.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:57.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:22:57.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86d239640 1 -- 172.21.15.114:0/1638966176 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd86809be00 con 0x7fd868137730 2024-09-16T05:22:57.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.901+0000 7fd86da3a640 1 --2- 172.21.15.114:0/1638966176 >> v2:172.21.15.114:3300/0 conn(0x7fd8680a47b0 0x7fd8680a4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:22:57.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86d239640 1 --2- 172.21.15.114:0/1638966176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd868137730 0x7fd868137b10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd868072f50 tx=0x7fd85c01c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35253197e521d7c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:57.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd85bfff640 1 -- 172.21.15.114:0/1638966176 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0b0070 con 0x7fd868137730 2024-09-16T05:22:57.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd85bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:57.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd85bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:22:57.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd85bfff640 10 monclient: dump: 2024-09-16T05:22:57.913 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:57.913 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:57.913 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:57.914 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:57.914 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:57.914 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:57.914 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:57.914 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:57.914 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:57.914 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:57.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd85bfff640 10 monclient: _finish_auth 0 2024-09-16T05:22:57.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd85bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:22:57.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd85bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:57.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd85bfff640 1 -- 172.21.15.114:0/1638966176 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd85c0ab600 con 0x7fd868137730 2024-09-16T05:22:57.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd85bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:57.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd85bfff640 1 -- 172.21.15.114:0/1638966176 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0ab9d0 con 0x7fd868137730 2024-09-16T05:22:57.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 5 monclient: authenticate success, global_id 104213 2024-09-16T05:22:57.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 10 monclient: get_monmap_and_config success 2024-09-16T05:22:57.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 4 set_mon_vals no callback set 2024-09-16T05:22:57.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd85bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:57.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd85bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:22:57.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd85bfff640 10 monclient: dump: 2024-09-16T05:22:57.915 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:57.915 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:57.915 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:57.915 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:57.915 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:57.915 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:57.915 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:57.915 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:57.916 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:57.916 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:57.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 10 monclient: shutdown 2024-09-16T05:22:57.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:57.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 1 -- 172.21.15.114:0/1638966176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd868137730 msgr2=0x7fd868137b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:57.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 1 --2- 172.21.15.114:0/1638966176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd868137730 0x7fd868137b10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd868072f50 tx=0x7fd85c01c3f0 comp rx=0 tx=0).stop 2024-09-16T05:22:57.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 1 -- 172.21.15.114:0/1638966176 shutdown_connections 2024-09-16T05:22:57.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 1 --2- 172.21.15.114:0/1638966176 >> v2:172.21.15.114:3300/0 conn(0x7fd8680a47b0 0x7fd8680a4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:57.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 1 --2- 172.21.15.114:0/1638966176 >> v2:172.21.15.149:3300/0 conn(0x7fd8680a5d00 0x7fd8680a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:57.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 1 --2- 172.21.15.114:0/1638966176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd868137730 0x7fd868137b10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:57.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 1 -- 172.21.15.114:0/1638966176 >> 172.21.15.114:0/1638966176 conn(0x7fd8680a9c70 msgr2=0x7fd8680aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:57.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 1 -- 172.21.15.114:0/1638966176 shutdown_connections 2024-09-16T05:22:57.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 1 -- 172.21.15.114:0/1638966176 wait complete. 2024-09-16T05:22:57.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 10 monclient: build_initial_monmap 2024-09-16T05:22:57.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 10 monclient: monmap: 2024-09-16T05:22:57.917 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:22:57.917 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:22:57.917 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:22:57.917 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:22:57.917 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:22:57.917 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:57.917 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:22:57.917 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:22:57.917 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:22:57.917 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:57.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 1 librados: starting msgr at 2024-09-16T05:22:57.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 1 librados: starting objecter 2024-09-16T05:22:57.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 1 Processor -- start 2024-09-16T05:22:57.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 1 -- start start 2024-09-16T05:22:57.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 1 librados: setting wanted keys 2024-09-16T05:22:57.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 1 librados: calling monclient init 2024-09-16T05:22:57.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.905+0000 7fd86f49d640 10 monclient: init 2024-09-16T05:22:57.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient: _reopen_session rank -1 2024-09-16T05:22:57.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:22:57.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8680a47b0 0x7fd868133440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:57.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient(hunting): picked mon.noname-c con 0x7fd8680a47b0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:22:57.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd8680a5d00 0x7fd868133980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:57.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient(hunting): picked mon.noname-b con 0x7fd8680a5d00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:22:57.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd868137730 0x7fd8681cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:57.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient(hunting): picked mon.noname-a con 0x7fd868137730 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:22:57.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:57.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd868137320 con 0x7fd868137730 2024-09-16T05:22:57.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:57.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fd8681371a0 con 0x7fd8680a5d00 2024-09-16T05:22:57.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:22:57.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd868140cf0 con 0x7fd8680a47b0 2024-09-16T05:22:57.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient(hunting): _renew_subs 2024-09-16T05:22:57.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient(hunting): authenticate will time out at 1862.540308s 2024-09-16T05:22:57.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8680a47b0 0x7fd868133440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:57.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8680a47b0 0x7fd868133440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:32940/0 (socket says 172.21.15.114:32940) 2024-09-16T05:22:57.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 1 -- 172.21.15.114:0/1942972394 learned_addr learned my addr 172.21.15.114:0/1942972394 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:22:57.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 10 monclient(hunting): get_auth_request con 0x7fd8680a47b0 auth_method 0 2024-09-16T05:22:57.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:57.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:57.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86da3a640 1 --2- 172.21.15.114:0/1942972394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd868137730 0x7fd8681cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86da3a640 10 monclient(hunting): get_auth_request con 0x7fd868137730 auth_method 0 2024-09-16T05:22:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86da3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:22:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86da3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:22:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86da3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:22:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86ca38640 1 -- 172.21.15.114:0/1942972394 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd8680a5d00 msgr2=0x7fd868133980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:22:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86ca38640 1 --2- 172.21.15.114:0/1942972394 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd8680a5d00 0x7fd868133980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:22:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86da3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:22:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86da3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:22:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86da3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:22:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 10 monclient(hunting): handle_auth_done global_id 104216 payload 824 2024-09-16T05:22:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 1 -- 172.21.15.114:0/1942972394 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd8680a5d00 msgr2=0x7fd868133980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:22:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 1 --2- 172.21.15.114:0/1942972394 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd8680a5d00 0x7fd868133980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 1 -- 172.21.15.114:0/1942972394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd868137730 msgr2=0x7fd8681cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 1 --2- 172.21.15.114:0/1942972394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd868137730 0x7fd8681cefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 10 monclient: _finish_hunting 0 2024-09-16T05:22:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 1 monclient: found mon.noname-c 2024-09-16T05:22:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:22:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:22:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 1 -- 172.21.15.114:0/1942972394 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd86813d1d0 con 0x7fd8680a47b0 2024-09-16T05:22:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86d239640 1 --2- 172.21.15.114:0/1942972394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8680a47b0 0x7fd868133440 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd85c01cc90 tx=0x7fd85c01ac30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 10 client.?.objecter ms_handle_connect 0x7fd8680a47b0 2024-09-16T05:22:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:22:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 1 -- 172.21.15.114:0/1942972394 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0b0050 con 0x7fd8680a47b0 2024-09-16T05:22:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:57.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:22:57.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 10 monclient: dump: 2024-09-16T05:22:57.922 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:57.922 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:57.922 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:57.922 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:57.922 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:57.922 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:57.922 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:57.922 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:57.922 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:57.922 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:57.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 10 monclient: _finish_auth 0 2024-09-16T05:22:57.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:22:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:22:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 1 -- 172.21.15.114:0/1942972394 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd85c0bc070 con 0x7fd8680a47b0 2024-09-16T05:22:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 5 monclient: authenticate success, global_id 104216 2024-09-16T05:22:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient: _renew_subs 2024-09-16T05:22:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 1 -- 172.21.15.114:0/1942972394 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd86813d4b0 con 0x7fd8680a47b0 2024-09-16T05:22:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:22:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 1 -- 172.21.15.114:0/1942972394 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0b5830 con 0x7fd8680a47b0 2024-09-16T05:22:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:22:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:22:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 10 monclient: dump: 2024-09-16T05:22:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:22:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:22:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:22:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:22:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:22:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:22:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:22:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:22:57.924 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:22:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd85affd640 4 set_mon_vals no callback set 2024-09-16T05:22:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 client.104216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:22:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient: _renew_subs 2024-09-16T05:22:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 1 -- 172.21.15.114:0/1942972394 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8680a6e20 con 0x7fd8680a47b0 2024-09-16T05:22:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 1 librados: init done 2024-09-16T05:22:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 1 -- 172.21.15.114:0/1942972394 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd85c0b59d0 con 0x7fd8680a47b0 2024-09-16T05:22:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86f49d640 1 -- 172.21.15.114:0/1942972394 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8680a4270 con 0x7fd8680a47b0 2024-09-16T05:22:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 1 --2- 172.21.15.114:0/1942972394 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd84c0377c0 0x7fd84c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:22:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 1 -- 172.21.15.114:0/1942972394 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(618..618 src has 1..618) ==== 5973+0+0 (secure 0 0 0) 0x7fd85c0e35a0 con 0x7fd8680a47b0 2024-09-16T05:22:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 10 client.104216.objecter ms_dispatch 0x7fd868000ba0 osd_map(618..618 src has 1..618) 2024-09-16T05:22:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 3 client.104216.objecter handle_osd_map got epochs [618,618] > 0 2024-09-16T05:22:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 3 client.104216.objecter handle_osd_map decoding full epoch 618 2024-09-16T05:22:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 20 client.104216.objecter dump_active .. 0 homeless 2024-09-16T05:22:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86ca38640 1 --2- 172.21.15.114:0/1942972394 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd84c0377c0 0x7fd84c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:22:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86ca38640 10 monclient: get_auth_request con 0x7fd84c0377c0 auth_method 0 2024-09-16T05:22:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd86ca38640 1 --2- 172.21.15.114:0/1942972394 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd84c0377c0 0x7fd84c039c80 secure :-1 s=READY pgs=3592 cs=0 l=1 rev1=1 crypto rx=0x7fd864006fd0 tx=0x7fd864006d60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:22:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.909+0000 7fd859ffb640 10 client.104216.objecter ms_handle_connect 0x7fd84c0377c0 2024-09-16T05:22:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.913+0000 7fd859ffb640 1 -- 172.21.15.114:0/1942972394 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd85c016070 con 0x7fd8680a47b0 2024-09-16T05:22:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.913+0000 7fd859ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:22:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:57.913+0000 7fd859ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:22:58.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.129+0000 7fd86f49d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:22:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.129+0000 7fd86f49d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:22:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.129+0000 7fd86f49d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:22:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.129+0000 7fd86f49d640 1 -- 172.21.15.114:0/1942972394 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd868068ec0 con 0x7fd8680a47b0 2024-09-16T05:22:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd859ffb640 1 -- 172.21.15.114:0/1942972394 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fd868068ec0 con 0x7fd8680a47b0 2024-09-16T05:22:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd859ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:22:58.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd859ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:22:58.131 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:22:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 10 librados: watch_flush enter 2024-09-16T05:22:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 10 librados: watch_flush exit 2024-09-16T05:22:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 20 client.104216.objecter shutdown clearing up homeless session... 2024-09-16T05:22:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 10 client.104216.objecter successfully canceled tick 2024-09-16T05:22:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 1 -- 172.21.15.114:0/1942972394 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd84c0377c0 msgr2=0x7fd84c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 1 --2- 172.21.15.114:0/1942972394 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd84c0377c0 0x7fd84c039c80 secure :-1 s=READY pgs=3592 cs=0 l=1 rev1=1 crypto rx=0x7fd864006fd0 tx=0x7fd864006d60 comp rx=0 tx=0).stop 2024-09-16T05:22:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 10 monclient: shutdown 2024-09-16T05:22:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:22:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 1 -- 172.21.15.114:0/1942972394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8680a47b0 msgr2=0x7fd868133440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:22:58.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 1 --2- 172.21.15.114:0/1942972394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8680a47b0 0x7fd868133440 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd85c01cc90 tx=0x7fd85c01ac30 comp rx=0 tx=0).stop 2024-09-16T05:22:58.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 1 -- 172.21.15.114:0/1942972394 shutdown_connections 2024-09-16T05:22:58.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 1 --2- 172.21.15.114:0/1942972394 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd84c0377c0 0x7fd84c039c80 unknown :-1 s=CLOSED pgs=3592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:58.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 1 --2- 172.21.15.114:0/1942972394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd868137730 0x7fd8681cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:58.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 1 --2- 172.21.15.114:0/1942972394 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd8680a5d00 0x7fd868133980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:58.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 1 --2- 172.21.15.114:0/1942972394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8680a47b0 0x7fd868133440 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:22:58.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 1 -- 172.21.15.114:0/1942972394 >> 172.21.15.114:0/1942972394 conn(0x7fd8680a9c70 msgr2=0x7fd8680a8600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:22:58.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 1 -- 172.21.15.114:0/1942972394 shutdown_connections 2024-09-16T05:22:58.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 1 -- 172.21.15.114:0/1942972394 wait complete. 2024-09-16T05:22:58.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:22:58.133+0000 7fd86f49d640 1 librados: shutdown 2024-09-16T05:23:01.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:23:01.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 10 monclient: get_monmap_and_config 2024-09-16T05:23:01.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 10 monclient: build_initial_monmap 2024-09-16T05:23:01.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 10 monclient: monmap: 2024-09-16T05:23:01.276 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:23:01.276 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:01.276 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:23:01.281698+0000 2024-09-16T05:23:01.276 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:23:01.281698+0000 2024-09-16T05:23:01.276 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:23:01.276 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:01.276 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:23:01.277 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:23:01.277 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:23:01.277 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:23:01.277 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:23:01.277 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:01.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 1 Processor -- start 2024-09-16T05:23:01.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 1 -- start start 2024-09-16T05:23:01.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 10 monclient: init 2024-09-16T05:23:01.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 10 monclient: _reopen_session rank -1 2024-09-16T05:23:01.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-16T05:23:01.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5800a0ec0 addr v1:172.21.15.114:6789/0 2024-09-16T05:23:01.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fa58009e1d0 0x7fa58009e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:01.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 10 monclient(hunting): picked mon.noname-c con 0x7fa58009e1d0 addr v2:172.21.15.149:3300/0 2024-09-16T05:23:01.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 0x7fa58013ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:01.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 10 monclient(hunting): picked mon.noname-b con 0x7fa580139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:23:01.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa58009a760 con 0x7fa5800a0ec0 2024-09-16T05:23:01.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:01.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fa5800a12a0 con 0x7fa58009e1d0 2024-09-16T05:23:01.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:01.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa5800994b0 con 0x7fa580139640 2024-09-16T05:23:01.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 10 monclient(hunting): _renew_subs 2024-09-16T05:23:01.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa586565640 10 monclient(hunting): authenticate will time out at 1865.910984s 2024-09-16T05:23:01.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa584b02640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 0x7fa58013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:01.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa57ffff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa5800a0ec0 0x7fa58009dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43036/0 (socket says 172.21.15.114:43036) 2024-09-16T05:23:01.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa57ffff640 1 -- 172.21.15.114:0/964723841 learned_addr learned my addr 172.21.15.114:0/964723841 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:23:01.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa57f7fe640 1 -- 172.21.15.114:0/964723841 >> v2:172.21.15.149:3300/0 conn(0x7fa58009e1d0 msgr2=0x7fa58009e5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:23:01.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa57f7fe640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.149:3300/0 conn(0x7fa58009e1d0 0x7fa58009e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:23:01.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa584b02640 10 monclient(hunting): get_auth_request con 0x7fa580139640 auth_method 0 2024-09-16T05:23:01.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa584b02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:01.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa584b02640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:01.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa584b02640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:01.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa584b02640 1 -- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 msgr2=0x7fa58013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:23:01.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa584b02640 1 -- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 msgr2=0x7fa58013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:23:01.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa584b02640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 0x7fa58013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:23:01.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.277+0000 7fa584b02640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 0x7fa58013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:23:01.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.481+0000 7fa57f7fe640 1 -- 172.21.15.114:0/964723841 >> v2:172.21.15.149:3300/0 conn(0x7fa58009e1d0 msgr2=0x7fa58009e5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:23:01.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.481+0000 7fa57f7fe640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.149:3300/0 conn(0x7fa58009e1d0 0x7fa58009e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:23:01.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.481+0000 7fa584b02640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 0x7fa58013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:01.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.481+0000 7fa584b02640 10 monclient(hunting): get_auth_request con 0x7fa580139640 auth_method 0 2024-09-16T05:23:01.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.481+0000 7fa584b02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:01.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.481+0000 7fa584b02640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:01.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.481+0000 7fa584b02640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:23:01.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.481+0000 7fa584b02640 1 -- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 msgr2=0x7fa58013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:23:01.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.481+0000 7fa584b02640 1 -- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 msgr2=0x7fa58013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:23:01.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.481+0000 7fa584b02640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 0x7fa58013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:23:01.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.481+0000 7fa584b02640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 0x7fa58013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:23:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.885+0000 7fa57f7fe640 1 -- 172.21.15.114:0/964723841 >> v2:172.21.15.149:3300/0 conn(0x7fa58009e1d0 msgr2=0x7fa58009e5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:23:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.885+0000 7fa57f7fe640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.149:3300/0 conn(0x7fa58009e1d0 0x7fa58009e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:23:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.885+0000 7fa584b02640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 0x7fa58013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.885+0000 7fa584b02640 10 monclient(hunting): get_auth_request con 0x7fa580139640 auth_method 0 2024-09-16T05:23:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.885+0000 7fa584b02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.885+0000 7fa584b02640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.885+0000 7fa584b02640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:23:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.885+0000 7fa584b02640 1 -- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 msgr2=0x7fa58013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:23:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.885+0000 7fa584b02640 1 -- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 msgr2=0x7fa58013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:23:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.885+0000 7fa584b02640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 0x7fa58013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:23:01.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:01.885+0000 7fa584b02640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 0x7fa58013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:23:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:02.688+0000 7fa57f7fe640 1 -- 172.21.15.114:0/964723841 >> v2:172.21.15.149:3300/0 conn(0x7fa58009e1d0 msgr2=0x7fa58009e5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:23:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:02.688+0000 7fa57f7fe640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.149:3300/0 conn(0x7fa58009e1d0 0x7fa58009e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:23:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:02.688+0000 7fa584b02640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 0x7fa58013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:02.688+0000 7fa584b02640 10 monclient(hunting): get_auth_request con 0x7fa580139640 auth_method 0 2024-09-16T05:23:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:02.688+0000 7fa584b02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:02.688+0000 7fa584b02640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:02.688+0000 7fa584b02640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:23:02.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:02.688+0000 7fa584b02640 1 -- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 msgr2=0x7fa58013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:23:02.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:02.688+0000 7fa584b02640 1 -- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 msgr2=0x7fa58013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:23:02.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:02.688+0000 7fa584b02640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 0x7fa58013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:23:02.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:02.688+0000 7fa584b02640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 0x7fa58013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:23:04.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 10 monclient(hunting): tick 2024-09-16T05:23:04.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:23:04.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 1 monclient(hunting): continuing hunt 2024-09-16T05:23:04.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:23:04.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 1 -- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 msgr2=0x7fa58013ba30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:23:04.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa580139640 0x7fa58013ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 1 -- 172.21.15.114:0/964723841 >> v2:172.21.15.149:3300/0 conn(0x7fa58009e1d0 msgr2=0x7fa58009e5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:23:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.149:3300/0 conn(0x7fa58009e1d0 0x7fa58009e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 1 -- 172.21.15.114:0/964723841 >> v1:172.21.15.114:6789/0 conn(0x7fa5800a0ec0 legacy=0x7fa58009dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 10 monclient: _add_conns ranks=[3,1] 2024-09-16T05:23:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 0x7fa560005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 10 monclient(hunting): picked mon.noname-a con 0x7fa5600036d0 addr v2:172.21.15.114:3300/0 2024-09-16T05:23:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa5600083b0 addr v1:172.21.15.149:6789/0 2024-09-16T05:23:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 1 -- 172.21.15.114:0/964723841 --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5800a1870 con 0x7fa5600083b0 2024-09-16T05:23:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 1 -- 172.21.15.114:0/964723841 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa5800a12a0 con 0x7fa5600036d0 2024-09-16T05:23:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57dffb640 10 monclient(hunting): _renew_subs 2024-09-16T05:23:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57f7fe640 1 -- 172.21.15.114:0/964723841 >> v1:172.21.15.149:6789/0 conn(0x7fa5600083b0 legacy=0x7fa56000a870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:23:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57ffff640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 0x7fa560005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:04.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57ffff640 10 monclient(hunting): get_auth_request con 0x7fa5600036d0 auth_method 0 2024-09-16T05:23:04.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:04.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:04.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:04.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57ffff640 1 -- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 msgr2=0x7fa560005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:23:04.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57ffff640 1 -- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 msgr2=0x7fa560005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:23:04.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57ffff640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 0x7fa560005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:23:04.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.280+0000 7fa57ffff640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 0x7fa560005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:23:04.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.484+0000 7fa57f7fe640 1 -- 172.21.15.114:0/964723841 >> v1:172.21.15.149:6789/0 conn(0x7fa5600083b0 legacy=0x7fa56000a870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:23:04.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.484+0000 7fa57ffff640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 0x7fa560005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:04.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.484+0000 7fa57ffff640 10 monclient(hunting): get_auth_request con 0x7fa5600036d0 auth_method 0 2024-09-16T05:23:04.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.484+0000 7fa57ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:04.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.484+0000 7fa57ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:04.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.484+0000 7fa57ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:23:04.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.484+0000 7fa57ffff640 1 -- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 msgr2=0x7fa560005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:23:04.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.484+0000 7fa57ffff640 1 -- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 msgr2=0x7fa560005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:23:04.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.484+0000 7fa57ffff640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 0x7fa560005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:23:04.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.484+0000 7fa57ffff640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 0x7fa560005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.888+0000 7fa57f7fe640 1 -- 172.21.15.114:0/964723841 >> v1:172.21.15.149:6789/0 conn(0x7fa5600083b0 legacy=0x7fa56000a870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.888+0000 7fa57ffff640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 0x7fa560005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.888+0000 7fa57ffff640 10 monclient(hunting): get_auth_request con 0x7fa5600036d0 auth_method 0 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.888+0000 7fa57ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.888+0000 7fa57ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.888+0000 7fa57ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.888+0000 7fa57ffff640 1 -- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 msgr2=0x7fa560005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.888+0000 7fa57ffff640 1 -- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 msgr2=0x7fa560005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.888+0000 7fa57ffff640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 0x7fa560005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:04.888+0000 7fa57ffff640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 0x7fa560005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:23:05.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:05.692+0000 7fa57f7fe640 1 -- 172.21.15.114:0/964723841 >> v1:172.21.15.149:6789/0 conn(0x7fa5600083b0 legacy=0x7fa56000a870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:23:05.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:05.692+0000 7fa57ffff640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 0x7fa560005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:05.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:05.692+0000 7fa57ffff640 10 monclient(hunting): get_auth_request con 0x7fa5600036d0 auth_method 0 2024-09-16T05:23:05.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:05.692+0000 7fa57ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:05.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:05.692+0000 7fa57ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:05.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:05.692+0000 7fa57ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:23:05.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:05.692+0000 7fa57ffff640 1 -- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 msgr2=0x7fa560005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:23:05.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:05.692+0000 7fa57ffff640 1 -- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 msgr2=0x7fa560005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:23:05.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:05.692+0000 7fa57ffff640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 0x7fa560005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:23:05.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:05.692+0000 7fa57ffff640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 0x7fa560005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:23:07.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 10 monclient(hunting): tick 2024-09-16T05:23:07.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:23:07.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 1 monclient(hunting): continuing hunt 2024-09-16T05:23:07.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:23:07.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 1 -- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 msgr2=0x7fa560005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:23:07.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa5600036d0 0x7fa560005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 1 -- 172.21.15.114:0/964723841 >> v1:172.21.15.149:6789/0 conn(0x7fa5600083b0 legacy=0x7fa56000a870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:23:07.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa584b02640 1 -- 172.21.15.114:0/964723841 reap_dead start 2024-09-16T05:23:07.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T05:23:07.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa56000c130 0x7fa560005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 10 monclient(hunting): picked mon.noname-b con 0x7fa56000c130 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:23:07.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa56000c600 0x7fa56000a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 10 monclient(hunting): picked mon.noname-a con 0x7fa56000c600 addr v2:172.21.15.114:3300/0 2024-09-16T05:23:07.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa56000c9e0 addr v1:172.21.15.114:6789/0 2024-09-16T05:23:07.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 1 -- 172.21.15.114:0/964723841 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa56000d160 con 0x7fa56000c9e0 2024-09-16T05:23:07.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 1 -- 172.21.15.114:0/964723841 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa5800a12a0 con 0x7fa56000c600 2024-09-16T05:23:07.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 1 -- 172.21.15.114:0/964723841 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa5800994b0 con 0x7fa56000c130 2024-09-16T05:23:07.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.276+0000 7fa57dffb640 10 monclient(hunting): _renew_subs 2024-09-16T05:23:07.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57ffff640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa56000c130 0x7fa560005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 1 --2- 172.21.15.114:0/964723841 >> v2:172.21.15.114:3300/0 conn(0x7fa56000c600 0x7fa56000a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57ffff640 10 monclient(hunting): get_auth_request con 0x7fa56000c130 auth_method 0 2024-09-16T05:23:07.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:07.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 10 monclient(hunting): get_auth_request con 0x7fa56000c600 auth_method 0 2024-09-16T05:23:07.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:07.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:23:07.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:23:07.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:23:07.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:23:07.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:23:07.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:23:07.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 10 monclient(hunting): handle_auth_done global_id 74736 payload 274 2024-09-16T05:23:07.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 1 -- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa56000c130 msgr2=0x7fa560005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:23:07.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa56000c130 0x7fa560005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 1 -- 172.21.15.114:0/964723841 >> v1:172.21.15.114:6789/0 conn(0x7fa56000c9e0 legacy=0x7fa56000b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:23:07.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 1 monclient: found mon.noname-a 2024-09-16T05:23:07.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:23:07.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:23:07.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57f7fe640 1 -- 172.21.15.114:0/964723841 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5800a1870 con 0x7fa56000c600 2024-09-16T05:23:07.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.280+0000 7fa57ffff640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa56000c130 0x7fa560005b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:23:07.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa57f7fe640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa56000c600 0x7fa56000a870 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa568003990 tx=0x7fa56801ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8580ab9b6f2bffd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:23:07.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa57effd640 1 -- 172.21.15.114:0/964723841 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5680a5a40 con 0x7fa56000c600 2024-09-16T05:23:07.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa57effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa57effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:23:07.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa57effd640 10 monclient: dump: 2024-09-16T05:23:07.286 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.286 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.286 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.286 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.286 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.286 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.286 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.286 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.286 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.287 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa57effd640 10 monclient: _finish_auth 0 2024-09-16T05:23:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa57effd640 10 monclient: _check_auth_tickets 2024-09-16T05:23:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa57effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:23:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa586565640 5 monclient: authenticate success, global_id 74736 2024-09-16T05:23:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa57effd640 1 -- 172.21.15.114:0/964723841 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5680a9070 con 0x7fa56000c600 2024-09-16T05:23:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa57effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:23:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa586565640 10 monclient: get_monmap_and_config success 2024-09-16T05:23:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa586565640 4 set_mon_vals no callback set 2024-09-16T05:23:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa57effd640 1 -- 172.21.15.114:0/964723841 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5680a5e00 con 0x7fa56000c600 2024-09-16T05:23:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa57effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa57effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:23:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.284+0000 7fa57effd640 10 monclient: dump: 2024-09-16T05:23:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.288 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.288 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.288 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.288 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.288 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.288 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.288 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.288 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.288 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 10 monclient: shutdown 2024-09-16T05:23:07.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:23:07.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 -- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa56000c600 msgr2=0x7fa56000a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa56000c600 0x7fa56000a870 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa568003990 tx=0x7fa56801ca60 comp rx=0 tx=0).stop 2024-09-16T05:23:07.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 -- 172.21.15.114:0/964723841 shutdown_connections 2024-09-16T05:23:07.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa56000c600 0x7fa56000a870 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 --2- 172.21.15.114:0/964723841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa56000c130 0x7fa560005b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 -- 172.21.15.114:0/964723841 >> 172.21.15.114:0/964723841 conn(0x7fa5800a1fb0 msgr2=0x7fa5800a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:23:07.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 -- 172.21.15.114:0/964723841 shutdown_connections 2024-09-16T05:23:07.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 -- 172.21.15.114:0/964723841 wait complete. 2024-09-16T05:23:07.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 10 monclient: build_initial_monmap 2024-09-16T05:23:07.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 10 monclient: monmap: 2024-09-16T05:23:07.294 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:23:07.294 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:23:07.294 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:23:07.295 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:23:07.295 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:23:07.295 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.295 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:23:07.295 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:23:07.295 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:23:07.295 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 librados: starting msgr at 2024-09-16T05:23:07.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 librados: starting objecter 2024-09-16T05:23:07.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 Processor -- start 2024-09-16T05:23:07.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 -- start start 2024-09-16T05:23:07.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 librados: setting wanted keys 2024-09-16T05:23:07.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 librados: calling monclient init 2024-09-16T05:23:07.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 10 monclient: init 2024-09-16T05:23:07.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 10 monclient: _reopen_session rank -1 2024-09-16T05:23:07.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:23:07.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa56000c130 0x7fa580095ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 10 monclient(hunting): picked mon.noname-c con 0x7fa56000c130 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:23:07.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa56000c600 0x7fa5801cca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 10 monclient(hunting): picked mon.noname-a con 0x7fa56000c600 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:23:07.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa56000c9e0 0x7fa5801ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 10 monclient(hunting): picked mon.noname-b con 0x7fa56000c9e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:23:07.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa5800a1df0 con 0x7fa56000c600 2024-09-16T05:23:07.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fa5800a1c70 con 0x7fa56000c9e0 2024-09-16T05:23:07.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa58009c230 con 0x7fa56000c130 2024-09-16T05:23:07.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 10 monclient(hunting): _renew_subs 2024-09-16T05:23:07.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa586565640 10 monclient(hunting): authenticate will time out at 1871.923819s 2024-09-16T05:23:07.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa57ffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa56000c130 0x7fa580095ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.288+0000 7fa57ffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa56000c130 0x7fa580095ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:46102/0 (socket says 172.21.15.114:46102) 2024-09-16T05:23:07.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 1 -- 172.21.15.114:0/2226779659 learned_addr learned my addr 172.21.15.114:0/2226779659 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:23:07.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 10 monclient(hunting): get_auth_request con 0x7fa56000c130 auth_method 0 2024-09-16T05:23:07.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa584b02640 1 -- 172.21.15.114:0/2226779659 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa56000c9e0 msgr2=0x7fa5801ccfb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:23:07.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:07.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57f7fe640 1 --2- 172.21.15.114:0/2226779659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa56000c600 0x7fa5801cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa584b02640 1 --2- 172.21.15.114:0/2226779659 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa56000c9e0 0x7fa5801ccfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:23:07.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57f7fe640 10 monclient(hunting): get_auth_request con 0x7fa56000c600 auth_method 0 2024-09-16T05:23:07.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:07.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:23:07.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:23:07.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:23:07.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:23:07.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:23:07.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:23:07.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 10 monclient(hunting): handle_auth_done global_id 104222 payload 824 2024-09-16T05:23:07.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 1 -- 172.21.15.114:0/2226779659 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa56000c9e0 msgr2=0x7fa5801ccfb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:23:07.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 1 --2- 172.21.15.114:0/2226779659 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa56000c9e0 0x7fa5801ccfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 1 -- 172.21.15.114:0/2226779659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa56000c600 msgr2=0x7fa5801cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 1 --2- 172.21.15.114:0/2226779659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa56000c600 0x7fa5801cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:23:07.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 1 monclient: found mon.noname-c 2024-09-16T05:23:07.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:23:07.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:23:07.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 1 -- 172.21.15.114:0/2226779659 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa580094e10 con 0x7fa56000c130 2024-09-16T05:23:07.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57ffff640 1 --2- 172.21.15.114:0/2226779659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa56000c130 0x7fa580095ad0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa570001520 tx=0x7fa5700019e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:23:07.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 10 client.?.objecter ms_handle_connect 0x7fa56000c130 2024-09-16T05:23:07.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:23:07.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 1 -- 172.21.15.114:0/2226779659 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa570005cb0 con 0x7fa56000c130 2024-09-16T05:23:07.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:23:07.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 10 monclient: dump: 2024-09-16T05:23:07.299 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.300 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.300 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.300 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.300 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.300 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.300 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.300 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.300 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.300 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 10 monclient: _finish_auth 0 2024-09-16T05:23:07.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 10 monclient: _check_auth_tickets 2024-09-16T05:23:07.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:23:07.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 1 -- 172.21.15.114:0/2226779659 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa570005e50 con 0x7fa56000c130 2024-09-16T05:23:07.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa586565640 5 monclient: authenticate success, global_id 104222 2024-09-16T05:23:07.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa586565640 10 monclient: _renew_subs 2024-09-16T05:23:07.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa586565640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:23:07.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa586565640 1 -- 172.21.15.114:0/2226779659 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5800953d0 con 0x7fa56000c130 2024-09-16T05:23:07.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:23:07.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 1 -- 172.21.15.114:0/2226779659 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5700049f0 con 0x7fa56000c130 2024-09-16T05:23:07.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:23:07.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa586565640 10 client.104222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:23:07.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 10 monclient: dump: 2024-09-16T05:23:07.301 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.301 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.301 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.301 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.302 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.302 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.302 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.302 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.302 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.302 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa586565640 10 monclient: _renew_subs 2024-09-16T05:23:07.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa586565640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:23:07.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa586565640 1 -- 172.21.15.114:0/2226779659 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5801cd4f0 con 0x7fa56000c130 2024-09-16T05:23:07.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57e7fc640 4 set_mon_vals no callback set 2024-09-16T05:23:07.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa586565640 1 librados: init done 2024-09-16T05:23:07.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa586565640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:23:07.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa586565640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:23:07.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa586565640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:23:07.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa586565640 1 -- 172.21.15.114:0/2226779659 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa58009e670 con 0x7fa56000c130 2024-09-16T05:23:07.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 1 -- 172.21.15.114:0/2226779659 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa570004b90 con 0x7fa56000c130 2024-09-16T05:23:07.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 1 --2- 172.21.15.114:0/2226779659 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa56003c720 0x7fa56003ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 1 -- 172.21.15.114:0/2226779659 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(620..620 src has 1..620) ==== 5973+0+0 (secure 0 0 0) 0x7fa570049fc0 con 0x7fa56000c130 2024-09-16T05:23:07.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 10 client.104222.objecter ms_dispatch 0x7fa580000ba0 osd_map(620..620 src has 1..620) 2024-09-16T05:23:07.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 3 client.104222.objecter handle_osd_map got epochs [620,620] > 0 2024-09-16T05:23:07.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 3 client.104222.objecter handle_osd_map decoding full epoch 620 2024-09-16T05:23:07.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 20 client.104222.objecter dump_active .. 0 homeless 2024-09-16T05:23:07.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57f7fe640 1 --2- 172.21.15.114:0/2226779659 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa56003c720 0x7fa56003ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57f7fe640 10 monclient: get_auth_request con 0x7fa56003c720 auth_method 0 2024-09-16T05:23:07.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57cff9640 10 client.104222.objecter ms_handle_connect 0x7fa56003c720 2024-09-16T05:23:07.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.292+0000 7fa57f7fe640 1 --2- 172.21.15.114:0/2226779659 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa56003c720 0x7fa56003ebe0 secure :-1 s=READY pgs=3593 cs=0 l=1 rev1=1 crypto rx=0x7fa5680ad000 tx=0x7fa5680a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:23:07.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.296+0000 7fa57cff9640 1 -- 172.21.15.114:0/2226779659 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fa570027020 con 0x7fa56000c130 2024-09-16T05:23:07.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.296+0000 7fa57cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:23:07.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.296+0000 7fa57cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:23:07.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.528+0000 7fa586565640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:23:07.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.528+0000 7fa586565640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:23:07.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.528+0000 7fa586565640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:23:07.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.528+0000 7fa586565640 1 -- 172.21.15.114:0/2226779659 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa5800686f0 con 0x7fa56000c130 2024-09-16T05:23:07.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.528+0000 7fa57cff9640 1 -- 172.21.15.114:0/2226779659 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fa570022cb0 con 0x7fa56000c130 2024-09-16T05:23:07.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.528+0000 7fa57cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:23:07.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.528+0000 7fa57cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:23:07.529 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":66,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:23:07.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 10 librados: watch_flush enter 2024-09-16T05:23:07.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 10 librados: watch_flush exit 2024-09-16T05:23:07.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 20 client.104222.objecter shutdown clearing up homeless session... 2024-09-16T05:23:07.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 10 client.104222.objecter successfully canceled tick 2024-09-16T05:23:07.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 1 -- 172.21.15.114:0/2226779659 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa56003c720 msgr2=0x7fa56003ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 1 --2- 172.21.15.114:0/2226779659 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa56003c720 0x7fa56003ebe0 secure :-1 s=READY pgs=3593 cs=0 l=1 rev1=1 crypto rx=0x7fa5680ad000 tx=0x7fa5680a7070 comp rx=0 tx=0).stop 2024-09-16T05:23:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 10 monclient: shutdown 2024-09-16T05:23:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:23:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 1 -- 172.21.15.114:0/2226779659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa56000c130 msgr2=0x7fa580095ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 1 --2- 172.21.15.114:0/2226779659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa56000c130 0x7fa580095ad0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa570001520 tx=0x7fa5700019e0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 1 -- 172.21.15.114:0/2226779659 shutdown_connections 2024-09-16T05:23:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 1 --2- 172.21.15.114:0/2226779659 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa56003c720 0x7fa56003ebe0 unknown :-1 s=CLOSED pgs=3593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 1 --2- 172.21.15.114:0/2226779659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa56000c600 0x7fa5801cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 1 --2- 172.21.15.114:0/2226779659 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa56000c9e0 0x7fa5801ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 1 --2- 172.21.15.114:0/2226779659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa56000c130 0x7fa580095ad0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 1 -- 172.21.15.114:0/2226779659 >> 172.21.15.114:0/2226779659 conn(0x7fa5800a1fb0 msgr2=0x7fa58009da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:23:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 1 -- 172.21.15.114:0/2226779659 shutdown_connections 2024-09-16T05:23:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 1 -- 172.21.15.114:0/2226779659 wait complete. 2024-09-16T05:23:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.532+0000 7fa586565640 1 librados: shutdown 2024-09-16T05:23:07.548 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:23:07.549 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T05:23:07.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:23:07.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 10 monclient: get_monmap_and_config 2024-09-16T05:23:07.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 10 monclient: build_initial_monmap 2024-09-16T05:23:07.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 10 monclient: monmap: 2024-09-16T05:23:07.671 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:23:07.671 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.671 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:23:07.676718+0000 2024-09-16T05:23:07.671 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:23:07.676718+0000 2024-09-16T05:23:07.671 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:23:07.671 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.671 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:23:07.671 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:23:07.671 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:23:07.671 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:23:07.672 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:23:07.672 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 1 Processor -- start 2024-09-16T05:23:07.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 1 -- start start 2024-09-16T05:23:07.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 10 monclient: init 2024-09-16T05:23:07.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 10 monclient: _reopen_session rank -1 2024-09-16T05:23:07.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-16T05:23:07.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fda8c09b6c0 0x7fda8c0a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 10 monclient(hunting): picked mon.noname-c con 0x7fda8c09b6c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:23:07.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda8c0a1400 addr v1:172.21.15.114:6789/0 2024-09-16T05:23:07.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda8c0a1810 0x7fda8c13cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 10 monclient(hunting): picked mon.noname-b con 0x7fda8c0a1810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:23:07.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda8c09d4e0 con 0x7fda8c0a1400 2024-09-16T05:23:07.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fda8c09e390 con 0x7fda8c09b6c0 2024-09-16T05:23:07.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fda8c09ece0 con 0x7fda8c0a1810 2024-09-16T05:23:07.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 10 monclient(hunting): _renew_subs 2024-09-16T05:23:07.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda93bba640 10 monclient(hunting): authenticate will time out at 1872.306391s 2024-09-16T05:23:07.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda8c0a1810 0x7fda8c13cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda91155640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fda8c0a1400 0x7fda8c139640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55564/0 (socket says 172.21.15.114:55564) 2024-09-16T05:23:07.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda91155640 1 -- 172.21.15.114:0/213358886 learned_addr learned my addr 172.21.15.114:0/213358886 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:23:07.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda91956640 1 -- 172.21.15.114:0/213358886 >> v2:172.21.15.149:3300/0 conn(0x7fda8c09b6c0 msgr2=0x7fda8c0a0ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:23:07.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 10 monclient(hunting): get_auth_request con 0x7fda8c0a1810 auth_method 0 2024-09-16T05:23:07.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:07.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda91956640 1 --2- 172.21.15.114:0/213358886 >> v2:172.21.15.149:3300/0 conn(0x7fda8c09b6c0 0x7fda8c0a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:23:07.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:23:07.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:23:07.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:23:07.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda90954640 1 -- 172.21.15.114:0/213358886 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4032649074 0 0) 0x7fda8c09d4e0 con 0x7fda8c0a1400 2024-09-16T05:23:07.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda90954640 10 monclient(hunting): _negotiate 2024-09-16T05:23:07.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda90954640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda90954640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda90954640 10 monclient(hunting): my global_id is 74745 2024-09-16T05:23:07.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda90954640 1 -- 172.21.15.114:0/213358886 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda78003530 con 0x7fda8c0a1400 2024-09-16T05:23:07.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 10 monclient(hunting): handle_auth_done global_id 104225 payload 274 2024-09-16T05:23:07.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 1 -- 172.21.15.114:0/213358886 >> v2:172.21.15.149:3300/0 conn(0x7fda8c09b6c0 msgr2=0x7fda8c0a0ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:23:07.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 1 --2- 172.21.15.114:0/213358886 >> v2:172.21.15.149:3300/0 conn(0x7fda8c09b6c0 0x7fda8c0a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda90954640 1 -- 172.21.15.114:0/213358886 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2528863992 0 0) 0x7fda78003530 con 0x7fda8c0a1400 2024-09-16T05:23:07.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 1 -- 172.21.15.114:0/213358886 >> v1:172.21.15.114:6789/0 conn(0x7fda8c0a1400 legacy=0x7fda8c139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 10 monclient: _finish_hunting 0 2024-09-16T05:23:07.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 1 monclient: found mon.noname-b 2024-09-16T05:23:07.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:23:07.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:23:07.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda92157640 1 -- 172.21.15.114:0/213358886 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8c09f430 con 0x7fda8c0a1810 2024-09-16T05:23:07.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.672+0000 7fda90954640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:23:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda92157640 1 --2- 172.21.15.114:0/213358886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda8c0a1810 0x7fda8c13cd70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fda84009d90 tx=0x7fda8401c770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80256a42b0330db6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:23:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda90954640 1 -- 172.21.15.114:0/213358886 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda840ac070 con 0x7fda8c0a1810 2024-09-16T05:23:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda90954640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda90954640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:23:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda90954640 10 monclient: dump: 2024-09-16T05:23:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.686 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda90954640 10 monclient: _finish_auth 0 2024-09-16T05:23:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda90954640 10 monclient: _check_auth_tickets 2024-09-16T05:23:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda90954640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:23:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda90954640 1 -- 172.21.15.114:0/213358886 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda840a6470 con 0x7fda8c0a1810 2024-09-16T05:23:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 5 monclient: authenticate success, global_id 104225 2024-09-16T05:23:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:23:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda90954640 10 monclient: handle_config config(0 keys) 2024-09-16T05:23:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 10 monclient: get_monmap_and_config success 2024-09-16T05:23:07.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda90954640 1 -- 172.21.15.114:0/213358886 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c001820 con 0x7fda8c0a1810 2024-09-16T05:23:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 4 set_mon_vals no callback set 2024-09-16T05:23:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda90954640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda90954640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:23:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda90954640 10 monclient: dump: 2024-09-16T05:23:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.687 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 10 monclient: shutdown 2024-09-16T05:23:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:23:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 1 -- 172.21.15.114:0/213358886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda8c0a1810 msgr2=0x7fda8c13cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 1 --2- 172.21.15.114:0/213358886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda8c0a1810 0x7fda8c13cd70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fda84009d90 tx=0x7fda8401c770 comp rx=0 tx=0).stop 2024-09-16T05:23:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 1 -- 172.21.15.114:0/213358886 shutdown_connections 2024-09-16T05:23:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 1 --2- 172.21.15.114:0/213358886 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda8c0a1810 0x7fda8c13cd70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 1 --2- 172.21.15.114:0/213358886 >> v2:172.21.15.149:3300/0 conn(0x7fda8c09b6c0 0x7fda8c0a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 1 -- 172.21.15.114:0/213358886 >> 172.21.15.114:0/213358886 conn(0x7fda8c0a1fb0 msgr2=0x7fda8c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:23:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 1 -- 172.21.15.114:0/213358886 shutdown_connections 2024-09-16T05:23:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 1 -- 172.21.15.114:0/213358886 wait complete. 2024-09-16T05:23:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 10 monclient: build_initial_monmap 2024-09-16T05:23:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 10 monclient: monmap: 2024-09-16T05:23:07.688 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:23:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:23:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:23:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:23:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:23:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:23:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:23:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:23:07.689 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 1 librados: starting msgr at 2024-09-16T05:23:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 1 librados: starting objecter 2024-09-16T05:23:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 1 Processor -- start 2024-09-16T05:23:07.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 1 -- start start 2024-09-16T05:23:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 1 librados: setting wanted keys 2024-09-16T05:23:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 1 librados: calling monclient init 2024-09-16T05:23:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.676+0000 7fda93bba640 10 monclient: init 2024-09-16T05:23:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 monclient: _reopen_session rank -1 2024-09-16T05:23:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:23:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda8c09b6c0 0x7fda8c098210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 monclient(hunting): picked mon.noname-c con 0x7fda8c09b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:23:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fda8c0a1400 0x7fda8c08fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 monclient(hunting): picked mon.noname-b con 0x7fda8c0a1400 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:23:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c0a1810 0x7fda8c08ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 monclient(hunting): picked mon.noname-a con 0x7fda8c0a1810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:23:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c09cad0 con 0x7fda8c0a1810 2024-09-16T05:23:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c09dd40 con 0x7fda8c0a1400 2024-09-16T05:23:07.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fda8c0994b0 con 0x7fda8c09b6c0 2024-09-16T05:23:07.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 monclient(hunting): _renew_subs 2024-09-16T05:23:07.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 monclient(hunting): authenticate will time out at 1872.312330s 2024-09-16T05:23:07.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91155640 1 -- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fda8c0a1400 msgr2=0x7fda8c08fa10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:23:07.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91155640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fda8c0a1400 0x7fda8c08fa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:23:07.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda8c09b6c0 0x7fda8c098210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda8c09b6c0 0x7fda8c098210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:46122/0 (socket says 172.21.15.114:46122) 2024-09-16T05:23:07.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 1 -- 172.21.15.114:0/2765943306 learned_addr learned my addr 172.21.15.114:0/2765943306 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:23:07.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 10 monclient(hunting): get_auth_request con 0x7fda8c09b6c0 auth_method 0 2024-09-16T05:23:07.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:07.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda92157640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c0a1810 0x7fda8c08ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda92157640 10 monclient(hunting): get_auth_request con 0x7fda8c0a1810 auth_method 0 2024-09-16T05:23:07.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda92157640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:07.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda92157640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda92157640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:23:07.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:23:07.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:23:07.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda92157640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:23:07.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda92157640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:23:07.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda92157640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:23:07.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 10 monclient(hunting): handle_auth_done global_id 104228 payload 824 2024-09-16T05:23:07.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 1 -- 172.21.15.114:0/2765943306 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fda8c0a1400 msgr2=0x7fda8c08fa10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:23:07.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fda8c0a1400 0x7fda8c08fa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 1 -- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c0a1810 msgr2=0x7fda8c08ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c0a1810 0x7fda8c08ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 10 monclient: _finish_hunting 0 2024-09-16T05:23:07.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 1 monclient: found mon.noname-c 2024-09-16T05:23:07.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:23:07.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:23:07.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 1 -- 172.21.15.114:0/2765943306 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8c09aed0 con 0x7fda8c09b6c0 2024-09-16T05:23:07.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91956640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda8c09b6c0 0x7fda8c098210 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fda8800b530 tx=0x7fda8800b9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:23:07.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 client.?.objecter ms_handle_connect 0x7fda8c09b6c0 2024-09-16T05:23:07.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:23:07.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 1 -- 172.21.15.114:0/2765943306 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88013020 con 0x7fda8c09b6c0 2024-09-16T05:23:07.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:23:07.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 monclient: dump: 2024-09-16T05:23:07.694 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.694 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.694 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.694 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.694 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.694 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.694 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.694 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.695 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.695 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 monclient: _finish_auth 0 2024-09-16T05:23:07.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 monclient: _check_auth_tickets 2024-09-16T05:23:07.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:23:07.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 1 -- 172.21.15.114:0/2765943306 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda88003df0 con 0x7fda8c09b6c0 2024-09-16T05:23:07.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:23:07.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 1 -- 172.21.15.114:0/2765943306 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda880052b0 con 0x7fda8c09b6c0 2024-09-16T05:23:07.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:23:07.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 monclient: dump: 2024-09-16T05:23:07.695 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.695 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.695 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.695 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.696 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.696 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.696 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.696 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.696 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.696 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 5 monclient: authenticate success, global_id 104228 2024-09-16T05:23:07.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda83fff640 4 set_mon_vals no callback set 2024-09-16T05:23:07.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 monclient: _renew_subs 2024-09-16T05:23:07.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:23:07.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 1 -- 172.21.15.114:0/2765943306 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda8c09c6a0 con 0x7fda8c09b6c0 2024-09-16T05:23:07.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 client.104228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:23:07.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 monclient: _renew_subs 2024-09-16T05:23:07.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:23:07.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 1 -- 172.21.15.114:0/2765943306 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fda8c093680 con 0x7fda8c09b6c0 2024-09-16T05:23:07.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 1 librados: init done 2024-09-16T05:23:07.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:23:07.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c090490 0x7fda8c0709b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda93bba640 1 -- 172.21.15.114:0/2765943306 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c072c00 con 0x7fda8c090490 2024-09-16T05:23:07.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91155640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c090490 0x7fda8c0709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 1 -- 172.21.15.114:0/2765943306 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fda88016020 con 0x7fda8c09b6c0 2024-09-16T05:23:07.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91155640 10 monclient: get_auth_request con 0x7fda8c090490 auth_method 0 2024-09-16T05:23:07.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91155640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:07.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91155640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91155640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91155640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:23:07.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91155640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:23:07.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91155640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:23:07.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 1 --2- 172.21.15.114:0/2765943306 >> v2:172.21.15.149:6812/1185531135 conn(0x7fda700377c0 0x7fda70039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 1 -- 172.21.15.114:0/2765943306 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(620..620 src has 1..620) ==== 5973+0+0 (secure 0 0 0) 0x7fda88048bb0 con 0x7fda8c09b6c0 2024-09-16T05:23:07.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 client.104228.objecter ms_dispatch 0x7fda8c000ba0 osd_map(620..620 src has 1..620) 2024-09-16T05:23:07.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 3 client.104228.objecter handle_osd_map got epochs [620,620] > 0 2024-09-16T05:23:07.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 3 client.104228.objecter handle_osd_map decoding full epoch 620 2024-09-16T05:23:07.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91155640 10 monclient(hunting): handle_auth_done global_id 74751 payload 824 2024-09-16T05:23:07.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91155640 1 -- 172.21.15.114:0/2765943306 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda8c090870 con 0x7fda8c090490 2024-09-16T05:23:07.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 20 client.104228.objecter dump_active .. 0 homeless 2024-09-16T05:23:07.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 client.104228.objecter ms_handle_connect 0x7fda8c090490 2024-09-16T05:23:07.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 client.104228.objecter resend_mon_ops 2024-09-16T05:23:07.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda91155640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c090490 0x7fda8c0709b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fda8c091270 tx=0x7fda7c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:23:07.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda92157640 1 --2- 172.21.15.114:0/2765943306 >> v2:172.21.15.149:6812/1185531135 conn(0x7fda700377c0 0x7fda70039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda92157640 10 monclient: get_auth_request con 0x7fda700377c0 auth_method 0 2024-09-16T05:23:07.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 1 -- 172.21.15.114:0/2765943306 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c007270 con 0x7fda8c090490 2024-09-16T05:23:07.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:23:07.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 monclient: dump: 2024-09-16T05:23:07.699 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.699 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.699 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.699 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.699 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.699 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.699 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.699 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.699 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.700 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 1 -- 172.21.15.114:0/2765943306 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fda7c007270 con 0x7fda8c090490 2024-09-16T05:23:07.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:23:07.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:23:07.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 1 -- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c090490 msgr2=0x7fda8c0709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c090490 0x7fda8c0709b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fda8c091270 tx=0x7fda7c00cb10 comp rx=0 tx=0).stop 2024-09-16T05:23:07.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 1 -- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c090490 msgr2=0x7fda8c0709b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:23:07.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c090490 0x7fda8c0709b0 secure :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fda8c091270 tx=0x7fda7c00cb10 comp rx=0 tx=0).stop 2024-09-16T05:23:07.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda827fc640 10 client.104228.objecter ms_handle_connect 0x7fda700377c0 2024-09-16T05:23:07.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.680+0000 7fda92157640 1 --2- 172.21.15.114:0/2765943306 >> v2:172.21.15.149:6812/1185531135 conn(0x7fda700377c0 0x7fda70039c80 secure :-1 s=READY pgs=3594 cs=0 l=1 rev1=1 crypto rx=0x7fda840ae000 tx=0x7fda840a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:23:07.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:23:07.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c070ef0 0x7fda8c071300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 1 -- 172.21.15.114:0/2765943306 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c0a0ec0 con 0x7fda8c070ef0 2024-09-16T05:23:07.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda91155640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c070ef0 0x7fda8c071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda91155640 10 monclient: get_auth_request con 0x7fda8c070ef0 auth_method 0 2024-09-16T05:23:07.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda91155640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:07.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda91155640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda91155640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda91155640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:23:07.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda91155640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:23:07.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda91155640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:23:07.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda91155640 10 monclient(hunting): handle_auth_done global_id 74754 payload 824 2024-09-16T05:23:07.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda91155640 1 -- 172.21.15.114:0/2765943306 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fda8c091f50 con 0x7fda8c070ef0 2024-09-16T05:23:07.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda827fc640 10 client.104228.objecter ms_handle_connect 0x7fda8c070ef0 2024-09-16T05:23:07.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda91155640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c070ef0 0x7fda8c071300 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fda8c091270 tx=0x7fda7c00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:23:07.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda827fc640 10 client.104228.objecter resend_mon_ops 2024-09-16T05:23:07.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda827fc640 1 -- 172.21.15.114:0/2765943306 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c01ba30 con 0x7fda8c070ef0 2024-09-16T05:23:07.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:23:07.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda827fc640 10 monclient: dump: 2024-09-16T05:23:07.703 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.703 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.703 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.703 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.703 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.703 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.703 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.704 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.704 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.704 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda827fc640 1 -- 172.21.15.114:0/2765943306 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fda7c01ba30 con 0x7fda8c070ef0 2024-09-16T05:23:07.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda827fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:23:07.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda827fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:23:07.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda827fc640 1 -- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c070ef0 msgr2=0x7fda8c071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda827fc640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c070ef0 0x7fda8c071300 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fda8c091270 tx=0x7fda7c00f000 comp rx=0 tx=0).stop 2024-09-16T05:23:07.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda827fc640 1 -- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c070ef0 msgr2=0x7fda8c071300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:23:07.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda827fc640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda8c070ef0 0x7fda8c071300 secure :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fda8c091270 tx=0x7fda7c00f000 comp rx=0 tx=0).stop 2024-09-16T05:23:07.704 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:23:07.705 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:23:07.705 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:23:07.705 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:23:07.705 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 66, 2024-09-16T05:23:07.705 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 546744, 2024-09-16T05:23:07.705 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:23:07.705 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:23:07.705 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:23:07.705 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:23:07.705 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 1, 2024-09-16T05:23:07.705 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:23:07.705 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:23:07.705 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:23:07.706 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:23:07.706 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:23:07.706 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:23:07.706 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:23:07.706 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:23:07.706 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:23:07.706 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:23:07.706 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:23:07.706 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:23:07.706 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:23:07.706 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:23:07.706 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:23:07.706 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:23:07.706 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:23:07.707 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:23:07.707 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:23:07.707 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:23:07.707 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:23:07.707 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:23:07.707 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:23:07.707 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:23:07.707 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:23:07.707 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:23:07.707 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:23:07.707 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:23:07.707 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:23:07.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.708 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:23:07.708 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:23:07.708 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:23:07.708 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:23:07.708 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:23:07.708 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:23:07.708 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:23:07.708 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:23:07.708 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:23:07.708 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:23:07.708 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:23:07.708 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:23:07.708 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:23:07.708 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:23:07.709 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:23:07.709 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:23:07.709 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:23:07.709 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:23:07.709 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:23:07.709 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:23:07.709 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:23:07.709 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:23:07.709 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:23:07.709 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:23:07.709 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:23:07.709 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:23:07.709 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:23:07.709 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:23:07.710 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:23:07.710 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:23:07.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.710 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:23:07.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.710 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:23:07.710 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:23:07.710 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:23:07.710 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:23:07.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.710 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:23:07.710 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:23:07.710 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:23:07.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.711 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:23:07.711 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:23:07.711 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:23:07.711 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:23:07.711 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:23:07.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.711 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:23:07.711 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:23:07.711 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:23:07.711 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:23:07.711 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:23:07.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.712 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:23:07.712 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:23:07.712 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:23:07.712 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:23:07.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.712 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:23:07.712 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:23:07.712 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:23:07.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.712 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:23:07.712 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:23:07.713 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:23:07.713 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:23:07.713 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:23:07.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.713 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:23:07.713 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:23:07.713 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:23:07.713 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:23:07.713 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:23:07.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.713 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.713 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:23:07.713 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:23:07.713 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:23:07.714 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:23:07.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.714 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:23:07.714 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:23:07.714 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:23:07.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.714 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:23:07.714 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:23:07.714 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:23:07.714 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:23:07.714 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:23:07.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.714 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:23:07.715 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:23:07.715 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:23:07.715 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:23:07.715 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:23:07.715 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:23:07.715 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:23:07.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.715 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:23:07.715 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:23:07.715 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.715 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:23:07.715 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:23:07.715 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:23:07.715 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:23:07.716 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:23:07.716 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:23:07.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.716 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:23:07.716 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:23:07.716 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:23:07.716 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:23:07.716 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:23:07.716 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:23:07.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.716 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:23:07.716 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:23:07.716 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:23:07.717 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:23:07.717 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:23:07.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.717 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:23:07.717 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:23:07.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 10 librados: watch_flush enter 2024-09-16T05:23:07.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 10 librados: watch_flush exit 2024-09-16T05:23:07.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 20 client.104228.objecter shutdown clearing up homeless session... 2024-09-16T05:23:07.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 10 client.104228.objecter successfully canceled tick 2024-09-16T05:23:07.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 1 -- 172.21.15.114:0/2765943306 >> v2:172.21.15.149:6812/1185531135 conn(0x7fda700377c0 msgr2=0x7fda70039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 1 --2- 172.21.15.114:0/2765943306 >> v2:172.21.15.149:6812/1185531135 conn(0x7fda700377c0 0x7fda70039c80 secure :-1 s=READY pgs=3594 cs=0 l=1 rev1=1 crypto rx=0x7fda840ae000 tx=0x7fda840a7000 comp rx=0 tx=0).stop 2024-09-16T05:23:07.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 10 monclient: shutdown 2024-09-16T05:23:07.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:23:07.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 1 -- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda8c09b6c0 msgr2=0x7fda8c098210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda8c09b6c0 0x7fda8c098210 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fda8800b530 tx=0x7fda8800b9f0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda92157640 1 -- 172.21.15.114:0/2765943306 reap_dead start 2024-09-16T05:23:07.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 1 -- 172.21.15.114:0/2765943306 shutdown_connections 2024-09-16T05:23:07.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 1 --2- 172.21.15.114:0/2765943306 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda8c09b6c0 0x7fda8c098210 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 1 -- 172.21.15.114:0/2765943306 >> 172.21.15.114:0/2765943306 conn(0x7fda8c0a1fb0 msgr2=0x7fda8c0a32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:23:07.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 1 -- 172.21.15.114:0/2765943306 shutdown_connections 2024-09-16T05:23:07.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 1 -- 172.21.15.114:0/2765943306 wait complete. 2024-09-16T05:23:07.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.688+0000 7fda93bba640 1 librados: shutdown 2024-09-16T05:23:07.720 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:23:07.720 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T05:23:07.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:23:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 10 monclient: get_monmap_and_config 2024-09-16T05:23:07.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 10 monclient: build_initial_monmap 2024-09-16T05:23:07.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 10 monclient: monmap: 2024-09-16T05:23:07.835 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:23:07.835 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.835 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:23:07.840733+0000 2024-09-16T05:23:07.835 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:23:07.840733+0000 2024-09-16T05:23:07.835 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:23:07.835 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.835 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:23:07.835 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:23:07.835 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:23:07.835 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:23:07.836 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:23:07.836 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 1 Processor -- start 2024-09-16T05:23:07.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 1 -- start start 2024-09-16T05:23:07.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 10 monclient: init 2024-09-16T05:23:07.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 10 monclient: _reopen_session rank -1 2024-09-16T05:23:07.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-16T05:23:07.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22440a0ec0 0x7f224409dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 10 monclient(hunting): picked mon.noname-b con 0x7f22440a0ec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:23:07.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f224409e000 addr v1:172.21.15.149:6789/0 2024-09-16T05:23:07.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2244139640 addr v1:172.21.15.114:6789/0 2024-09-16T05:23:07.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f224409a760 con 0x7f2244139640 2024-09-16T05:23:07.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22440a1870 con 0x7f224409e000 2024-09-16T05:23:07.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f22440a12a0 con 0x7f22440a0ec0 2024-09-16T05:23:07.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 10 monclient(hunting): _renew_subs 2024-09-16T05:23:07.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2249147640 10 monclient(hunting): authenticate will time out at 1872.470106s 2024-09-16T05:23:07.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f224259c640 1 -- >> v1:172.21.15.149:6789/0 conn(0x7f224409e000 legacy=0x7f224409e3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.149:6789/0 2024-09-16T05:23:07.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f224359e640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2244139640 0x7f224413ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55566/0 (socket says 172.21.15.114:55566) 2024-09-16T05:23:07.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f224359e640 1 -- 172.21.15.114:0/3435039389 learned_addr learned my addr 172.21.15.114:0/3435039389 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:23:07.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 1 --2- 172.21.15.114:0/3435039389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22440a0ec0 0x7f224409dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 10 monclient(hunting): get_auth_request con 0x7f22440a0ec0 auth_method 0 2024-09-16T05:23:07.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:07.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:23:07.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:23:07.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:23:07.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2241d9b640 1 -- 172.21.15.114:0/3435039389 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2879472767 0 0) 0x7f224409a760 con 0x7f2244139640 2024-09-16T05:23:07.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2241d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:23:07.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2241d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2241d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2241d9b640 10 monclient(hunting): my global_id is 74757 2024-09-16T05:23:07.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2241d9b640 1 -- 172.21.15.114:0/3435039389 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2230003530 con 0x7f2244139640 2024-09-16T05:23:07.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 10 monclient(hunting): handle_auth_done global_id 104231 payload 274 2024-09-16T05:23:07.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 1 -- 172.21.15.114:0/3435039389 >> v1:172.21.15.149:6789/0 conn(0x7f224409e000 legacy=0x7f224409e3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:23:07.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 1 -- 172.21.15.114:0/3435039389 >> v1:172.21.15.114:6789/0 conn(0x7f2244139640 legacy=0x7f224413ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 10 monclient: _finish_hunting 0 2024-09-16T05:23:07.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 1 monclient: found mon.noname-b 2024-09-16T05:23:07.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:23:07.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:23:07.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.836+0000 7f2242d9d640 1 -- 172.21.15.114:0/3435039389 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2244099790 con 0x7f22440a0ec0 2024-09-16T05:23:07.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 1 --2- 172.21.15.114:0/3435039389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22440a0ec0 0x7f224409dac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f223c009d90 tx=0x7f223c01c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fef4064591c379e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:23:07.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2241d9b640 1 -- 172.21.15.114:0/3435039389 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223c0b2070 con 0x7f22440a0ec0 2024-09-16T05:23:07.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2241d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2241d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:23:07.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2241d9b640 10 monclient: dump: 2024-09-16T05:23:07.849 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.849 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.849 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.850 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.850 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.850 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.850 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.850 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.850 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.850 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2241d9b640 10 monclient: _finish_auth 0 2024-09-16T05:23:07.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2241d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:23:07.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2241d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:23:07.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 5 monclient: authenticate success, global_id 104231 2024-09-16T05:23:07.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:23:07.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2241d9b640 1 -- 172.21.15.114:0/3435039389 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f223c0a5870 con 0x7f22440a0ec0 2024-09-16T05:23:07.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2241d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:23:07.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2241d9b640 1 -- 172.21.15.114:0/3435039389 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223c0a5c20 con 0x7f22440a0ec0 2024-09-16T05:23:07.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 10 monclient: get_monmap_and_config success 2024-09-16T05:23:07.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 4 set_mon_vals no callback set 2024-09-16T05:23:07.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2241d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2241d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:23:07.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2241d9b640 10 monclient: dump: 2024-09-16T05:23:07.851 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.851 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.851 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.851 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.851 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.851 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.851 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.852 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.852 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.852 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 10 monclient: shutdown 2024-09-16T05:23:07.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:23:07.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 -- 172.21.15.114:0/3435039389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22440a0ec0 msgr2=0x7f224409dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 --2- 172.21.15.114:0/3435039389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22440a0ec0 0x7f224409dac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f223c009d90 tx=0x7f223c01c8a0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 -- 172.21.15.114:0/3435039389 shutdown_connections 2024-09-16T05:23:07.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 --2- 172.21.15.114:0/3435039389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22440a0ec0 0x7f224409dac0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 -- 172.21.15.114:0/3435039389 >> 172.21.15.114:0/3435039389 conn(0x7f22440a1fb0 msgr2=0x7f22440a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:23:07.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 -- 172.21.15.114:0/3435039389 shutdown_connections 2024-09-16T05:23:07.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 -- 172.21.15.114:0/3435039389 wait complete. 2024-09-16T05:23:07.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 10 monclient: build_initial_monmap 2024-09-16T05:23:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 10 monclient: monmap: 2024-09-16T05:23:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:23:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:23:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:23:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:23:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:23:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:23:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:23:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:23:07.853 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 librados: starting msgr at 2024-09-16T05:23:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 librados: starting objecter 2024-09-16T05:23:07.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 Processor -- start 2024-09-16T05:23:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 -- start start 2024-09-16T05:23:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 librados: setting wanted keys 2024-09-16T05:23:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 librados: calling monclient init 2024-09-16T05:23:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 10 monclient: init 2024-09-16T05:23:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 10 monclient: _reopen_session rank -1 2024-09-16T05:23:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:23:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f224409e000 0x7f2244098460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 10 monclient(hunting): picked mon.noname-c con 0x7f224409e000 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:23:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f22440a0ec0 0x7f224408d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 10 monclient(hunting): picked mon.noname-b con 0x7f22440a0ec0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:23:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2244139640 0x7f224408d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 10 monclient(hunting): picked mon.noname-a con 0x7f2244139640 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:23:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f224413faf0 con 0x7f2244139640 2024-09-16T05:23:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f224413f970 con 0x7f22440a0ec0 2024-09-16T05:23:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f224413fc70 con 0x7f224409e000 2024-09-16T05:23:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 10 monclient(hunting): _renew_subs 2024-09-16T05:23:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2249147640 10 monclient(hunting): authenticate will time out at 1872.475496s 2024-09-16T05:23:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f224259c640 1 -- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f22440a0ec0 msgr2=0x7f224408d000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.149:3300/0 2024-09-16T05:23:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f224409e000 0x7f2244098460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f224259c640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f22440a0ec0 0x7f224408d000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:23:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f224359e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2244139640 0x7f224408d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f224409e000 0x7f2244098460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:46136/0 (socket says 172.21.15.114:46136) 2024-09-16T05:23:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 1 -- 172.21.15.114:0/404349527 learned_addr learned my addr 172.21.15.114:0/404349527 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:23:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 10 monclient(hunting): get_auth_request con 0x7f224409e000 auth_method 0 2024-09-16T05:23:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f224359e640 10 monclient(hunting): get_auth_request con 0x7f2244139640 auth_method 0 2024-09-16T05:23:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f224359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f224359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f224359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f224359e640 1 -- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2244139640 msgr2=0x7f224408d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T05:23:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f224359e640 1 -- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2244139640 msgr2=0x7f224408d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:23:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f224359e640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2244139640 0x7f224408d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:23:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:23:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:23:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f224359e640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2244139640 0x7f224408d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:23:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:23:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 10 monclient(hunting): handle_auth_done global_id 104234 payload 824 2024-09-16T05:23:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 1 -- 172.21.15.114:0/404349527 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f22440a0ec0 msgr2=0x7f224408d000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:23:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f22440a0ec0 0x7f224408d000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 1 -- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2244139640 msgr2=0x7f224408d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:23:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2244139640 0x7f224408d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 10 monclient: _finish_hunting 0 2024-09-16T05:23:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 1 monclient: found mon.noname-c 2024-09-16T05:23:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:23:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:23:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.840+0000 7f2242d9d640 1 -- 172.21.15.114:0/404349527 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2244090a30 con 0x7f224409e000 2024-09-16T05:23:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f2242d9d640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f224409e000 0x7f2244098460 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f223c01af50 tx=0x7f223c019640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:23:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 client.?.objecter ms_handle_connect 0x7f224409e000 2024-09-16T05:23:07.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:23:07.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 1 -- 172.21.15.114:0/404349527 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223c0bd030 con 0x7f224409e000 2024-09-16T05:23:07.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:23:07.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 monclient: dump: 2024-09-16T05:23:07.858 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.858 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.858 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.858 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.858 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.858 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.858 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.858 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.858 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.858 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:23:07.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:23:07.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:23:07.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 1 -- 172.21.15.114:0/404349527 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f223c0039a0 con 0x7f224409e000 2024-09-16T05:23:07.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f2249147640 5 monclient: authenticate success, global_id 104234 2024-09-16T05:23:07.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f2249147640 10 monclient: _renew_subs 2024-09-16T05:23:07.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f2249147640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:23:07.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f2249147640 1 -- 172.21.15.114:0/404349527 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2244090ff0 con 0x7f224409e000 2024-09-16T05:23:07.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:23:07.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 1 -- 172.21.15.114:0/404349527 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223c0b2040 con 0x7f224409e000 2024-09-16T05:23:07.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:23:07.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 monclient: dump: 2024-09-16T05:23:07.859 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.860 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f2249147640 10 client.104234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:23:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f2249147640 10 monclient: _renew_subs 2024-09-16T05:23:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f2249147640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:23:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f2249147640 1 -- 172.21.15.114:0/404349527 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f224408e940 con 0x7f224409e000 2024-09-16T05:23:07.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f2249147640 1 librados: init done 2024-09-16T05:23:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f224159a640 4 set_mon_vals no callback set 2024-09-16T05:23:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f2249147640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:23:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f2249147640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22440989a0 0x7f224408da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f2249147640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f2249147640 1 -- 172.21.15.114:0/404349527 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2244069390 con 0x7f22440989a0 2024-09-16T05:23:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f224259c640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22440989a0 0x7f224408da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f224259c640 10 monclient: get_auth_request con 0x7f22440989a0 auth_method 0 2024-09-16T05:23:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f224259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f224259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f224259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 1 -- 172.21.15.114:0/404349527 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f223c0c0070 con 0x7f224409e000 2024-09-16T05:23:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f224259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:23:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f224259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:23:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f224259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:23:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f224259c640 10 monclient(hunting): handle_auth_done global_id 104237 payload 824 2024-09-16T05:23:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f224259c640 1 -- 172.21.15.114:0/404349527 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f224408f080 con 0x7f22440989a0 2024-09-16T05:23:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 1 --2- 172.21.15.114:0/404349527 >> v2:172.21.15.149:6812/1185531135 conn(0x7f22240377c0 0x7f2224039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 1 -- 172.21.15.114:0/404349527 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(620..620 src has 1..620) ==== 5973+0+0 (secure 0 0 0) 0x7f223c0e3bd0 con 0x7f224409e000 2024-09-16T05:23:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 client.104234.objecter ms_dispatch 0x7f2244000ba0 osd_map(620..620 src has 1..620) 2024-09-16T05:23:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 3 client.104234.objecter handle_osd_map got epochs [620,620] > 0 2024-09-16T05:23:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 3 client.104234.objecter handle_osd_map decoding full epoch 620 2024-09-16T05:23:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f224259c640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22440989a0 0x7f224408da80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f224413c830 tx=0x7f222c00b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:23:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 20 client.104234.objecter dump_active .. 0 homeless 2024-09-16T05:23:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 client.104234.objecter ms_handle_connect 0x7f22440989a0 2024-09-16T05:23:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 client.104234.objecter resend_mon_ops 2024-09-16T05:23:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 1 -- 172.21.15.114:0/404349527 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c003c70 con 0x7f22440989a0 2024-09-16T05:23:07.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:23:07.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 monclient: dump: 2024-09-16T05:23:07.863 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.863 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.863 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.863 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.863 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.863 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.863 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.863 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.863 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.863 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f224359e640 1 --2- 172.21.15.114:0/404349527 >> v2:172.21.15.149:6812/1185531135 conn(0x7f22240377c0 0x7f2224039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f224359e640 10 monclient: get_auth_request con 0x7f22240377c0 auth_method 0 2024-09-16T05:23:07.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 1 -- 172.21.15.114:0/404349527 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f222c003c70 con 0x7f22440989a0 2024-09-16T05:23:07.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:23:07.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:23:07.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 1 -- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22440989a0 msgr2=0x7f224408da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22440989a0 0x7f224408da80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f224413c830 tx=0x7f222c00b8c0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 1 -- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22440989a0 msgr2=0x7f224408da80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:23:07.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f22440989a0 0x7f224408da80 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f224413c830 tx=0x7f222c00b8c0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f224359e640 1 --2- 172.21.15.114:0/404349527 >> v2:172.21.15.149:6812/1185531135 conn(0x7f22240377c0 0x7f2224039c80 secure :-1 s=READY pgs=3595 cs=0 l=1 rev1=1 crypto rx=0x7f22340011f0 tx=0x7f2234009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:23:07.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.844+0000 7f223b7fe640 10 client.104234.objecter ms_handle_connect 0x7f22240377c0 2024-09-16T05:23:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:23:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2244138b50 0x7f224406f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:23:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 10 monclient(hunting): start opening mon connection 2024-09-16T05:23:07.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 1 -- 172.21.15.114:0/404349527 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2244090620 con 0x7f2244138b50 2024-09-16T05:23:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f224259c640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2244138b50 0x7f224406f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:23:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f224259c640 10 monclient: get_auth_request con 0x7f2244138b50 auth_method 0 2024-09-16T05:23:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f224259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:23:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f224259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:23:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f224259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:23:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f224259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:23:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f224259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:23:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f224259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:23:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f224259c640 10 monclient(hunting): handle_auth_done global_id 104240 payload 824 2024-09-16T05:23:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f224259c640 1 -- 172.21.15.114:0/404349527 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f22440907e0 con 0x7f2244138b50 2024-09-16T05:23:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f224259c640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2244138b50 0x7f224406f9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f222c013000 tx=0x7f222c005a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:23:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f223b7fe640 10 client.104234.objecter ms_handle_connect 0x7f2244138b50 2024-09-16T05:23:07.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f223b7fe640 10 client.104234.objecter resend_mon_ops 2024-09-16T05:23:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f223b7fe640 1 -- 172.21.15.114:0/404349527 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c0055d0 con 0x7f2244138b50 2024-09-16T05:23:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f223b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:23:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f223b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:23:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f223b7fe640 10 monclient: dump: 2024-09-16T05:23:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:23:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:23:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:23:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:23:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:23:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:23:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:23:07.868 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:23:07.869 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:23:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f223b7fe640 1 -- 172.21.15.114:0/404349527 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f222c0055d0 con 0x7f2244138b50 2024-09-16T05:23:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f223b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:23:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f223b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:23:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f223b7fe640 1 -- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2244138b50 msgr2=0x7f224406f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f223b7fe640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2244138b50 0x7f224406f9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f222c013000 tx=0x7f222c005a80 comp rx=0 tx=0).stop 2024-09-16T05:23:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f223b7fe640 1 -- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2244138b50 msgr2=0x7f224406f9a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:23:07.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f223b7fe640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2244138b50 0x7f224406f9a0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f222c013000 tx=0x7f222c005a80 comp rx=0 tx=0).stop 2024-09-16T05:23:07.869 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:23:07.870 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:23:07.870 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:23:07.870 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:23:07.870 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 66, 2024-09-16T05:23:07.870 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 164017, 2024-09-16T05:23:07.870 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:23:07.870 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:23:07.870 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:23:07.870 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:23:07.870 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:23:07.870 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:23:07.870 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:23:07.870 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:23:07.871 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:23:07.871 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:23:07.871 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:23:07.871 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:23:07.871 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:23:07.871 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:23:07.871 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:23:07.871 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:23:07.871 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:23:07.871 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:23:07.871 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:23:07.871 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:23:07.872 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:23:07.872 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:23:07.872 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:23:07.872 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:23:07.872 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:23:07.872 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:23:07.872 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:23:07.872 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:23:07.872 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:23:07.872 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:23:07.872 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:23:07.872 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:23:07.872 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:23:07.873 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:23:07.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.873 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:23:07.873 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:23:07.873 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:23:07.873 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:23:07.873 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:23:07.873 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:23:07.873 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:23:07.873 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:23:07.873 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:23:07.873 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:23:07.873 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:23:07.874 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:23:07.874 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:23:07.874 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:23:07.874 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:23:07.874 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:23:07.874 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:23:07.874 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:23:07.874 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:23:07.874 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:23:07.874 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:23:07.874 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:23:07.874 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:23:07.874 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:23:07.875 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:23:07.875 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:23:07.875 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:23:07.875 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:23:07.875 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:23:07.875 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:23:07.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.875 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:23:07.875 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.875 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:23:07.875 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:23:07.875 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:23:07.875 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:23:07.876 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.876 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:23:07.876 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:23:07.876 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:23:07.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.876 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.876 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:23:07.876 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:23:07.876 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:23:07.876 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:23:07.876 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:23:07.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.876 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:23:07.877 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:23:07.877 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:23:07.877 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:23:07.877 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:23:07.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.877 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.877 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:23:07.877 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:23:07.877 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:23:07.877 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:23:07.877 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.877 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:23:07.877 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:23:07.878 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:23:07.878 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.878 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.878 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:23:07.878 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:23:07.878 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:23:07.878 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:23:07.878 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:23:07.878 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.878 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:23:07.878 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:23:07.878 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:23:07.878 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:23:07.879 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:23:07.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.879 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:23:07.879 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:23:07.879 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:23:07.879 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:23:07.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.879 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:23:07.879 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:23:07.879 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:23:07.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.880 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:23:07.880 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:23:07.880 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:23:07.880 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:23:07.880 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:23:07.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.880 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:23:07.880 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:23:07.880 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:23:07.880 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:23:07.880 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:23:07.880 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:23:07.880 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:23:07.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.881 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:23:07.881 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:23:07.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.881 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:23:07.881 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:23:07.881 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:23:07.881 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:23:07.881 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:23:07.881 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:23:07.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.881 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:23:07.881 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:23:07.882 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:23:07.882 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:23:07.882 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:23:07.882 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:23:07.882 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.882 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:23:07.882 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:23:07.882 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:23:07.882 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:23:07.882 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:23:07.882 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:23:07.882 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:23:07.882 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:23:07.883 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:23:07.883 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:23:07.883 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:23:07.883 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:23:07.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:23:07.883 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:23:07.883 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:23:07.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 10 librados: watch_flush enter 2024-09-16T05:23:07.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 10 librados: watch_flush exit 2024-09-16T05:23:07.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 20 client.104234.objecter shutdown clearing up homeless session... 2024-09-16T05:23:07.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 10 client.104234.objecter successfully canceled tick 2024-09-16T05:23:07.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 1 -- 172.21.15.114:0/404349527 >> v2:172.21.15.149:6812/1185531135 conn(0x7f22240377c0 msgr2=0x7f2224039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 1 --2- 172.21.15.114:0/404349527 >> v2:172.21.15.149:6812/1185531135 conn(0x7f22240377c0 0x7f2224039c80 secure :-1 s=READY pgs=3595 cs=0 l=1 rev1=1 crypto rx=0x7f22340011f0 tx=0x7f2234009040 comp rx=0 tx=0).stop 2024-09-16T05:23:07.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 10 monclient: shutdown 2024-09-16T05:23:07.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:23:07.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 1 -- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f224409e000 msgr2=0x7f2244098460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:23:07.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f224409e000 0x7f2244098460 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f223c01af50 tx=0x7f223c019640 comp rx=0 tx=0).stop 2024-09-16T05:23:07.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f224359e640 1 -- 172.21.15.114:0/404349527 reap_dead start 2024-09-16T05:23:07.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 1 -- 172.21.15.114:0/404349527 shutdown_connections 2024-09-16T05:23:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 1 --2- 172.21.15.114:0/404349527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f224409e000 0x7f2244098460 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:23:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 1 -- 172.21.15.114:0/404349527 >> 172.21.15.114:0/404349527 conn(0x7f22440a1fb0 msgr2=0x7f224413aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:23:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 1 -- 172.21.15.114:0/404349527 shutdown_connections 2024-09-16T05:23:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 1 -- 172.21.15.114:0/404349527 wait complete. 2024-09-16T05:23:07.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:23:07.852+0000 7f2249147640 1 librados: shutdown 2024-09-16T05:23:07.886 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:23:07.886 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:24:37.887 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:24:37.887 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T05:24:37.888 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T05:24:37.888 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T05:24:37.891 INFO:tasks.ceph.mon.b:Started 2024-09-16T05:24:37.891 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:24:37.892 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:24:38.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 10 monclient: get_monmap_and_config 2024-09-16T05:24:38.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 10 monclient: build_initial_monmap 2024-09-16T05:24:38.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 10 monclient: monmap: 2024-09-16T05:24:38.023 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:38.023 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:38.023 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:24:38.028921+0000 2024-09-16T05:24:38.023 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:24:38.028921+0000 2024-09-16T05:24:38.023 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:38.023 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:38.024 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:24:38.024 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:24:38.024 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:24:38.024 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:24:38.024 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:24:38.024 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:38.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 1 Processor -- start 2024-09-16T05:24:38.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 1 -- start start 2024-09-16T05:24:38.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 10 monclient: init 2024-09-16T05:24:38.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:38.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-16T05:24:38.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f76640a0ec0 addr v1:172.21.15.149:6789/0 2024-09-16T05:24:38.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f766409e1d0 0x7f766409e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:38.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 10 monclient(hunting): picked mon.noname-c con 0x7f766409e1d0 addr v2:172.21.15.149:3300/0 2024-09-16T05:24:38.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7664139640 0x7f766413ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:38.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 10 monclient(hunting): picked mon.noname-b con 0x7f7664139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:38.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f766409a760 con 0x7f76640a0ec0 2024-09-16T05:24:38.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:38.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f76640a12a0 con 0x7f766409e1d0 2024-09-16T05:24:38.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:38.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f76640994b0 con 0x7f7664139640 2024-09-16T05:24:38.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:38.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f766b4ea640 10 monclient(hunting): authenticate will time out at 1962.658081s 2024-09-16T05:24:38.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7664139640 0x7f766413ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:38.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7664139640 0x7f766413ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56230/0 (socket says 172.21.15.114:56230) 2024-09-16T05:24:38.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 1 -- 172.21.15.114:0/1248625986 learned_addr learned my addr 172.21.15.114:0/1248625986 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:38.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 10 monclient(hunting): get_auth_request con 0x7f7664139640 auth_method 0 2024-09-16T05:24:38.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:38.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:38.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:38.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7668a85640 1 --2- 172.21.15.114:0/1248625986 >> v2:172.21.15.149:3300/0 conn(0x7f766409e1d0 0x7f766409e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:38.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:38.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:38.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:38.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7668a85640 10 monclient(hunting): get_auth_request con 0x7f766409e1d0 auth_method 0 2024-09-16T05:24:38.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7668a85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:38.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7668a85640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:38.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7668a85640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:38.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7668a85640 1 -- 172.21.15.114:0/1248625986 >> v2:172.21.15.149:3300/0 conn(0x7f766409e1d0 msgr2=0x7f766409e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:24:38.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7668a85640 1 -- 172.21.15.114:0/1248625986 >> v2:172.21.15.149:3300/0 conn(0x7f766409e1d0 msgr2=0x7f766409e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:24:38.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7668a85640 1 --2- 172.21.15.114:0/1248625986 >> v2:172.21.15.149:3300/0 conn(0x7f766409e1d0 0x7f766409e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:24:38.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7668a85640 1 --2- 172.21.15.114:0/1248625986 >> v2:172.21.15.149:3300/0 conn(0x7f766409e1d0 0x7f766409e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:24:38.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 10 monclient(hunting): handle_auth_done global_id 104243 payload 274 2024-09-16T05:24:38.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 1 -- 172.21.15.114:0/1248625986 >> v2:172.21.15.149:3300/0 conn(0x7f766409e1d0 msgr2=0x7f766409e5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:24:38.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 1 --2- 172.21.15.114:0/1248625986 >> v2:172.21.15.149:3300/0 conn(0x7f766409e1d0 0x7f766409e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:38.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 1 -- 172.21.15.114:0/1248625986 >> v1:172.21.15.149:6789/0 conn(0x7f76640a0ec0 legacy=0x7f766409dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:38.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 10 monclient: _finish_hunting 0 2024-09-16T05:24:38.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 1 monclient: found mon.noname-b 2024-09-16T05:24:38.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:38.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:24:38.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.026+0000 7f7669a87640 1 -- 172.21.15.114:0/1248625986 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76640a1870 con 0x7f7664139640 2024-09-16T05:24:38.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7669a87640 1 --2- 172.21.15.114:0/1248625986 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7664139640 0x7f766413ba30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f765c00bb10 tx=0x7f765c01eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=335e07be7725708e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:38.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7657fff640 1 -- 172.21.15.114:0/1248625986 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765c0b4070 con 0x7f7664139640 2024-09-16T05:24:38.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7657fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:38.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7657fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:24:38.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7657fff640 10 monclient: dump: 2024-09-16T05:24:38.034 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:38.034 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:38.034 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:38.034 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:38.035 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:38.035 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:38.035 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:38.035 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:38.035 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:38.035 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:38.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7657fff640 10 monclient: _finish_auth 0 2024-09-16T05:24:38.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7657fff640 10 monclient: _check_auth_tickets 2024-09-16T05:24:38.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7657fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:38.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7657fff640 1 -- 172.21.15.114:0/1248625986 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f765c0a7a60 con 0x7f7664139640 2024-09-16T05:24:38.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7657fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:38.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7657fff640 1 -- 172.21.15.114:0/1248625986 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765c0a7e30 con 0x7f7664139640 2024-09-16T05:24:38.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 5 monclient: authenticate success, global_id 104243 2024-09-16T05:24:38.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 10 monclient: get_monmap_and_config success 2024-09-16T05:24:38.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 4 set_mon_vals no callback set 2024-09-16T05:24:38.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7657fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:38.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7657fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:38.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7657fff640 10 monclient: dump: 2024-09-16T05:24:38.036 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:38.036 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:38.036 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:38.036 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:38.036 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:38.036 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:38.036 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:38.036 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:38.037 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 10 monclient: shutdown 2024-09-16T05:24:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 -- 172.21.15.114:0/1248625986 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7664139640 msgr2=0x7f766413ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 --2- 172.21.15.114:0/1248625986 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7664139640 0x7f766413ba30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f765c00bb10 tx=0x7f765c01eb80 comp rx=0 tx=0).stop 2024-09-16T05:24:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 -- 172.21.15.114:0/1248625986 shutdown_connections 2024-09-16T05:24:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 --2- 172.21.15.114:0/1248625986 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7664139640 0x7f766413ba30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 --2- 172.21.15.114:0/1248625986 >> v2:172.21.15.149:3300/0 conn(0x7f766409e1d0 0x7f766409e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 -- 172.21.15.114:0/1248625986 >> 172.21.15.114:0/1248625986 conn(0x7f76640a1fb0 msgr2=0x7f76640a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 -- 172.21.15.114:0/1248625986 shutdown_connections 2024-09-16T05:24:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 -- 172.21.15.114:0/1248625986 wait complete. 2024-09-16T05:24:38.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 10 monclient: build_initial_monmap 2024-09-16T05:24:38.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 10 monclient: monmap: 2024-09-16T05:24:38.038 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:38.038 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:24:38.038 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:24:38.038 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:24:38.038 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:38.038 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:38.038 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:24:38.038 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:24:38.038 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:24:38.038 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:38.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 librados: starting msgr at 2024-09-16T05:24:38.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 librados: starting objecter 2024-09-16T05:24:38.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 Processor -- start 2024-09-16T05:24:38.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 -- start start 2024-09-16T05:24:38.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 librados: setting wanted keys 2024-09-16T05:24:38.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 librados: calling monclient init 2024-09-16T05:24:38.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 10 monclient: init 2024-09-16T05:24:38.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:38.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:24:38.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f766409e1d0 0x7f7664138fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:38.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 10 monclient(hunting): picked mon.noname-b con 0x7f766409e1d0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:24:38.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76640a0ec0 0x7f7664092490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:38.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 10 monclient(hunting): picked mon.noname-a con 0x7f76640a0ec0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:38.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7664139640 0x7f76640929d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:38.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 10 monclient(hunting): picked mon.noname-c con 0x7f7664139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:38.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:38.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f766409d340 con 0x7f76640a0ec0 2024-09-16T05:24:38.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:38.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f766409d1c0 con 0x7f766409e1d0 2024-09-16T05:24:38.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:38.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f766409d4c0 con 0x7f7664139640 2024-09-16T05:24:38.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:38.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f766b4ea640 10 monclient(hunting): authenticate will time out at 1962.663541s 2024-09-16T05:24:38.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7668a85640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76640a0ec0 0x7f7664092490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7669a87640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7664139640 0x7f76640929d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7668a85640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76640a0ec0 0x7f7664092490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40116/0 (socket says 172.21.15.114:40116) 2024-09-16T05:24:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7668a85640 1 -- 172.21.15.114:0/2450198320 learned_addr learned my addr 172.21.15.114:0/2450198320 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7668a85640 10 monclient(hunting): get_auth_request con 0x7f76640a0ec0 auth_method 0 2024-09-16T05:24:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7668a85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7668a85640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7668a85640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7669a87640 10 monclient(hunting): get_auth_request con 0x7f7664139640 auth_method 0 2024-09-16T05:24:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7669a87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7669a87640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7669a87640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7668a85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7668a85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7669286640 1 --2- 172.21.15.114:0/2450198320 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f766409e1d0 0x7f7664138fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7668a85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7669a87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7669a87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7669a87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7669286640 10 monclient(hunting): get_auth_request con 0x7f766409e1d0 auth_method 0 2024-09-16T05:24:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7669286640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7669286640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7669286640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.030+0000 7f7669286640 1 -- 172.21.15.114:0/2450198320 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f766409e1d0 msgr2=0x7f7664138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:24:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7669286640 1 -- 172.21.15.114:0/2450198320 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f766409e1d0 msgr2=0x7f7664138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:24:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7669286640 1 --2- 172.21.15.114:0/2450198320 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f766409e1d0 0x7f7664138fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:24:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7669286640 1 --2- 172.21.15.114:0/2450198320 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f766409e1d0 0x7f7664138fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:24:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7668a85640 10 monclient(hunting): handle_auth_done global_id 74763 payload 824 2024-09-16T05:24:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7668a85640 1 -- 172.21.15.114:0/2450198320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7664139640 msgr2=0x7f76640929d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7668a85640 1 --2- 172.21.15.114:0/2450198320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7664139640 0x7f76640929d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7668a85640 1 -- 172.21.15.114:0/2450198320 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f766409e1d0 msgr2=0x7f7664138fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:24:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7668a85640 1 --2- 172.21.15.114:0/2450198320 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f766409e1d0 0x7f7664138fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7668a85640 10 monclient: _finish_hunting 0 2024-09-16T05:24:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7668a85640 1 monclient: found mon.noname-a 2024-09-16T05:24:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7668a85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7668a85640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:24:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7668a85640 1 -- 172.21.15.114:0/2450198320 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7664094ce0 con 0x7f76640a0ec0 2024-09-16T05:24:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7669a87640 1 --2- 172.21.15.114:0/2450198320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7664139640 0x7f76640929d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:24:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7668a85640 1 --2- 172.21.15.114:0/2450198320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76640a0ec0 0x7f7664092490 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7658009f80 tx=0x7f765800e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:38.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 10 client.?.objecter ms_handle_connect 0x7f76640a0ec0 2024-09-16T05:24:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:24:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 1 -- 172.21.15.114:0/2450198320 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7658014420 con 0x7f76640a0ec0 2024-09-16T05:24:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:24:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 10 monclient: dump: 2024-09-16T05:24:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:38.044 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:38.045 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 10 monclient: _finish_auth 0 2024-09-16T05:24:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:24:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 1 -- 172.21.15.114:0/2450198320 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76580145e0 con 0x7f76640a0ec0 2024-09-16T05:24:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f766b4ea640 5 monclient: authenticate success, global_id 74763 2024-09-16T05:24:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f766b4ea640 10 monclient: _renew_subs 2024-09-16T05:24:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f766b4ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:24:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f766b4ea640 1 -- 172.21.15.114:0/2450198320 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7664096020 con 0x7f76640a0ec0 2024-09-16T05:24:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 1 -- 172.21.15.114:0/2450198320 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76580149d0 con 0x7f76640a0ec0 2024-09-16T05:24:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:38.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:24:38.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 10 monclient: dump: 2024-09-16T05:24:38.046 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:38.046 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:38.046 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:38.046 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:38.046 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:38.046 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:38.046 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:38.046 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:38.046 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:38.046 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:38.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f76577fe640 4 set_mon_vals no callback set 2024-09-16T05:24:38.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f766b4ea640 10 client.74763.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:24:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f766b4ea640 10 monclient: _renew_subs 2024-09-16T05:24:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f766b4ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:24:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f766b4ea640 1 -- 172.21.15.114:0/2450198320 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76640a4450 con 0x7f76640a0ec0 2024-09-16T05:24:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f766b4ea640 1 librados: init done 2024-09-16T05:24:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f766b4ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f766b4ea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f766b4ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:24:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f766b4ea640 1 -- 172.21.15.114:0/2450198320 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f766409c6a0 con 0x7f76640a0ec0 2024-09-16T05:24:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 1 -- 172.21.15.114:0/2450198320 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f765801b030 con 0x7f76640a0ec0 2024-09-16T05:24:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 1 --2- 172.21.15.114:0/2450198320 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7648037810 0x7f7648039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 1 -- 172.21.15.114:0/2450198320 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(682..682 src has 1..682) ==== 5965+0+0 (secure 0 0 0) 0x7f76580491c0 con 0x7f76640a0ec0 2024-09-16T05:24:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 10 client.74763.objecter ms_dispatch 0x7f7664000ba0 osd_map(682..682 src has 1..682) 2024-09-16T05:24:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 3 client.74763.objecter handle_osd_map got epochs [682,682] > 0 2024-09-16T05:24:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 3 client.74763.objecter handle_osd_map decoding full epoch 682 2024-09-16T05:24:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 20 client.74763.objecter dump_active .. 0 homeless 2024-09-16T05:24:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7669286640 1 --2- 172.21.15.114:0/2450198320 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7648037810 0x7f7648039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7669286640 10 monclient: get_auth_request con 0x7f7648037810 auth_method 0 2024-09-16T05:24:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7655ffb640 10 client.74763.objecter ms_handle_connect 0x7f7648037810 2024-09-16T05:24:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.034+0000 7f7669286640 1 --2- 172.21.15.114:0/2450198320 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7648037810 0x7f7648039cd0 secure :-1 s=READY pgs=3597 cs=0 l=1 rev1=1 crypto rx=0x7f7660007fd0 tx=0x7f7660009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.038+0000 7f7655ffb640 1 -- 172.21.15.114:0/2450198320 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7658022060 con 0x7f76640a0ec0 2024-09-16T05:24:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.038+0000 7f7655ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.038+0000 7f7655ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:24:38.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.270+0000 7f766b4ea640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:24:38.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.270+0000 7f766b4ea640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:24:38.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.270+0000 7f766b4ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:24:38.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.270+0000 7f766b4ea640 1 -- 172.21.15.114:0/2450198320 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f766409e5e0 con 0x7f76640a0ec0 2024-09-16T05:24:38.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.270+0000 7f7655ffb640 1 -- 172.21.15.114:0/2450198320 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f7664094ce0 con 0x7f76640a0ec0 2024-09-16T05:24:38.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.270+0000 7f7655ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:24:38.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.270+0000 7f7655ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:24:38.268 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":66,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":92,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:24:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 10 librados: watch_flush enter 2024-09-16T05:24:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 10 librados: watch_flush exit 2024-09-16T05:24:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 20 client.74763.objecter shutdown clearing up homeless session... 2024-09-16T05:24:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 10 client.74763.objecter successfully canceled tick 2024-09-16T05:24:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 1 -- 172.21.15.114:0/2450198320 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7648037810 msgr2=0x7f7648039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 1 --2- 172.21.15.114:0/2450198320 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7648037810 0x7f7648039cd0 secure :-1 s=READY pgs=3597 cs=0 l=1 rev1=1 crypto rx=0x7f7660007fd0 tx=0x7f7660009040 comp rx=0 tx=0).stop 2024-09-16T05:24:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 10 monclient: shutdown 2024-09-16T05:24:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 1 -- 172.21.15.114:0/2450198320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76640a0ec0 msgr2=0x7f7664092490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:38.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 1 --2- 172.21.15.114:0/2450198320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76640a0ec0 0x7f7664092490 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7658009f80 tx=0x7f765800e570 comp rx=0 tx=0).stop 2024-09-16T05:24:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 1 -- 172.21.15.114:0/2450198320 shutdown_connections 2024-09-16T05:24:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 1 --2- 172.21.15.114:0/2450198320 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7648037810 0x7f7648039cd0 unknown :-1 s=CLOSED pgs=3597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 1 --2- 172.21.15.114:0/2450198320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76640a0ec0 0x7f7664092490 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 1 --2- 172.21.15.114:0/2450198320 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7664139640 0x7f76640929d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 1 --2- 172.21.15.114:0/2450198320 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f766409e1d0 0x7f7664138fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 1 -- 172.21.15.114:0/2450198320 >> 172.21.15.114:0/2450198320 conn(0x7f76640a1fb0 msgr2=0x7f766413a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 1 -- 172.21.15.114:0/2450198320 shutdown_connections 2024-09-16T05:24:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 1 -- 172.21.15.114:0/2450198320 wait complete. 2024-09-16T05:24:38.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:38.274+0000 7f766b4ea640 1 librados: shutdown 2024-09-16T05:24:41.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:24:41.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 10 monclient: get_monmap_and_config 2024-09-16T05:24:41.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 10 monclient: build_initial_monmap 2024-09-16T05:24:41.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 10 monclient: monmap: 2024-09-16T05:24:41.417 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:41.417 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.417 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:24:41.422649+0000 2024-09-16T05:24:41.417 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:24:41.422649+0000 2024-09-16T05:24:41.417 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:41.417 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.417 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:24:41.417 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:24:41.417 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:24:41.418 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:24:41.418 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:24:41.418 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 1 Processor -- start 2024-09-16T05:24:41.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 1 -- start start 2024-09-16T05:24:41.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 10 monclient: init 2024-09-16T05:24:41.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:41.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-16T05:24:41.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f941c071da0 addr v1:172.21.15.149:6789/0 2024-09-16T05:24:41.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f941c0a1ab0 0x7f941c072890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 10 monclient(hunting): picked mon.noname-b con 0x7f941c0a1ab0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:41.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f941c099cc0 0x7f941c13e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 10 monclient(hunting): picked mon.noname-c con 0x7f941c099cc0 addr v2:172.21.15.149:3300/0 2024-09-16T05:24:41.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f941c0a1870 con 0x7f941c071da0 2024-09-16T05:24:41.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f941c073e20 con 0x7f941c099cc0 2024-09-16T05:24:41.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f941c0a1120 con 0x7f941c0a1ab0 2024-09-16T05:24:41.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:41.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f9423fe4640 10 monclient(hunting): authenticate will time out at 1966.051791s 2024-09-16T05:24:41.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.418+0000 7f942157f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f941c0a1ab0 0x7f941c072890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f941c0a1ab0 0x7f941c072890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56252/0 (socket says 172.21.15.114:56252) 2024-09-16T05:24:41.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 1 -- 172.21.15.114:0/447440333 learned_addr learned my addr 172.21.15.114:0/447440333 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:41.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 10 monclient(hunting): get_auth_request con 0x7f941c0a1ab0 auth_method 0 2024-09-16T05:24:41.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:41.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:41.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:41.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 10 monclient(hunting): handle_auth_done global_id 104249 payload 274 2024-09-16T05:24:41.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 1 -- 172.21.15.114:0/447440333 >> v2:172.21.15.149:3300/0 conn(0x7f941c099cc0 msgr2=0x7f941c13e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:41.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9422581640 1 --2- 172.21.15.114:0/447440333 >> v2:172.21.15.149:3300/0 conn(0x7f941c099cc0 0x7f941c13e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 1 --2- 172.21.15.114:0/447440333 >> v2:172.21.15.149:3300/0 conn(0x7f941c099cc0 0x7f941c13e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 1 -- 172.21.15.114:0/447440333 >> v1:172.21.15.149:6789/0 conn(0x7f941c071da0 legacy=0x7f941c072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:41.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 10 monclient: _finish_hunting 0 2024-09-16T05:24:41.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 1 monclient: found mon.noname-b 2024-09-16T05:24:41.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:41.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:24:41.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 1 -- 172.21.15.114:0/447440333 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941c073420 con 0x7f941c0a1ab0 2024-09-16T05:24:41.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f942157f640 1 --2- 172.21.15.114:0/447440333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f941c0a1ab0 0x7f941c072890 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9418006fd0 tx=0x7f941801c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=893cfecb620621a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:41.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9420d7e640 1 -- 172.21.15.114:0/447440333 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94180b0070 con 0x7f941c0a1ab0 2024-09-16T05:24:41.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9420d7e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9420d7e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:24:41.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9420d7e640 10 monclient: dump: 2024-09-16T05:24:41.428 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.429 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9420d7e640 10 monclient: _finish_auth 0 2024-09-16T05:24:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9420d7e640 10 monclient: _check_auth_tickets 2024-09-16T05:24:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9420d7e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9420d7e640 1 -- 172.21.15.114:0/447440333 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94180ab600 con 0x7f941c0a1ab0 2024-09-16T05:24:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9423fe4640 5 monclient: authenticate success, global_id 104249 2024-09-16T05:24:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9423fe4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:24:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9420d7e640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9423fe4640 10 monclient: get_monmap_and_config success 2024-09-16T05:24:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9420d7e640 1 -- 172.21.15.114:0/447440333 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94180ab9d0 con 0x7f941c0a1ab0 2024-09-16T05:24:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9423fe4640 4 set_mon_vals no callback set 2024-09-16T05:24:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9420d7e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9420d7e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9420d7e640 10 monclient: dump: 2024-09-16T05:24:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.430 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.431 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9423fe4640 10 monclient: shutdown 2024-09-16T05:24:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9423fe4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9423fe4640 1 -- 172.21.15.114:0/447440333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f941c0a1ab0 msgr2=0x7f941c072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9423fe4640 1 --2- 172.21.15.114:0/447440333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f941c0a1ab0 0x7f941c072890 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9418006fd0 tx=0x7f941801c3c0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9423fe4640 1 -- 172.21.15.114:0/447440333 shutdown_connections 2024-09-16T05:24:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9423fe4640 1 --2- 172.21.15.114:0/447440333 >> v2:172.21.15.149:3300/0 conn(0x7f941c099cc0 0x7f941c13e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9423fe4640 1 --2- 172.21.15.114:0/447440333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f941c0a1ab0 0x7f941c072890 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9423fe4640 1 -- 172.21.15.114:0/447440333 >> 172.21.15.114:0/447440333 conn(0x7f941c0a1fb0 msgr2=0x7f941c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9423fe4640 1 -- 172.21.15.114:0/447440333 shutdown_connections 2024-09-16T05:24:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.422+0000 7f9423fe4640 1 -- 172.21.15.114:0/447440333 wait complete. 2024-09-16T05:24:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient: build_initial_monmap 2024-09-16T05:24:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient: monmap: 2024-09-16T05:24:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:24:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:24:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:24:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:24:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:24:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:24:41.433 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 librados: starting msgr at 2024-09-16T05:24:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 librados: starting objecter 2024-09-16T05:24:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 Processor -- start 2024-09-16T05:24:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 -- start start 2024-09-16T05:24:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 librados: setting wanted keys 2024-09-16T05:24:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 librados: calling monclient init 2024-09-16T05:24:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient: init 2024-09-16T05:24:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:24:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f941c071da0 0x7f941c08d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient(hunting): picked mon.noname-b con 0x7f941c071da0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:24:41.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f941c099cc0 0x7f941c08d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient(hunting): picked mon.noname-c con 0x7f941c099cc0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f941c0a1ab0 0x7f941c08df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient(hunting): picked mon.noname-a con 0x7f941c0a1ab0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f941c141800 con 0x7f941c0a1ab0 2024-09-16T05:24:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f941c0a12a0 con 0x7f941c071da0 2024-09-16T05:24:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f941c141980 con 0x7f941c099cc0 2024-09-16T05:24:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient(hunting): authenticate will time out at 1966.057054s 2024-09-16T05:24:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f941c099cc0 0x7f941c08d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f941c099cc0 0x7f941c08d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56264/0 (socket says 172.21.15.114:56264) 2024-09-16T05:24:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 1 -- 172.21.15.114:0/909383606 learned_addr learned my addr 172.21.15.114:0/909383606 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 10 monclient(hunting): get_auth_request con 0x7f941c099cc0 auth_method 0 2024-09-16T05:24:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9422581640 1 --2- 172.21.15.114:0/909383606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f941c0a1ab0 0x7f941c08df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9422581640 10 monclient(hunting): get_auth_request con 0x7f941c0a1ab0 auth_method 0 2024-09-16T05:24:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9422581640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9422581640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9422581640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:41.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9422581640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9422581640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9422581640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9421d80640 1 --2- 172.21.15.114:0/909383606 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f941c071da0 0x7f941c08d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9421d80640 10 monclient(hunting): get_auth_request con 0x7f941c071da0 auth_method 0 2024-09-16T05:24:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9421d80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9421d80640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9421d80640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 10 monclient(hunting): handle_auth_done global_id 104252 payload 824 2024-09-16T05:24:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 1 -- 172.21.15.114:0/909383606 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f941c071da0 msgr2=0x7f941c08d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 1 --2- 172.21.15.114:0/909383606 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f941c071da0 0x7f941c08d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 1 -- 172.21.15.114:0/909383606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f941c0a1ab0 msgr2=0x7f941c08df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 1 --2- 172.21.15.114:0/909383606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f941c0a1ab0 0x7f941c08df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 10 monclient: _finish_hunting 0 2024-09-16T05:24:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 1 monclient: found mon.noname-c 2024-09-16T05:24:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:24:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 1 -- 172.21.15.114:0/909383606 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941c09c6a0 con 0x7f941c099cc0 2024-09-16T05:24:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9422581640 1 --2- 172.21.15.114:0/909383606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f941c0a1ab0 0x7f941c08df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:24:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f942157f640 1 --2- 172.21.15.114:0/909383606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f941c099cc0 0x7f941c08d9d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f941801cc60 tx=0x7f9418019160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 10 client.?.objecter ms_handle_connect 0x7f941c099cc0 2024-09-16T05:24:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:24:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 1 -- 172.21.15.114:0/909383606 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94180b0040 con 0x7f941c099cc0 2024-09-16T05:24:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:24:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 10 monclient: dump: 2024-09-16T05:24:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.437 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.438 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.438 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.438 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.438 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.438 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.438 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.438 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.438 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 10 monclient: _finish_auth 0 2024-09-16T05:24:41.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 10 monclient: _check_auth_tickets 2024-09-16T05:24:41.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:41.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 1 -- 172.21.15.114:0/909383606 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94180059a0 con 0x7f941c099cc0 2024-09-16T05:24:41.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 5 monclient: authenticate success, global_id 104252 2024-09-16T05:24:41.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient: _renew_subs 2024-09-16T05:24:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 -- 172.21.15.114:0/909383606 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f941c0a41b0 con 0x7f941c099cc0 2024-09-16T05:24:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 1 -- 172.21.15.114:0/909383606 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418005d90 con 0x7f941c099cc0 2024-09-16T05:24:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 10 monclient: dump: 2024-09-16T05:24:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.439 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.440 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.440 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.440 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9413fff640 4 set_mon_vals no callback set 2024-09-16T05:24:41.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 client.104252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:24:41.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient: _renew_subs 2024-09-16T05:24:41.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:41.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 -- 172.21.15.114:0/909383606 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f941c08f0f0 con 0x7f941c099cc0 2024-09-16T05:24:41.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 librados: init done 2024-09-16T05:24:41.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 1 -- 172.21.15.114:0/909383606 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9418003620 con 0x7f941c099cc0 2024-09-16T05:24:41.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 1 --2- 172.21.15.114:0/909383606 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9400037840 0x7f9400039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:41.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:41.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:41.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 1 -- 172.21.15.114:0/909383606 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(686..686 src has 1..686) ==== 5592+0+0 (secure 0 0 0) 0x7f94180e27b0 con 0x7f941c099cc0 2024-09-16T05:24:41.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 10 client.104252.objecter ms_dispatch 0x7f941c000ba0 osd_map(686..686 src has 1..686) 2024-09-16T05:24:41.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 3 client.104252.objecter handle_osd_map got epochs [686,686] > 0 2024-09-16T05:24:41.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 3 client.104252.objecter handle_osd_map decoding full epoch 686 2024-09-16T05:24:41.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9423fe4640 1 -- 172.21.15.114:0/909383606 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f941c072180 con 0x7f941c099cc0 2024-09-16T05:24:41.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f94127fc640 20 client.104252.objecter dump_active .. 0 homeless 2024-09-16T05:24:41.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9421d80640 1 --2- 172.21.15.114:0/909383606 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9400037840 0x7f9400039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.426+0000 7f9421d80640 10 monclient: get_auth_request con 0x7f9400037840 auth_method 0 2024-09-16T05:24:41.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.430+0000 7f94127fc640 10 client.104252.objecter ms_handle_connect 0x7f9400037840 2024-09-16T05:24:41.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.430+0000 7f9421d80640 1 --2- 172.21.15.114:0/909383606 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9400037840 0x7f9400039d00 secure :-1 s=READY pgs=3598 cs=0 l=1 rev1=1 crypto rx=0x7f940c004e30 tx=0x7f940c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:41.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.434+0000 7f94127fc640 1 -- 172.21.15.114:0/909383606 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f941c09c6a0 con 0x7f941c099cc0 2024-09-16T05:24:41.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.434+0000 7f94127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:41.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.434+0000 7f94127fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:24:41.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.658+0000 7f9423fe4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:24:41.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.658+0000 7f9423fe4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:24:41.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.658+0000 7f9423fe4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:41.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.658+0000 7f9423fe4640 1 -- 172.21.15.114:0/909383606 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f941c13b690 con 0x7f941c099cc0 2024-09-16T05:24:41.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.658+0000 7f94127fc640 1 -- 172.21.15.114:0/909383606 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f94180160d0 con 0x7f941c099cc0 2024-09-16T05:24:41.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.658+0000 7f94127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:24:41.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.658+0000 7f94127fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:24:41.658 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:24:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 10 librados: watch_flush enter 2024-09-16T05:24:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 10 librados: watch_flush exit 2024-09-16T05:24:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 20 client.104252.objecter shutdown clearing up homeless session... 2024-09-16T05:24:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 10 client.104252.objecter successfully canceled tick 2024-09-16T05:24:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 1 -- 172.21.15.114:0/909383606 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9400037840 msgr2=0x7f9400039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 1 --2- 172.21.15.114:0/909383606 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9400037840 0x7f9400039d00 secure :-1 s=READY pgs=3598 cs=0 l=1 rev1=1 crypto rx=0x7f940c004e30 tx=0x7f940c009040 comp rx=0 tx=0).stop 2024-09-16T05:24:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 10 monclient: shutdown 2024-09-16T05:24:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 1 -- 172.21.15.114:0/909383606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f941c099cc0 msgr2=0x7f941c08d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 1 --2- 172.21.15.114:0/909383606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f941c099cc0 0x7f941c08d9d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f941801cc60 tx=0x7f9418019160 comp rx=0 tx=0).stop 2024-09-16T05:24:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 1 -- 172.21.15.114:0/909383606 shutdown_connections 2024-09-16T05:24:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 1 --2- 172.21.15.114:0/909383606 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9400037840 0x7f9400039d00 unknown :-1 s=CLOSED pgs=3598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 1 --2- 172.21.15.114:0/909383606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f941c0a1ab0 0x7f941c08df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 1 --2- 172.21.15.114:0/909383606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f941c099cc0 0x7f941c08d9d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 1 --2- 172.21.15.114:0/909383606 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f941c071da0 0x7f941c08d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 1 -- 172.21.15.114:0/909383606 >> 172.21.15.114:0/909383606 conn(0x7f941c0a1fb0 msgr2=0x7f941c138860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 1 -- 172.21.15.114:0/909383606 shutdown_connections 2024-09-16T05:24:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 1 -- 172.21.15.114:0/909383606 wait complete. 2024-09-16T05:24:41.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.662+0000 7f9423fe4640 1 librados: shutdown 2024-09-16T05:24:41.675 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:24:41.675 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:24:41.675 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:24:41.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 10 monclient: get_monmap_and_config 2024-09-16T05:24:41.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 10 monclient: build_initial_monmap 2024-09-16T05:24:41.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 10 monclient: monmap: 2024-09-16T05:24:41.795 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:41.795 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.795 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:24:41.800507+0000 2024-09-16T05:24:41.795 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:24:41.800507+0000 2024-09-16T05:24:41.795 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:41.795 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.795 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:24:41.795 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:24:41.795 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:24:41.796 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:24:41.796 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:24:41.796 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 1 Processor -- start 2024-09-16T05:24:41.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 1 -- start start 2024-09-16T05:24:41.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 10 monclient: init 2024-09-16T05:24:41.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:41.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-16T05:24:41.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85541376e0 0x7f8554137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 10 monclient(hunting): picked mon.noname-b con 0x7f85541376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:41.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f85541370a0 0x7f85540a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 10 monclient(hunting): picked mon.noname-a con 0x7f85541370a0 addr v2:172.21.15.114:3300/0 2024-09-16T05:24:41.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85540a5d00 addr v1:172.21.15.149:6789/0 2024-09-16T05:24:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f855409bd20 con 0x7f85540a5d00 2024-09-16T05:24:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f8554138000 con 0x7f85541370a0 2024-09-16T05:24:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f855409c5c0 con 0x7f85541376e0 2024-09-16T05:24:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f855b22c640 10 monclient(hunting): authenticate will time out at 1966.429712s 2024-09-16T05:24:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85541376e0 0x7f8554137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85541376e0 0x7f8554137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56278/0 (socket says 172.21.15.114:56278) 2024-09-16T05:24:41.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 1 -- 172.21.15.114:0/2151911898 learned_addr learned my addr 172.21.15.114:0/2151911898 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:41.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 10 monclient(hunting): get_auth_request con 0x7f85541376e0 auth_method 0 2024-09-16T05:24:41.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8553fff640 1 --2- 172.21.15.114:0/2151911898 >> v2:172.21.15.114:3300/0 conn(0x7f85541370a0 0x7f85540a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8553fff640 10 monclient(hunting): get_auth_request con 0x7f85541370a0 auth_method 0 2024-09-16T05:24:41.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8553fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:41.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:41.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:41.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8553fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:41.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8553fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:41.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8553fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:41.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 10 monclient(hunting): handle_auth_done global_id 104255 payload 274 2024-09-16T05:24:41.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 1 -- 172.21.15.114:0/2151911898 >> v2:172.21.15.114:3300/0 conn(0x7f85541370a0 msgr2=0x7f85540a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:41.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 1 --2- 172.21.15.114:0/2151911898 >> v2:172.21.15.114:3300/0 conn(0x7f85541370a0 0x7f85540a4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 1 -- 172.21.15.114:0/2151911898 >> v1:172.21.15.149:6789/0 conn(0x7f85540a5d00 legacy=0x7f85540a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:41.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 10 monclient: _finish_hunting 0 2024-09-16T05:24:41.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 1 monclient: found mon.noname-b 2024-09-16T05:24:41.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:41.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:24:41.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8558fc8640 1 -- 172.21.15.114:0/2151911898 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855409c000 con 0x7f85541376e0 2024-09-16T05:24:41.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.798+0000 7f8553fff640 1 --2- 172.21.15.114:0/2151911898 >> v2:172.21.15.114:3300/0 conn(0x7f85541370a0 0x7f85540a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:24:41.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 1 --2- 172.21.15.114:0/2151911898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85541376e0 0x7f8554137ac0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f854c009e50 tx=0x7f854c01c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c2b69553b754ce4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:41.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85537fe640 1 -- 172.21.15.114:0/2151911898 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c0a5640 con 0x7f85541376e0 2024-09-16T05:24:41.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:24:41.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85537fe640 10 monclient: dump: 2024-09-16T05:24:41.811 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.811 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.812 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85537fe640 10 monclient: _finish_auth 0 2024-09-16T05:24:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85537fe640 10 monclient: _check_auth_tickets 2024-09-16T05:24:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 5 monclient: authenticate success, global_id 104255 2024-09-16T05:24:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:24:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85537fe640 1 -- 172.21.15.114:0/2151911898 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f854c0a57e0 con 0x7f85541376e0 2024-09-16T05:24:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85537fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient: get_monmap_and_config success 2024-09-16T05:24:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 4 set_mon_vals no callback set 2024-09-16T05:24:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85537fe640 1 -- 172.21.15.114:0/2151911898 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c0a5bb0 con 0x7f85541376e0 2024-09-16T05:24:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85537fe640 10 monclient: dump: 2024-09-16T05:24:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.814 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.814 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.814 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.814 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.814 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.814 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient: shutdown 2024-09-16T05:24:41.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:41.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 -- 172.21.15.114:0/2151911898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85541376e0 msgr2=0x7f8554137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 --2- 172.21.15.114:0/2151911898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85541376e0 0x7f8554137ac0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f854c009e50 tx=0x7f854c01c8b0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 -- 172.21.15.114:0/2151911898 shutdown_connections 2024-09-16T05:24:41.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 --2- 172.21.15.114:0/2151911898 >> v2:172.21.15.114:3300/0 conn(0x7f85541370a0 0x7f85540a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 --2- 172.21.15.114:0/2151911898 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85541376e0 0x7f8554137ac0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 -- 172.21.15.114:0/2151911898 >> 172.21.15.114:0/2151911898 conn(0x7f85540a9a40 msgr2=0x7f85540a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:41.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 -- 172.21.15.114:0/2151911898 shutdown_connections 2024-09-16T05:24:41.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 -- 172.21.15.114:0/2151911898 wait complete. 2024-09-16T05:24:41.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient: build_initial_monmap 2024-09-16T05:24:41.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient: monmap: 2024-09-16T05:24:41.815 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:41.815 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:24:41.815 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:24:41.815 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:24:41.815 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:41.815 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.815 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:24:41.815 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:24:41.816 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:24:41.816 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 librados: starting msgr at 2024-09-16T05:24:41.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 librados: starting objecter 2024-09-16T05:24:41.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 Processor -- start 2024-09-16T05:24:41.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 -- start start 2024-09-16T05:24:41.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 librados: setting wanted keys 2024-09-16T05:24:41.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 librados: calling monclient init 2024-09-16T05:24:41.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient: init 2024-09-16T05:24:41.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:41.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:24:41.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85540a5d00 0x7f85541cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient(hunting): picked mon.noname-c con 0x7f85540a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:41.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85541370a0 0x7f8554136070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient(hunting): picked mon.noname-a con 0x7f85541370a0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:41.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85541376e0 0x7f855413bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient(hunting): picked mon.noname-b con 0x7f85541376e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:24:41.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8554141d80 con 0x7f85541370a0 2024-09-16T05:24:41.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f8554141c00 con 0x7f85541376e0 2024-09-16T05:24:41.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8554141f00 con 0x7f85540a5d00 2024-09-16T05:24:41.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:41.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient(hunting): authenticate will time out at 1966.434714s 2024-09-16T05:24:41.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8553fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85541370a0 0x7f8554136070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85540a5d00 0x7f85541cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85540a5d00 0x7f85541cf0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56280/0 (socket says 172.21.15.114:56280) 2024-09-16T05:24:41.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8553fff640 10 monclient(hunting): get_auth_request con 0x7f85541370a0 auth_method 0 2024-09-16T05:24:41.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8553fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 1 -- 172.21.15.114:0/3772698234 learned_addr learned my addr 172.21.15.114:0/3772698234 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:41.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 10 monclient(hunting): get_auth_request con 0x7f85540a5d00 auth_method 0 2024-09-16T05:24:41.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:41.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:41.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:41.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8553fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:41.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8553fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:41.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8553fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:41.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 10 monclient(hunting): handle_auth_done global_id 104258 payload 824 2024-09-16T05:24:41.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 1 -- 172.21.15.114:0/3772698234 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85541376e0 msgr2=0x7f855413bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85597c9640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85541376e0 0x7f855413bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85541376e0 0x7f855413bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 1 -- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85541370a0 msgr2=0x7f8554136070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85541370a0 0x7f8554136070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 10 monclient: _finish_hunting 0 2024-09-16T05:24:41.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 1 monclient: found mon.noname-c 2024-09-16T05:24:41.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:41.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:24:41.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 1 -- 172.21.15.114:0/3772698234 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855413e9d0 con 0x7f85540a5d00 2024-09-16T05:24:41.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8553fff640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85541370a0 0x7f8554136070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:24:41.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8558fc8640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85540a5d00 0x7f85541cf0e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f854c01af60 tx=0x7f854c019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:41.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85517fa640 10 client.?.objecter ms_handle_connect 0x7f85540a5d00 2024-09-16T05:24:41.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85517fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:24:41.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85517fa640 1 -- 172.21.15.114:0/3772698234 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c0bc020 con 0x7f85540a5d00 2024-09-16T05:24:41.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85517fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:24:41.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85517fa640 10 monclient: dump: 2024-09-16T05:24:41.820 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.820 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.821 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.821 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.821 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.821 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.821 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.821 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.821 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.821 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85517fa640 10 monclient: _finish_auth 0 2024-09-16T05:24:41.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85517fa640 10 monclient: _check_auth_tickets 2024-09-16T05:24:41.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:41.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85517fa640 1 -- 172.21.15.114:0/3772698234 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f854c0b8690 con 0x7f85540a5d00 2024-09-16T05:24:41.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 5 monclient: authenticate success, global_id 104258 2024-09-16T05:24:41.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient: _renew_subs 2024-09-16T05:24:41.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:41.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 -- 172.21.15.114:0/3772698234 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f855413e510 con 0x7f85540a5d00 2024-09-16T05:24:41.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85517fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:41.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85517fa640 1 -- 172.21.15.114:0/3772698234 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c0b8a80 con 0x7f85540a5d00 2024-09-16T05:24:41.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85517fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:41.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f85517fa640 10 monclient: dump: 2024-09-16T05:24:41.822 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.822 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.823 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.823 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.823 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.823 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.823 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.823 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.823 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.823 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f8552ffd640 4 set_mon_vals no callback set 2024-09-16T05:24:41.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 client.104258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:24:41.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient: _renew_subs 2024-09-16T05:24:41.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:41.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 -- 172.21.15.114:0/3772698234 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8554120440 con 0x7f85540a5d00 2024-09-16T05:24:41.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.802+0000 7f855b22c640 1 librados: init done 2024-09-16T05:24:41.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f855b22c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:41.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f855b22c640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8554120610 0x7f855413c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f855b22c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f855b22c640 1 -- 172.21.15.114:0/3772698234 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f855413dc30 con 0x7f8554120610 2024-09-16T05:24:41.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f8553fff640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8554120610 0x7f855413c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f8553fff640 10 monclient: get_auth_request con 0x7f8554120610 auth_method 0 2024-09-16T05:24:41.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f8553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f8553fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f8553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 1 -- 172.21.15.114:0/3772698234 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f854c0bf020 con 0x7f85540a5d00 2024-09-16T05:24:41.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f8553fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:41.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f8553fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:41.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f8553fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:41.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f8553fff640 10 monclient(hunting): handle_auth_done global_id 74775 payload 824 2024-09-16T05:24:41.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f8553fff640 1 -- 172.21.15.114:0/3772698234 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8554137ac0 con 0x7f8554120610 2024-09-16T05:24:41.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 1 --2- 172.21.15.114:0/3772698234 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8534037810 0x7f8534039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 1 -- 172.21.15.114:0/3772698234 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(686..686 src has 1..686) ==== 5592+0+0 (secure 0 0 0) 0x7f854c0e42a0 con 0x7f85540a5d00 2024-09-16T05:24:41.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 10 client.104258.objecter ms_dispatch 0x7f8554000ba0 osd_map(686..686 src has 1..686) 2024-09-16T05:24:41.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 3 client.104258.objecter handle_osd_map got epochs [686,686] > 0 2024-09-16T05:24:41.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 3 client.104258.objecter handle_osd_map decoding full epoch 686 2024-09-16T05:24:41.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f8553fff640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8554120610 0x7f855413c4d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f855413d710 tx=0x7f853c00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:41.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 20 client.104258.objecter dump_active .. 0 homeless 2024-09-16T05:24:41.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 10 client.104258.objecter ms_handle_connect 0x7f8554120610 2024-09-16T05:24:41.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 10 client.104258.objecter resend_mon_ops 2024-09-16T05:24:41.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 1 -- 172.21.15.114:0/3772698234 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c019070 con 0x7f8554120610 2024-09-16T05:24:41.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:24:41.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 10 monclient: dump: 2024-09-16T05:24:41.826 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.826 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.826 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.826 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.826 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.826 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.826 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.826 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.827 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.827 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85597c9640 1 --2- 172.21.15.114:0/3772698234 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8534037810 0x7f8534039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85597c9640 10 monclient: get_auth_request con 0x7f8534037810 auth_method 0 2024-09-16T05:24:41.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 1 -- 172.21.15.114:0/3772698234 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f853c004030 con 0x7f8554120610 2024-09-16T05:24:41.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:41.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:24:41.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 1 -- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8554120610 msgr2=0x7f855413c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8554120610 0x7f855413c4d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f855413d710 tx=0x7f853c00f610 comp rx=0 tx=0).stop 2024-09-16T05:24:41.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 1 -- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8554120610 msgr2=0x7f855413c4d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:24:41.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8554120610 0x7f855413c4d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85517fa640 10 client.104258.objecter ms_handle_connect 0x7f8534037810 2024-09-16T05:24:41.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.806+0000 7f85597c9640 1 --2- 172.21.15.114:0/3772698234 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8534037810 0x7f8534039cd0 secure :-1 s=READY pgs=3599 cs=0 l=1 rev1=1 crypto rx=0x7f8544003ce0 tx=0x7f8544009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:41.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.810+0000 7f855b22c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:24:41.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.810+0000 7f855b22c640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85541365b0 0x7f855413ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.810+0000 7f855b22c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.810+0000 7f855b22c640 1 -- 172.21.15.114:0/3772698234 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8554068820 con 0x7f85541365b0 2024-09-16T05:24:41.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.810+0000 7f8553fff640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85541365b0 0x7f855413ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.810+0000 7f8553fff640 10 monclient: get_auth_request con 0x7f85541365b0 auth_method 0 2024-09-16T05:24:41.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.810+0000 7f8553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.810+0000 7f8553fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.810+0000 7f8553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.810+0000 7f8553fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:41.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.810+0000 7f8553fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:41.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.810+0000 7f8553fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:41.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.810+0000 7f8553fff640 10 monclient(hunting): handle_auth_done global_id 74778 payload 824 2024-09-16T05:24:41.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.810+0000 7f8553fff640 1 -- 172.21.15.114:0/3772698234 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f85541da430 con 0x7f85541365b0 2024-09-16T05:24:41.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f85517fa640 10 client.104258.objecter ms_handle_connect 0x7f85541365b0 2024-09-16T05:24:41.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f85517fa640 10 client.104258.objecter resend_mon_ops 2024-09-16T05:24:41.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f8553fff640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85541365b0 0x7f855413ca10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8554073f00 tx=0x7f853c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:41.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f85517fa640 1 -- 172.21.15.114:0/3772698234 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c019040 con 0x7f85541365b0 2024-09-16T05:24:41.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f85517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f85517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:24:41.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f85517fa640 10 monclient: dump: 2024-09-16T05:24:41.829 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.829 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.829 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.830 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.830 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.830 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.830 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.830 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.830 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.830 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f85517fa640 1 -- 172.21.15.114:0/3772698234 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f853c01e450 con 0x7f85541365b0 2024-09-16T05:24:41.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f85517fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:24:41.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f85517fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:24:41.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f85517fa640 1 -- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85541365b0 msgr2=0x7f855413ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f85517fa640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85541365b0 0x7f855413ca10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8554073f00 tx=0x7f853c0046d0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f85517fa640 1 -- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85541365b0 msgr2=0x7f855413ca10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:24:41.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f85517fa640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85541365b0 0x7f855413ca10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.831 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:24:41.831 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:24:41.831 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:24:41.831 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:24:41.831 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 68, 2024-09-16T05:24:41.831 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 640870, 2024-09-16T05:24:41.831 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:24:41.832 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:24:41.832 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:24:41.832 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:24:41.832 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:41.832 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:24:41.832 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:24:41.832 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:24:41.832 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:24:41.832 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:41.832 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:41.832 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:41.832 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:41.832 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:41.833 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:41.833 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:41.833 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:41.833 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:41.833 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:41.833 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:41.833 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:41.833 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:24:41.833 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:24:41.833 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:41.833 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:41.833 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:41.833 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:41.834 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:41.834 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:41.834 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:41.834 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:41.834 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:41.834 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:41.834 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:41.834 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:41.834 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.834 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:24:41.834 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:24:41.834 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:24:41.834 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:24:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:24:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:24:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:24:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:24:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:24:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:24:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:24:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:24:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:24:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:24:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:24:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:24:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:24:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:41.836 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:41.836 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:41.836 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:41.836 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:41.836 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:41.836 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:41.836 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:41.836 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:41.836 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:41.836 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:41.836 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:41.836 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:24:41.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.837 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:24:41.837 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.837 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:24:41.837 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:24:41.837 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:41.837 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:41.837 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.837 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:41.837 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:24:41.838 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:41.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.838 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.838 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:41.838 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:24:41.838 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:41.838 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:41.838 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:41.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.838 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:24:41.838 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:24:41.838 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:41.838 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:41.839 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:41.839 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.839 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.839 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:24:41.839 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:24:41.839 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:41.839 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:41.839 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.839 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:41.839 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:24:41.839 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:41.839 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.839 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.839 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:41.840 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:24:41.840 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:41.840 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:41.840 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:41.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.840 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:24:41.840 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:24:41.840 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:41.840 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:41.840 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:41.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.840 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.840 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:24:41.841 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:24:41.841 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:41.841 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:41.841 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.841 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:41.841 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:24:41.841 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:41.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.841 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.841 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:41.841 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:24:41.841 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:41.841 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:41.841 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:41.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.842 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:24:41.842 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:24:41.842 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:41.842 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:41.842 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:41.842 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:41.842 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:41.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.842 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:24:41.842 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:24:41.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.842 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:41.843 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:41.843 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:24:41.843 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:41.843 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:41.843 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:24:41.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.843 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:41.843 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:41.843 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:24:41.843 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:41.843 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:41.843 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:24:41.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.843 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:41.844 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:41.844 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:24:41.844 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:41.844 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:41.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.844 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:24:41.844 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:24:41.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f855b22c640 10 librados: watch_flush enter 2024-09-16T05:24:41.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f855b22c640 10 librados: watch_flush exit 2024-09-16T05:24:41.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f855b22c640 20 client.104258.objecter shutdown clearing up homeless session... 2024-09-16T05:24:41.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f855b22c640 10 client.104258.objecter successfully canceled tick 2024-09-16T05:24:41.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f855b22c640 1 -- 172.21.15.114:0/3772698234 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8534037810 msgr2=0x7f8534039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f855b22c640 1 --2- 172.21.15.114:0/3772698234 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8534037810 0x7f8534039cd0 secure :-1 s=READY pgs=3599 cs=0 l=1 rev1=1 crypto rx=0x7f8544003ce0 tx=0x7f8544009040 comp rx=0 tx=0).stop 2024-09-16T05:24:41.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f855b22c640 10 monclient: shutdown 2024-09-16T05:24:41.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f855b22c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:41.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f855b22c640 1 -- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85540a5d00 msgr2=0x7f85541cf0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f855b22c640 1 --2- 172.21.15.114:0/3772698234 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85540a5d00 0x7f85541cf0e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f854c01af60 tx=0x7f854c019650 comp rx=0 tx=0).stop 2024-09-16T05:24:41.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f85597c9640 1 -- 172.21.15.114:0/3772698234 reap_dead start 2024-09-16T05:24:41.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f85597c9640 1 -- 172.21.15.114:0/3772698234 reap_dead start 2024-09-16T05:24:41.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f855b22c640 1 -- 172.21.15.114:0/3772698234 shutdown_connections 2024-09-16T05:24:41.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f855b22c640 1 -- 172.21.15.114:0/3772698234 >> 172.21.15.114:0/3772698234 conn(0x7f85540a9a40 msgr2=0x7f855409df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:41.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f855b22c640 1 -- 172.21.15.114:0/3772698234 shutdown_connections 2024-09-16T05:24:41.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f855b22c640 1 -- 172.21.15.114:0/3772698234 wait complete. 2024-09-16T05:24:41.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.814+0000 7f855b22c640 1 librados: shutdown 2024-09-16T05:24:41.848 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:24:41.848 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:24:41.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:24:41.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.954+0000 7f8279c6e640 10 monclient: get_monmap_and_config 2024-09-16T05:24:41.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.954+0000 7f8279c6e640 10 monclient: build_initial_monmap 2024-09-16T05:24:41.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.954+0000 7f8279c6e640 10 monclient: monmap: 2024-09-16T05:24:41.954 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:41.954 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.954 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:24:41.959950+0000 2024-09-16T05:24:41.954 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:24:41.959950+0000 2024-09-16T05:24:41.954 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:41.954 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.954 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:24:41.955 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:24:41.955 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:24:41.955 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:24:41.955 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:24:41.955 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 1 Processor -- start 2024-09-16T05:24:41.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 1 -- start start 2024-09-16T05:24:41.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 10 monclient: init 2024-09-16T05:24:41.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:41.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-16T05:24:41.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827409b6c0 0x7f82740a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 10 monclient(hunting): picked mon.noname-b con 0x7f827409b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:41.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82740a1400 addr v1:172.21.15.114:6789/0 2024-09-16T05:24:41.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f82740a17e0 0x7f827413cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 10 monclient(hunting): picked mon.noname-c con 0x7f82740a17e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:24:41.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f827409d4e0 con 0x7f82740a1400 2024-09-16T05:24:41.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f827409e390 con 0x7f82740a17e0 2024-09-16T05:24:41.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f827409ece0 con 0x7f827409b6c0 2024-09-16T05:24:41.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:41.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 10 monclient(hunting): authenticate will time out at 1966.589004s 2024-09-16T05:24:41.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827409b6c0 0x7f82740a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f826affd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f82740a1400 0x7f8274139640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37958/0 (socket says 172.21.15.114:37958) 2024-09-16T05:24:41.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 10 monclient(hunting): get_auth_request con 0x7f827409b6c0 auth_method 0 2024-09-16T05:24:41.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f826affd640 1 -- 172.21.15.114:0/1340525462 learned_addr learned my addr 172.21.15.114:0/1340525462 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:24:41.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:41.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:41.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:41.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8273fff640 1 --2- 172.21.15.114:0/1340525462 >> v2:172.21.15.149:3300/0 conn(0x7f82740a17e0 0x7f827413cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 10 monclient(hunting): handle_auth_done global_id 104261 payload 274 2024-09-16T05:24:41.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 1 -- 172.21.15.114:0/1340525462 >> v2:172.21.15.149:3300/0 conn(0x7f82740a17e0 msgr2=0x7f827413cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:41.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 1 --2- 172.21.15.114:0/1340525462 >> v2:172.21.15.149:3300/0 conn(0x7f82740a17e0 0x7f827413cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 1 -- 172.21.15.114:0/1340525462 >> v1:172.21.15.114:6789/0 conn(0x7f82740a1400 legacy=0x7f8274139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 10 monclient: _finish_hunting 0 2024-09-16T05:24:41.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 1 monclient: found mon.noname-b 2024-09-16T05:24:41.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:41.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:24:41.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 1 -- 172.21.15.114:0/1340525462 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827409f430 con 0x7f827409b6c0 2024-09-16T05:24:41.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8273fff640 10 monclient: get_auth_request con 0x7f82740a17e0 auth_method 0 2024-09-16T05:24:41.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8273fff640 1 --2- 172.21.15.114:0/1340525462 >> v2:172.21.15.149:3300/0 conn(0x7f82740a17e0 0x7f827413cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:24:41.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f82737fe640 1 --2- 172.21.15.114:0/1340525462 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827409b6c0 0x7f82740a0ec0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f82640097c0 tx=0x7f826401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12f3eb94cc0fad2a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:41.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8272ffd640 1 -- 172.21.15.114:0/1340525462 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82640b0070 con 0x7f827409b6c0 2024-09-16T05:24:41.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8272ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8272ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:24:41.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8272ffd640 10 monclient: dump: 2024-09-16T05:24:41.971 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.971 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.971 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.971 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.971 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.971 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.971 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.971 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.971 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.971 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8272ffd640 10 monclient: _finish_auth 0 2024-09-16T05:24:41.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8272ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:24:41.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8272ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:41.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8272ffd640 1 -- 172.21.15.114:0/1340525462 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82640ab600 con 0x7f827409b6c0 2024-09-16T05:24:41.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 5 monclient: authenticate success, global_id 104261 2024-09-16T05:24:41.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:24:41.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8272ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:41.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 10 monclient: get_monmap_and_config success 2024-09-16T05:24:41.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8272ffd640 1 -- 172.21.15.114:0/1340525462 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82640ab9d0 con 0x7f827409b6c0 2024-09-16T05:24:41.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8272ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8279c6e640 4 set_mon_vals no callback set 2024-09-16T05:24:41.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8272ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:41.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.958+0000 7f8272ffd640 10 monclient: dump: 2024-09-16T05:24:41.972 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.973 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.973 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.973 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.973 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.973 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.973 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.973 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.973 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.973 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient: shutdown 2024-09-16T05:24:41.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:41.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 -- 172.21.15.114:0/1340525462 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827409b6c0 msgr2=0x7f82740a0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 --2- 172.21.15.114:0/1340525462 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827409b6c0 0x7f82740a0ec0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f82640097c0 tx=0x7f826401c360 comp rx=0 tx=0).stop 2024-09-16T05:24:41.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 -- 172.21.15.114:0/1340525462 shutdown_connections 2024-09-16T05:24:41.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 --2- 172.21.15.114:0/1340525462 >> v2:172.21.15.149:3300/0 conn(0x7f82740a17e0 0x7f827413cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 --2- 172.21.15.114:0/1340525462 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827409b6c0 0x7f82740a0ec0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 -- 172.21.15.114:0/1340525462 >> 172.21.15.114:0/1340525462 conn(0x7f82740a1fb0 msgr2=0x7f82740a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:41.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 -- 172.21.15.114:0/1340525462 shutdown_connections 2024-09-16T05:24:41.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 -- 172.21.15.114:0/1340525462 wait complete. 2024-09-16T05:24:41.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient: build_initial_monmap 2024-09-16T05:24:41.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient: monmap: 2024-09-16T05:24:41.974 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:41.974 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:24:41.974 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:24:41.974 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:24:41.974 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:41.974 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.975 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:24:41.975 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:24:41.975 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:24:41.975 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 librados: starting msgr at 2024-09-16T05:24:41.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 librados: starting objecter 2024-09-16T05:24:41.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 Processor -- start 2024-09-16T05:24:41.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 -- start start 2024-09-16T05:24:41.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 librados: setting wanted keys 2024-09-16T05:24:41.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 librados: calling monclient init 2024-09-16T05:24:41.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient: init 2024-09-16T05:24:41.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:41.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:24:41.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f827409b6c0 0x7f82740984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient(hunting): picked mon.noname-b con 0x7f827409b6c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:24:41.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82740a1400 0x7f82740994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient(hunting): picked mon.noname-a con 0x7f82740a1400 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:41.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82740a17e0 0x7f8274099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient(hunting): picked mon.noname-c con 0x7f82740a17e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:41.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f82740a1bf0 con 0x7f82740a1400 2024-09-16T05:24:41.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f827409cad0 con 0x7f827409b6c0 2024-09-16T05:24:41.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f827413fad0 con 0x7f82740a17e0 2024-09-16T05:24:41.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:41.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient(hunting): authenticate will time out at 1966.593879s 2024-09-16T05:24:41.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82740a17e0 0x7f8274099a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f826affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82740a1400 0x7f82740994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82740a17e0 0x7f8274099a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56298/0 (socket says 172.21.15.114:56298) 2024-09-16T05:24:41.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 1 -- 172.21.15.114:0/4014829019 learned_addr learned my addr 172.21.15.114:0/4014829019 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:41.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 10 monclient(hunting): get_auth_request con 0x7f82740a17e0 auth_method 0 2024-09-16T05:24:41.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f826affd640 10 monclient(hunting): get_auth_request con 0x7f82740a1400 auth_method 0 2024-09-16T05:24:41.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f826affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f826affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f826affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:41.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:41.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:41.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f826affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:41.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f826affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:41.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f826affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:41.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f827409b6c0 0x7f82740984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 10 monclient(hunting): get_auth_request con 0x7f827409b6c0 auth_method 0 2024-09-16T05:24:41.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 10 monclient(hunting): handle_auth_done global_id 104264 payload 824 2024-09-16T05:24:41.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 1 -- 172.21.15.114:0/4014829019 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f827409b6c0 msgr2=0x7f82740984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f827409b6c0 0x7f82740984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 1 -- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82740a1400 msgr2=0x7f82740994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82740a1400 0x7f82740994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 10 monclient: _finish_hunting 0 2024-09-16T05:24:41.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 1 monclient: found mon.noname-c 2024-09-16T05:24:41.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:41.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:24:41.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 1 -- 172.21.15.114:0/4014829019 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8274097620 con 0x7f82740a17e0 2024-09-16T05:24:41.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f826affd640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82740a1400 0x7f82740994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:24:41.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8273fff640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82740a17e0 0x7f8274099a20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f826c00cef0 tx=0x7f826c00a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:41.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 10 client.?.objecter ms_handle_connect 0x7f82740a17e0 2024-09-16T05:24:41.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:24:41.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 1 -- 172.21.15.114:0/4014829019 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c01a070 con 0x7f82740a17e0 2024-09-16T05:24:41.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:24:41.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 10 monclient: dump: 2024-09-16T05:24:41.980 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.980 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.980 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.980 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.980 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.980 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.980 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.980 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.980 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.980 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 10 monclient: _finish_auth 0 2024-09-16T05:24:41.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:24:41.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:41.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 1 -- 172.21.15.114:0/4014829019 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f826c00d860 con 0x7f82740a17e0 2024-09-16T05:24:41.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:41.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 1 -- 172.21.15.114:0/4014829019 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c00dc50 con 0x7f82740a17e0 2024-09-16T05:24:41.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 5 monclient: authenticate success, global_id 104264 2024-09-16T05:24:41.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient: _renew_subs 2024-09-16T05:24:41.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:41.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 -- 172.21.15.114:0/4014829019 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8274097900 con 0x7f82740a17e0 2024-09-16T05:24:41.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82727fc640 4 set_mon_vals no callback set 2024-09-16T05:24:41.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:41.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 10 monclient: dump: 2024-09-16T05:24:41.981 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.981 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.981 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.981 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.982 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.982 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.982 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.982 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.982 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.982 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 client.104264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:24:41.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient: _renew_subs 2024-09-16T05:24:41.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:41.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 -- 172.21.15.114:0/4014829019 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8274097340 con 0x7f82740a17e0 2024-09-16T05:24:41.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 librados: init done 2024-09-16T05:24:41.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:41.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827402d7d0 0x7f8274099f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8279c6e640 1 -- 172.21.15.114:0/4014829019 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8274069390 con 0x7f827402d7d0 2024-09-16T05:24:41.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827402d7d0 0x7f8274099f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 10 monclient: get_auth_request con 0x7f827402d7d0 auth_method 0 2024-09-16T05:24:41.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 1 -- 172.21.15.114:0/4014829019 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f826c01d070 con 0x7f82740a17e0 2024-09-16T05:24:41.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:41.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:41.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:41.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 1 --2- 172.21.15.114:0/4014829019 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8254037810 0x7f8254039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 10 monclient(hunting): handle_auth_done global_id 104267 payload 824 2024-09-16T05:24:41.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 1 -- 172.21.15.114:0/4014829019 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f827402dbb0 con 0x7f827402d7d0 2024-09-16T05:24:41.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 1 -- 172.21.15.114:0/4014829019 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(686..686 src has 1..686) ==== 5592+0+0 (secure 0 0 0) 0x7f826c0498f0 con 0x7f82740a17e0 2024-09-16T05:24:41.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 10 client.104264.objecter ms_dispatch 0x7f8274000ba0 osd_map(686..686 src has 1..686) 2024-09-16T05:24:41.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 3 client.104264.objecter handle_osd_map got epochs [686,686] > 0 2024-09-16T05:24:41.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 3 client.104264.objecter handle_osd_map decoding full epoch 686 2024-09-16T05:24:41.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 20 client.104264.objecter dump_active .. 0 homeless 2024-09-16T05:24:41.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 10 client.104264.objecter ms_handle_connect 0x7f827402d7d0 2024-09-16T05:24:41.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f8270ff9640 10 client.104264.objecter resend_mon_ops 2024-09-16T05:24:41.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.962+0000 7f82737fe640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827402d7d0 0x7f8274099f60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f82740a3070 tx=0x7f826401aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:41.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.966+0000 7f8270ff9640 1 -- 172.21.15.114:0/4014829019 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82640b0040 con 0x7f827402d7d0 2024-09-16T05:24:41.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.966+0000 7f8270ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.966+0000 7f8270ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:41.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.966+0000 7f8270ff9640 10 monclient: dump: 2024-09-16T05:24:41.984 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.984 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.985 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.985 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.985 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.985 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.985 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.985 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.985 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.985 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.966+0000 7f826affd640 1 --2- 172.21.15.114:0/4014829019 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8254037810 0x7f8254039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.966+0000 7f826affd640 10 monclient: get_auth_request con 0x7f8254037810 auth_method 0 2024-09-16T05:24:41.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.966+0000 7f8270ff9640 1 -- 172.21.15.114:0/4014829019 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f82640abde0 con 0x7f827402d7d0 2024-09-16T05:24:41.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.966+0000 7f8270ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:41.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.966+0000 7f8270ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:24:41.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.966+0000 7f8270ff9640 1 -- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827402d7d0 msgr2=0x7f8274099f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.966+0000 7f8270ff9640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827402d7d0 0x7f8274099f60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f82740a3070 tx=0x7f826401aba0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.966+0000 7f8270ff9640 1 -- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827402d7d0 msgr2=0x7f8274099f60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:24:41.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.966+0000 7f8270ff9640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f827402d7d0 0x7f8274099f60 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f82740a3070 tx=0x7f826401aba0 comp rx=0 tx=0).stop 2024-09-16T05:24:41.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.966+0000 7f8270ff9640 10 client.104264.objecter ms_handle_connect 0x7f8254037810 2024-09-16T05:24:41.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.966+0000 7f826affd640 1 --2- 172.21.15.114:0/4014829019 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8254037810 0x7f8254039cd0 secure :-1 s=READY pgs=3600 cs=0 l=1 rev1=1 crypto rx=0x7f827409ad30 tx=0x7f825c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:41.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:24:41.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8274062ea0 0x7f82740632b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:41.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:41.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 1 -- 172.21.15.114:0/4014829019 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f827408e710 con 0x7f8274062ea0 2024-09-16T05:24:41.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f82737fe640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8274062ea0 0x7f82740632b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:41.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f82737fe640 10 monclient: get_auth_request con 0x7f8274062ea0 auth_method 0 2024-09-16T05:24:41.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f82737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:41.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f82737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:41.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f82737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:41.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f82737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:41.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f82737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:41.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f82737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:41.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f82737fe640 10 monclient(hunting): handle_auth_done global_id 104270 payload 824 2024-09-16T05:24:41.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f82737fe640 1 -- 172.21.15.114:0/4014829019 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f82740961d0 con 0x7f8274062ea0 2024-09-16T05:24:41.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8270ff9640 10 client.104264.objecter ms_handle_connect 0x7f8274062ea0 2024-09-16T05:24:41.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f82737fe640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8274062ea0 0x7f82740632b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8264000c10 tx=0x7f8264019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:41.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8270ff9640 10 client.104264.objecter resend_mon_ops 2024-09-16T05:24:41.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8270ff9640 1 -- 172.21.15.114:0/4014829019 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82640b0050 con 0x7f8274062ea0 2024-09-16T05:24:41.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8270ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:41.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8270ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:41.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8270ff9640 10 monclient: dump: 2024-09-16T05:24:41.988 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:41.988 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:41.988 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.988 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:41.988 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:41.988 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:41.988 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:41.988 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:41.988 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:41.988 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:41.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8270ff9640 1 -- 172.21.15.114:0/4014829019 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f8264003d60 con 0x7f8274062ea0 2024-09-16T05:24:41.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8270ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:24:41.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8270ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:24:41.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8270ff9640 1 -- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8274062ea0 msgr2=0x7f82740632b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:41.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8270ff9640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8274062ea0 0x7f82740632b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8264000c10 tx=0x7f8264019100 comp rx=0 tx=0).stop 2024-09-16T05:24:41.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8270ff9640 1 -- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8274062ea0 msgr2=0x7f82740632b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:24:41.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8270ff9640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8274062ea0 0x7f82740632b0 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8264000c10 tx=0x7f8264019100 comp rx=0 tx=0).stop 2024-09-16T05:24:41.989 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:24:41.989 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:24:41.989 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:24:41.989 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:24:41.989 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 68, 2024-09-16T05:24:41.989 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 258135, 2024-09-16T05:24:41.990 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:24:41.990 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:24:41.990 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:24:41.990 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:24:41.990 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:41.990 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:24:41.990 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:24:41.990 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:24:41.990 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:24:41.990 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:41.990 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:41.990 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:41.990 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:41.990 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:41.991 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:41.991 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:41.991 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:41.991 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:41.991 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:41.991 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:41.991 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:41.991 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:24:41.991 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:24:41.991 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:41.991 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:41.991 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:41.991 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:41.991 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:41.991 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:41.992 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:41.992 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:41.992 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:41.992 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:41.992 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:41.992 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:41.992 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.992 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:24:41.992 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:24:41.992 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:24:41.992 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:24:41.992 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:24:41.992 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:24:41.993 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:24:41.993 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:24:41.993 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:24:41.993 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:24:41.993 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:24:41.993 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:24:41.993 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:24:41.993 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:24:41.993 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:24:41.993 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:24:41.993 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:24:41.993 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:41.993 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:41.993 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:41.994 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:41.994 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:41.994 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:41.994 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:41.994 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:41.994 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:41.994 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:41.994 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:41.994 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:41.994 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:24:41.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.994 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:24:41.994 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.994 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:24:41.995 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:24:41.995 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:41.995 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:41.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.995 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:41.995 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:24:41.995 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:41.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.995 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:41.995 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:24:41.995 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:41.995 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:41.995 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:41.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.996 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:24:41.996 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:24:41.996 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:41.996 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:41.996 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:41.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.997 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.997 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:24:41.997 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:24:41.997 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:41.997 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:41.997 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.997 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:41.997 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:24:41.998 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:41.998 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.998 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.998 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:41.998 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:24:41.998 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:41.998 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:41.998 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:41.998 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.998 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:24:41.998 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:24:41.998 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:41.998 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:41.998 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:41.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.999 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:24:41.999 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:24:41.999 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:41.999 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:41.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.999 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:41.999 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:24:41.999 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:41.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:41.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:41.999 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:42.000 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:24:42.000 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:42.000 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:42.000 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:42.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:42.000 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:24:42.000 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:24:42.001 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:42.001 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:42.001 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:42.001 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:42.001 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:42.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:42.001 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:24:42.001 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:24:42.001 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:42.001 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:42.001 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:42.002 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:24:42.002 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:42.002 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:42.002 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:24:42.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:42.002 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:42.002 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:42.002 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:24:42.002 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:42.002 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:42.002 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:24:42.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:42.002 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:42.002 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:42.003 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:24:42.003 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:42.003 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:42.003 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:24:42.003 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:42.003 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:42.003 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:42.003 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:24:42.003 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:42.003 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:42.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:42.003 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:24:42.003 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:24:42.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 10 librados: watch_flush enter 2024-09-16T05:24:42.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 10 librados: watch_flush exit 2024-09-16T05:24:42.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 20 client.104264.objecter shutdown clearing up homeless session... 2024-09-16T05:24:42.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 10 client.104264.objecter successfully canceled tick 2024-09-16T05:24:42.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 1 -- 172.21.15.114:0/4014829019 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8254037810 msgr2=0x7f8254039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 1 --2- 172.21.15.114:0/4014829019 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8254037810 0x7f8254039cd0 secure :-1 s=READY pgs=3600 cs=0 l=1 rev1=1 crypto rx=0x7f827409ad30 tx=0x7f825c009040 comp rx=0 tx=0).stop 2024-09-16T05:24:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 10 monclient: shutdown 2024-09-16T05:24:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 1 -- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82740a17e0 msgr2=0x7f8274099a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8273fff640 1 -- 172.21.15.114:0/4014829019 reap_dead start 2024-09-16T05:24:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82740a17e0 0x7f8274099a20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f826c00cef0 tx=0x7f826c00a710 comp rx=0 tx=0).stop 2024-09-16T05:24:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 1 -- 172.21.15.114:0/4014829019 shutdown_connections 2024-09-16T05:24:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 1 --2- 172.21.15.114:0/4014829019 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82740a17e0 0x7f8274099a20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 1 -- 172.21.15.114:0/4014829019 >> 172.21.15.114:0/4014829019 conn(0x7f82740a1fb0 msgr2=0x7f8274071b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 1 -- 172.21.15.114:0/4014829019 shutdown_connections 2024-09-16T05:24:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 1 -- 172.21.15.114:0/4014829019 wait complete. 2024-09-16T05:24:42.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:41.970+0000 7f8279c6e640 1 librados: shutdown 2024-09-16T05:24:42.007 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:24:42.007 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:24:42.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:24:42.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 10 monclient: get_monmap_and_config 2024-09-16T05:24:42.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 10 monclient: build_initial_monmap 2024-09-16T05:24:42.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 10 monclient: monmap: 2024-09-16T05:24:42.123 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:42.124 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:42.124 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:24:42.128581+0000 2024-09-16T05:24:42.124 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:24:42.128581+0000 2024-09-16T05:24:42.124 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:42.124 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:42.124 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:24:42.124 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:24:42.124 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:24:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:24:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:24:42.125 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 1 Processor -- start 2024-09-16T05:24:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 1 -- start start 2024-09-16T05:24:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 10 monclient: init 2024-09-16T05:24:42.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:42.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T05:24:42.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65340a0ec0 0x7f653409dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:42.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 10 monclient(hunting): picked mon.noname-b con 0x7f65340a0ec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:42.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f653409e000 addr v1:172.21.15.149:6789/0 2024-09-16T05:24:42.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f6534139640 0x7f653413ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:42.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 10 monclient(hunting): picked mon.noname-c con 0x7f6534139640 addr v2:172.21.15.149:3300/0 2024-09-16T05:24:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f653409a760 con 0x7f653409e000 2024-09-16T05:24:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f65340a12a0 con 0x7f6534139640 2024-09-16T05:24:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f65340994b0 con 0x7f65340a0ec0 2024-09-16T05:24:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f653937d640 10 monclient(hunting): authenticate will time out at 1966.757673s 2024-09-16T05:24:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65340a0ec0 0x7f653409dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65340a0ec0 0x7f653409dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56318/0 (socket says 172.21.15.114:56318) 2024-09-16T05:24:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 1 -- 172.21.15.114:0/546732879 learned_addr learned my addr 172.21.15.114:0/546732879 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 10 monclient(hunting): get_auth_request con 0x7f65340a0ec0 auth_method 0 2024-09-16T05:24:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f65337fe640 1 --2- 172.21.15.114:0/546732879 >> v2:172.21.15.149:3300/0 conn(0x7f6534139640 0x7f653413ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f65337fe640 10 monclient(hunting): get_auth_request con 0x7f6534139640 auth_method 0 2024-09-16T05:24:42.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f65337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:42.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f65337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:42.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f65337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:42.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 10 monclient(hunting): handle_auth_done global_id 104273 payload 274 2024-09-16T05:24:42.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 1 -- 172.21.15.114:0/546732879 >> v2:172.21.15.149:3300/0 conn(0x7f6534139640 msgr2=0x7f653413ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:42.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 1 --2- 172.21.15.114:0/546732879 >> v2:172.21.15.149:3300/0 conn(0x7f6534139640 0x7f653413ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:42.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 1 -- 172.21.15.114:0/546732879 >> v1:172.21.15.149:6789/0 conn(0x7f653409e000 legacy=0x7f653409e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:42.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:24:42.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 1 monclient: found mon.noname-b 2024-09-16T05:24:42.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:42.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:24:42.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.126+0000 7f6532ffd640 1 -- 172.21.15.114:0/546732879 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65340a1870 con 0x7f65340a0ec0 2024-09-16T05:24:42.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 1 --2- 172.21.15.114:0/546732879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65340a0ec0 0x7f653409dac0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6524009e50 tx=0x7f652401c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6125ec467ce00bc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6531ffb640 1 -- 172.21.15.114:0/546732879 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65240a5750 con 0x7f65340a0ec0 2024-09-16T05:24:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6531ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6531ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:24:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6531ffb640 10 monclient: dump: 2024-09-16T05:24:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:42.139 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:42.139 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:42.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6531ffb640 10 monclient: _finish_auth 0 2024-09-16T05:24:42.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6531ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:24:42.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6531ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:42.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 5 monclient: authenticate success, global_id 104273 2024-09-16T05:24:42.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6531ffb640 1 -- 172.21.15.114:0/546732879 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65240a58f0 con 0x7f65340a0ec0 2024-09-16T05:24:42.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:24:42.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6531ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:42.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient: get_monmap_and_config success 2024-09-16T05:24:42.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 4 set_mon_vals no callback set 2024-09-16T05:24:42.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6531ffb640 1 -- 172.21.15.114:0/546732879 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65240a5cc0 con 0x7f65340a0ec0 2024-09-16T05:24:42.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6531ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6531ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6531ffb640 10 monclient: dump: 2024-09-16T05:24:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:42.141 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient: shutdown 2024-09-16T05:24:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 -- 172.21.15.114:0/546732879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65340a0ec0 msgr2=0x7f653409dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 --2- 172.21.15.114:0/546732879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65340a0ec0 0x7f653409dac0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6524009e50 tx=0x7f652401c980 comp rx=0 tx=0).stop 2024-09-16T05:24:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 -- 172.21.15.114:0/546732879 shutdown_connections 2024-09-16T05:24:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 --2- 172.21.15.114:0/546732879 >> v2:172.21.15.149:3300/0 conn(0x7f6534139640 0x7f653413ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 --2- 172.21.15.114:0/546732879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65340a0ec0 0x7f653409dac0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 -- 172.21.15.114:0/546732879 >> 172.21.15.114:0/546732879 conn(0x7f65340a1fb0 msgr2=0x7f65340a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 -- 172.21.15.114:0/546732879 shutdown_connections 2024-09-16T05:24:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 -- 172.21.15.114:0/546732879 wait complete. 2024-09-16T05:24:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient: build_initial_monmap 2024-09-16T05:24:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient: monmap: 2024-09-16T05:24:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:24:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:24:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:24:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:24:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:24:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:24:42.142 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 librados: starting msgr at 2024-09-16T05:24:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 librados: starting objecter 2024-09-16T05:24:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 Processor -- start 2024-09-16T05:24:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 -- start start 2024-09-16T05:24:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 librados: setting wanted keys 2024-09-16T05:24:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 librados: calling monclient init 2024-09-16T05:24:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient: init 2024-09-16T05:24:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:24:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f653409e000 0x7f6534098a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient(hunting): picked mon.noname-c con 0x7f653409e000 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65340a0ec0 0x7f653408d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient(hunting): picked mon.noname-b con 0x7f65340a0ec0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:24:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6534139640 0x7f653408d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient(hunting): picked mon.noname-a con 0x7f6534139640 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f653413c900 con 0x7f6534139640 2024-09-16T05:24:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f653413c780 con 0x7f65340a0ec0 2024-09-16T05:24:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f653413ca80 con 0x7f653409e000 2024-09-16T05:24:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient(hunting): authenticate will time out at 1966.762805s 2024-09-16T05:24:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f653409e000 0x7f6534098a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f653409e000 0x7f6534098a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56330/0 (socket says 172.21.15.114:56330) 2024-09-16T05:24:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f65337fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6534139640 0x7f653408d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 1 -- 172.21.15.114:0/1889258558 learned_addr learned my addr 172.21.15.114:0/1889258558 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 10 monclient(hunting): get_auth_request con 0x7f653409e000 auth_method 0 2024-09-16T05:24:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f65337fe640 10 monclient(hunting): get_auth_request con 0x7f6534139640 auth_method 0 2024-09-16T05:24:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f65337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f65337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f65337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f65337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:42.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f65337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:42.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f65337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:42.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f65327fc640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65340a0ec0 0x7f653408d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:42.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 10 monclient(hunting): handle_auth_done global_id 104276 payload 824 2024-09-16T05:24:42.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 1 -- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65340a0ec0 msgr2=0x7f653408d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:42.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65340a0ec0 0x7f653408d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:42.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 1 -- 172.21.15.114:0/1889258558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6534139640 msgr2=0x7f653408d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:42.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6534139640 0x7f653408d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:42.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:24:42.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 1 monclient: found mon.noname-c 2024-09-16T05:24:42.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:42.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:24:42.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 1 -- 172.21.15.114:0/1889258558 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653408f7a0 con 0x7f653409e000 2024-09-16T05:24:42.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f65327fc640 10 monclient: get_auth_request con 0x7f65340a0ec0 auth_method 0 2024-09-16T05:24:42.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f65327fc640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65340a0ec0 0x7f653408d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:24:42.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f6532ffd640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f653409e000 0x7f6534098a80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f65240af040 tx=0x7f6524019720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:42.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f652b7fe640 10 client.?.objecter ms_handle_connect 0x7f653409e000 2024-09-16T05:24:42.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f652b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:24:42.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f652b7fe640 1 -- 172.21.15.114:0/1889258558 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65240bc020 con 0x7f653409e000 2024-09-16T05:24:42.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f652b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:42.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f652b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:24:42.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f652b7fe640 10 monclient: dump: 2024-09-16T05:24:42.147 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:42.147 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f652b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f652b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f652b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f652b7fe640 1 -- 172.21.15.114:0/1889258558 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65240b8690 con 0x7f653409e000 2024-09-16T05:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f652b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f652b7fe640 1 -- 172.21.15.114:0/1889258558 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65240b8a80 con 0x7f653409e000 2024-09-16T05:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f652b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f652b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f652b7fe640 10 monclient: dump: 2024-09-16T05:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 5 monclient: authenticate success, global_id 104276 2024-09-16T05:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient: _renew_subs 2024-09-16T05:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f65317fa640 4 set_mon_vals no callback set 2024-09-16T05:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 -- 172.21.15.114:0/1889258558 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65340a0a00 con 0x7f653409e000 2024-09-16T05:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 client.104276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient: _renew_subs 2024-09-16T05:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 -- 172.21.15.114:0/1889258558 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f653409c6a0 con 0x7f653409e000 2024-09-16T05:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.130+0000 7f653937d640 1 librados: init done 2024-09-16T05:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f653937d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f653937d640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 0x7f6534090940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f653937d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f653937d640 1 -- 172.21.15.114:0/1889258558 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f653408ec40 con 0x7f653408dcb0 2024-09-16T05:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f652b7fe640 1 -- 172.21.15.114:0/1889258558 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f65240bf020 con 0x7f653409e000 2024-09-16T05:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f65327fc640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 0x7f6534090940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f65327fc640 10 monclient: get_auth_request con 0x7f653408dcb0 auth_method 0 2024-09-16T05:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f65327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f65327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f65327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f652b7fe640 1 --2- 172.21.15.114:0/1889258558 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6514037810 0x7f6514039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f652b7fe640 1 -- 172.21.15.114:0/1889258558 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(686..686 src has 1..686) ==== 5592+0+0 (secure 0 0 0) 0x7f65240e3f60 con 0x7f653409e000 2024-09-16T05:24:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f652b7fe640 10 client.104276.objecter ms_dispatch 0x7f6534000ba0 osd_map(686..686 src has 1..686) 2024-09-16T05:24:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f652b7fe640 3 client.104276.objecter handle_osd_map got epochs [686,686] > 0 2024-09-16T05:24:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f652b7fe640 3 client.104276.objecter handle_osd_map decoding full epoch 686 2024-09-16T05:24:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f65327fc640 1 -- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 msgr2=0x7f6534090940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:24:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f65327fc640 1 -- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 msgr2=0x7f6534090940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:24:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f65327fc640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 0x7f6534090940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:24:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f65327fc640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 0x7f6534090940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:24:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f652b7fe640 20 client.104276.objecter dump_active .. 0 homeless 2024-09-16T05:24:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f65337fe640 1 --2- 172.21.15.114:0/1889258558 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6514037810 0x7f6514039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f65337fe640 10 monclient: get_auth_request con 0x7f6514037810 auth_method 0 2024-09-16T05:24:42.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f652b7fe640 10 client.104276.objecter ms_handle_connect 0x7f6514037810 2024-09-16T05:24:42.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.134+0000 7f65337fe640 1 --2- 172.21.15.114:0/1889258558 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6514037810 0x7f6514039cd0 secure :-1 s=READY pgs=3601 cs=0 l=1 rev1=1 crypto rx=0x7f6534090630 tx=0x7f652c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.338+0000 7f65327fc640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 0x7f6534090940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.338+0000 7f65327fc640 10 monclient: get_auth_request con 0x7f653408dcb0 auth_method 0 2024-09-16T05:24:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.338+0000 7f65327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.338+0000 7f65327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.338+0000 7f65327fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:24:42.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.338+0000 7f65327fc640 1 -- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 msgr2=0x7f6534090940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:24:42.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.338+0000 7f65327fc640 1 -- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 msgr2=0x7f6534090940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:24:42.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.338+0000 7f65327fc640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 0x7f6534090940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:24:42.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.338+0000 7f65327fc640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 0x7f6534090940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:24:42.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.742+0000 7f65327fc640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 0x7f6534090940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:42.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.742+0000 7f65327fc640 10 monclient: get_auth_request con 0x7f653408dcb0 auth_method 0 2024-09-16T05:24:42.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.742+0000 7f65327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:42.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.742+0000 7f65327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:42.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.742+0000 7f65327fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:24:42.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.742+0000 7f65327fc640 1 -- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 msgr2=0x7f6534090940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:24:42.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.742+0000 7f65327fc640 1 -- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 msgr2=0x7f6534090940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:24:42.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.742+0000 7f65327fc640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 0x7f6534090940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:24:42.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:42.742+0000 7f65327fc640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 0x7f6534090940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:24:43.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f65327fc640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 0x7f6534090940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:43.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f65327fc640 10 monclient: get_auth_request con 0x7f653408dcb0 auth_method 0 2024-09-16T05:24:43.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f65327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:43.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f65327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:43.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f65327fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:24:43.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f65327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:43.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f65327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:43.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f65327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:43.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f65327fc640 10 monclient(hunting): handle_auth_done global_id 124099 payload 824 2024-09-16T05:24:43.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f65327fc640 1 -- 172.21.15.114:0/1889258558 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f653408e090 con 0x7f653408dcb0 2024-09-16T05:24:43.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f65327fc640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 0x7f6534090940 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f65340a46e0 tx=0x7f651c012cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f652b7fe640 10 client.104276.objecter ms_handle_connect 0x7f653408dcb0 2024-09-16T05:24:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f652b7fe640 10 client.104276.objecter resend_mon_ops 2024-09-16T05:24:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f652b7fe640 1 -- 172.21.15.114:0/1889258558 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651c019070 con 0x7f653408dcb0 2024-09-16T05:24:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f652b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f652b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:24:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f652b7fe640 10 monclient: dump: 2024-09-16T05:24:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:43.546 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:43.546 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:43.546 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:43.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f652b7fe640 1 -- 172.21.15.114:0/1889258558 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f651c0050d0 con 0x7f653408dcb0 2024-09-16T05:24:43.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f652b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:43.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f652b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:24:43.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f652b7fe640 1 -- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 msgr2=0x7f6534090940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:43.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f652b7fe640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 0x7f6534090940 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f65340a46e0 tx=0x7f651c012cf0 comp rx=0 tx=0).stop 2024-09-16T05:24:43.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f652b7fe640 1 -- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 msgr2=0x7f6534090940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:24:43.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.546+0000 7f652b7fe640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f653408dcb0 0x7f6534090940 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f65340a46e0 tx=0x7f651c012cf0 comp rx=0 tx=0).stop 2024-09-16T05:24:43.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.554+0000 7f653937d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:24:43.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.554+0000 7f653937d640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6534090e80 0x7f653406f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:43.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.554+0000 7f653937d640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:43.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.554+0000 7f653937d640 1 -- 172.21.15.114:0/1889258558 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f6534069390 con 0x7f6534090e80 2024-09-16T05:24:43.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.554+0000 7f65327fc640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6534090e80 0x7f653406f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:43.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.554+0000 7f65327fc640 10 monclient: get_auth_request con 0x7f6534090e80 auth_method 0 2024-09-16T05:24:43.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.554+0000 7f65327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:43.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.554+0000 7f65327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:43.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.554+0000 7f65327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:43.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.554+0000 7f65327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:43.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.554+0000 7f65327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:43.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.554+0000 7f65327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:43.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.554+0000 7f65327fc640 10 monclient(hunting): handle_auth_done global_id 124102 payload 824 2024-09-16T05:24:43.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.554+0000 7f65327fc640 1 -- 172.21.15.114:0/1889258558 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f653409e3e0 con 0x7f6534090e80 2024-09-16T05:24:43.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f65327fc640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6534090e80 0x7f653406f9a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f651c01b000 tx=0x7f651c000bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:43.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f652b7fe640 10 client.104276.objecter ms_handle_connect 0x7f6534090e80 2024-09-16T05:24:43.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f652b7fe640 10 client.104276.objecter resend_mon_ops 2024-09-16T05:24:43.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f652b7fe640 1 -- 172.21.15.114:0/1889258558 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651c02c020 con 0x7f6534090e80 2024-09-16T05:24:43.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f652b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:43.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f652b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:24:43.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f652b7fe640 10 monclient: dump: 2024-09-16T05:24:43.555 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:43.556 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f652b7fe640 1 -- 172.21.15.114:0/1889258558 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f651c003b20 con 0x7f6534090e80 2024-09-16T05:24:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f652b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:24:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f652b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:24:43.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f652b7fe640 1 -- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6534090e80 msgr2=0x7f653406f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f652b7fe640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6534090e80 0x7f653406f9a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f651c01b000 tx=0x7f651c000bd0 comp rx=0 tx=0).stop 2024-09-16T05:24:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f652b7fe640 1 -- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6534090e80 msgr2=0x7f653406f9a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:24:43.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f652b7fe640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6534090e80 0x7f653406f9a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:43.557 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:24:43.557 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:24:43.557 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:24:43.557 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:24:43.557 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 68, 2024-09-16T05:24:43.557 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 5583, 2024-09-16T05:24:43.557 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:24:43.558 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:24:43.558 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:24:43.558 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:24:43.558 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:43.558 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 3, 2024-09-16T05:24:43.558 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:24:43.558 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:24:43.558 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:24:43.558 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:43.559 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:43.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:43.559 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:43.559 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:43.559 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:43.559 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:43.559 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:43.559 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:43.559 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:43.559 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:43.559 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:43.559 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:24:43.559 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:24:43.560 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:43.560 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:43.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:43.560 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:43.560 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:43.560 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:43.560 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:43.560 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:43.560 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:43.560 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:43.560 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:43.560 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:43.560 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:43.561 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:24:43.561 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:24:43.561 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:24:43.561 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:24:43.561 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:24:43.561 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:24:43.561 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:24:43.561 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:24:43.561 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:24:43.561 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:24:43.561 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:24:43.561 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:24:43.561 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:24:43.561 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:24:43.562 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:24:43.562 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:24:43.562 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:24:43.562 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:43.562 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:43.562 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:43.562 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:43.562 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:43.562 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:43.562 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:43.562 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:43.562 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:43.562 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:43.562 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:43.563 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:43.563 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:24:43.563 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:43.563 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:24:43.563 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:43.563 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:24:43.563 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:24:43.563 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:43.563 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:43.563 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:43.563 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:43.563 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:24:43.563 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:43.564 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:43.564 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:43.564 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:43.564 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:24:43.564 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:43.564 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:43.564 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:43.564 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:43.564 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:24:43.564 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:24:43.564 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:43.564 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:43.565 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:43.565 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:43.565 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:43.565 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:24:43.565 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:24:43.565 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:43.565 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:43.565 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:43.565 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:43.565 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:24:43.565 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:43.565 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:43.565 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:43.565 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:43.566 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:24:43.566 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:43.566 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:43.566 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:43.566 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:43.566 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:24:43.566 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:24:43.566 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:43.566 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:43.566 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:43.566 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:43.566 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:43.566 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:24:43.567 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:24:43.567 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:43.567 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:43.567 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:43.567 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:43.567 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:24:43.567 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:43.567 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:43.567 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:43.567 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:43.567 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:24:43.567 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:43.567 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:43.567 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:43.568 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:43.568 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:24:43.568 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:24:43.568 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:43.568 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:43.568 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:43.568 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:43.568 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:43.568 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:43.568 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:24:43.568 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:24:43.568 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:43.568 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:43.569 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:43.569 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:24:43.569 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:43.569 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:43.569 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:24:43.569 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:43.569 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:43.569 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:43.569 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:24:43.569 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:43.569 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:43.569 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:43.569 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:24:43.569 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:24:43.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f653937d640 10 librados: watch_flush enter 2024-09-16T05:24:43.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f653937d640 10 librados: watch_flush exit 2024-09-16T05:24:43.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f653937d640 20 client.104276.objecter shutdown clearing up homeless session... 2024-09-16T05:24:43.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f653937d640 10 client.104276.objecter successfully canceled tick 2024-09-16T05:24:43.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f653937d640 1 -- 172.21.15.114:0/1889258558 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6514037810 msgr2=0x7f6514039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:43.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f653937d640 1 --2- 172.21.15.114:0/1889258558 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6514037810 0x7f6514039cd0 secure :-1 s=READY pgs=3601 cs=0 l=1 rev1=1 crypto rx=0x7f6534090630 tx=0x7f652c008040 comp rx=0 tx=0).stop 2024-09-16T05:24:43.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f653937d640 10 monclient: shutdown 2024-09-16T05:24:43.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f653937d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:43.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f653937d640 1 -- 172.21.15.114:0/1889258558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f653409e000 msgr2=0x7f6534098a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:43.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f653937d640 1 --2- 172.21.15.114:0/1889258558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f653409e000 0x7f6534098a80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f65240af040 tx=0x7f6524019720 comp rx=0 tx=0).stop 2024-09-16T05:24:43.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f65337fe640 1 -- 172.21.15.114:0/1889258558 reap_dead start 2024-09-16T05:24:43.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f65337fe640 1 -- 172.21.15.114:0/1889258558 reap_dead start 2024-09-16T05:24:43.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f653937d640 1 -- 172.21.15.114:0/1889258558 shutdown_connections 2024-09-16T05:24:43.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f653937d640 1 -- 172.21.15.114:0/1889258558 >> 172.21.15.114:0/1889258558 conn(0x7f65340a1fb0 msgr2=0x7f653413a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:43.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f653937d640 1 -- 172.21.15.114:0/1889258558 shutdown_connections 2024-09-16T05:24:43.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f653937d640 1 -- 172.21.15.114:0/1889258558 wait complete. 2024-09-16T05:24:43.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.558+0000 7f653937d640 1 librados: shutdown 2024-09-16T05:24:43.576 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:24:43.576 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:24:43.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:24:43.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.702+0000 7fb90a315640 10 monclient: get_monmap_and_config 2024-09-16T05:24:43.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.702+0000 7fb90a315640 10 monclient: build_initial_monmap 2024-09-16T05:24:43.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.702+0000 7fb90a315640 10 monclient: monmap: 2024-09-16T05:24:43.701 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:43.701 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:43.701 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:24:43.707098+0000 2024-09-16T05:24:43.701 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:24:43.707098+0000 2024-09-16T05:24:43.702 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:43.702 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:43.702 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:24:43.702 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:24:43.702 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:24:43.702 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:24:43.702 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:24:43.702 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:43.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.702+0000 7fb90a315640 1 Processor -- start 2024-09-16T05:24:43.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.702+0000 7fb90a315640 1 -- start start 2024-09-16T05:24:43.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.702+0000 7fb90a315640 10 monclient: init 2024-09-16T05:24:43.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:43.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-16T05:24:43.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb904137730 0x7fb904137b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:43.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 10 monclient(hunting): picked mon.noname-b con 0x7fb904137730 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:43.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb9040a5d00 addr v1:172.21.15.114:6789/0 2024-09-16T05:24:43.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb9040a4980 addr v1:172.21.15.149:6789/0 2024-09-16T05:24:43.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9040a55e0 con 0x7fb9040a5d00 2024-09-16T05:24:43.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9040a3710 con 0x7fb9040a4980 2024-09-16T05:24:43.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:43.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb90409c450 con 0x7fb904137730 2024-09-16T05:24:43.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:43.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 10 monclient(hunting): authenticate will time out at 1968.336278s 2024-09-16T05:24:43.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb904137730 0x7fb904137b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:43.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb9037fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb9040a5d00 0x7fb9040a4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37972/0 (socket says 172.21.15.114:37972) 2024-09-16T05:24:43.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb9037fe640 1 -- 172.21.15.114:0/3519615148 learned_addr learned my addr 172.21.15.114:0/3519615148 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:24:43.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 10 monclient(hunting): get_auth_request con 0x7fb904137730 auth_method 0 2024-09-16T05:24:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 10 monclient(hunting): handle_auth_done global_id 104279 payload 274 2024-09-16T05:24:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 1 -- 172.21.15.114:0/3519615148 >> v1:172.21.15.149:6789/0 conn(0x7fb9040a4980 legacy=0x7fb90413f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 1 -- 172.21.15.114:0/3519615148 >> v1:172.21.15.114:6789/0 conn(0x7fb9040a5d00 legacy=0x7fb9040a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 10 monclient: _finish_hunting 0 2024-09-16T05:24:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 1 monclient: found mon.noname-b 2024-09-16T05:24:43.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:43.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:24:43.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 1 -- 172.21.15.114:0/3519615148 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90409be00 con 0x7fb904137730 2024-09-16T05:24:43.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb903fff640 1 --2- 172.21.15.114:0/3519615148 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb904137730 0x7fb904137b10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb8f40097f0 tx=0x7fb8f401bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61a606cd2f7a8aaa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:43.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb902ffd640 1 -- 172.21.15.114:0/3519615148 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f40af600 con 0x7fb904137730 2024-09-16T05:24:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb902ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb902ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:24:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb902ffd640 10 monclient: dump: 2024-09-16T05:24:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:43.713 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb902ffd640 10 monclient: _finish_auth 0 2024-09-16T05:24:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb902ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:24:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb902ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb902ffd640 1 -- 172.21.15.114:0/3519615148 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8f40af7a0 con 0x7fb904137730 2024-09-16T05:24:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb902ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb902ffd640 1 -- 172.21.15.114:0/3519615148 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f40afb70 con 0x7fb904137730 2024-09-16T05:24:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 5 monclient: authenticate success, global_id 104279 2024-09-16T05:24:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 10 monclient: get_monmap_and_config success 2024-09-16T05:24:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 4 set_mon_vals no callback set 2024-09-16T05:24:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb902ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb902ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb902ffd640 10 monclient: dump: 2024-09-16T05:24:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:43.714 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 10 monclient: shutdown 2024-09-16T05:24:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 1 -- 172.21.15.114:0/3519615148 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb904137730 msgr2=0x7fb904137b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 1 --2- 172.21.15.114:0/3519615148 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb904137730 0x7fb904137b10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb8f40097f0 tx=0x7fb8f401bf60 comp rx=0 tx=0).stop 2024-09-16T05:24:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 1 -- 172.21.15.114:0/3519615148 shutdown_connections 2024-09-16T05:24:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 1 --2- 172.21.15.114:0/3519615148 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb904137730 0x7fb904137b10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 1 -- 172.21.15.114:0/3519615148 >> 172.21.15.114:0/3519615148 conn(0x7fb9040a9c70 msgr2=0x7fb9040aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 1 -- 172.21.15.114:0/3519615148 shutdown_connections 2024-09-16T05:24:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.706+0000 7fb90a315640 1 -- 172.21.15.114:0/3519615148 wait complete. 2024-09-16T05:24:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient: build_initial_monmap 2024-09-16T05:24:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient: monmap: 2024-09-16T05:24:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:24:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:24:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:24:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:24:43.715 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:24:43.716 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:24:43.716 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:43.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 librados: starting msgr at 2024-09-16T05:24:43.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 librados: starting objecter 2024-09-16T05:24:43.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 Processor -- start 2024-09-16T05:24:43.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 -- start start 2024-09-16T05:24:43.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 librados: setting wanted keys 2024-09-16T05:24:43.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 librados: calling monclient init 2024-09-16T05:24:43.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient: init 2024-09-16T05:24:43.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:43.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:24:43.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9040a4980 0x7fb9041d0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:43.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient(hunting): picked mon.noname-b con 0x7fb9040a4980 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:24:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9040a5d00 0x7fb904136400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient(hunting): picked mon.noname-c con 0x7fb9040a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb904137730 0x7fb904136940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient(hunting): picked mon.noname-a con 0x7fb904137730 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb904143fd0 con 0x7fb904137730 2024-09-16T05:24:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb904143e50 con 0x7fb9040a4980 2024-09-16T05:24:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb904144150 con 0x7fb9040a5d00 2024-09-16T05:24:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient(hunting): authenticate will time out at 1968.341215s 2024-09-16T05:24:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9040a5d00 0x7fb904136400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9088b2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb904137730 0x7fb904136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9040a5d00 0x7fb904136400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56352/0 (socket says 172.21.15.114:56352) 2024-09-16T05:24:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 1 -- 172.21.15.114:0/2378634337 learned_addr learned my addr 172.21.15.114:0/2378634337 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9088b2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb904137730 0x7fb904136940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40170/0 (socket says 172.21.15.114:40170) 2024-09-16T05:24:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 10 monclient(hunting): get_auth_request con 0x7fb9040a5d00 auth_method 0 2024-09-16T05:24:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9088b2640 10 monclient(hunting): get_auth_request con 0x7fb904137730 auth_method 0 2024-09-16T05:24:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9088b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9088b2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9088b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:43.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:43.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:43.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9088b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:43.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9088b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:43.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9088b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:43.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb903fff640 1 --2- 172.21.15.114:0/2378634337 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9040a4980 0x7fb9041d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:43.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb903fff640 10 monclient(hunting): get_auth_request con 0x7fb9040a4980 auth_method 0 2024-09-16T05:24:43.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb903fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:43.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb903fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:43.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb903fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:43.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 10 monclient(hunting): handle_auth_done global_id 104282 payload 824 2024-09-16T05:24:43.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 1 -- 172.21.15.114:0/2378634337 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9040a4980 msgr2=0x7fb9041d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:43.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 1 --2- 172.21.15.114:0/2378634337 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9040a4980 0x7fb9041d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:43.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 1 -- 172.21.15.114:0/2378634337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb904137730 msgr2=0x7fb904136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:43.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 1 --2- 172.21.15.114:0/2378634337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb904137730 0x7fb904136940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:43.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 10 monclient: _finish_hunting 0 2024-09-16T05:24:43.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 1 monclient: found mon.noname-c 2024-09-16T05:24:43.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:43.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:24:43.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 1 -- 172.21.15.114:0/2378634337 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90413ff00 con 0x7fb9040a5d00 2024-09-16T05:24:43.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9088b2640 1 --2- 172.21.15.114:0/2378634337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb904137730 0x7fb904136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:24:43.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9037fe640 1 --2- 172.21.15.114:0/2378634337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9040a5d00 0x7fb904136400 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb8f800c780 tx=0x7fb8f800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:43.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 10 client.?.objecter ms_handle_connect 0x7fb9040a5d00 2024-09-16T05:24:43.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:24:43.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 1 -- 172.21.15.114:0/2378634337 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f8018070 con 0x7fb9040a5d00 2024-09-16T05:24:43.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:24:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 10 monclient: dump: 2024-09-16T05:24:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:43.721 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 10 monclient: _finish_auth 0 2024-09-16T05:24:43.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:24:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 1 -- 172.21.15.114:0/2378634337 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8f80072f0 con 0x7fb9040a5d00 2024-09-16T05:24:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 1 -- 172.21.15.114:0/2378634337 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f80047f0 con 0x7fb9040a5d00 2024-09-16T05:24:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 10 monclient: dump: 2024-09-16T05:24:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:43.723 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 5 monclient: authenticate success, global_id 104282 2024-09-16T05:24:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient: _renew_subs 2024-09-16T05:24:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 -- 172.21.15.114:0/2378634337 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb904140920 con 0x7fb9040a5d00 2024-09-16T05:24:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb9027fc640 4 set_mon_vals no callback set 2024-09-16T05:24:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 client.104282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:24:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient: _renew_subs 2024-09-16T05:24:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 -- 172.21.15.114:0/2378634337 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb90413e7d0 con 0x7fb9040a5d00 2024-09-16T05:24:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 librados: init done 2024-09-16T05:24:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 1 -- 172.21.15.114:0/2378634337 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb8f8004990 con 0x7fb9040a5d00 2024-09-16T05:24:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb90a315640 1 -- 172.21.15.114:0/2378634337 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb904137b10 con 0x7fb9040a5d00 2024-09-16T05:24:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 1 --2- 172.21.15.114:0/2378634337 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb8f0037810 0x7fb8f0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 1 -- 172.21.15.114:0/2378634337 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(688..688 src has 1..688) ==== 5978+0+0 (secure 0 0 0) 0x7fb8f80484d0 con 0x7fb9040a5d00 2024-09-16T05:24:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 10 client.104282.objecter ms_dispatch 0x7fb904000ba0 osd_map(688..688 src has 1..688) 2024-09-16T05:24:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 3 client.104282.objecter handle_osd_map got epochs [688,688] > 0 2024-09-16T05:24:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 3 client.104282.objecter handle_osd_map decoding full epoch 688 2024-09-16T05:24:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb900ff9640 20 client.104282.objecter dump_active .. 0 homeless 2024-09-16T05:24:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb903fff640 1 --2- 172.21.15.114:0/2378634337 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb8f0037810 0x7fb8f0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.710+0000 7fb903fff640 10 monclient: get_auth_request con 0x7fb8f0037810 auth_method 0 2024-09-16T05:24:43.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.714+0000 7fb900ff9640 10 client.104282.objecter ms_handle_connect 0x7fb8f0037810 2024-09-16T05:24:43.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.714+0000 7fb903fff640 1 --2- 172.21.15.114:0/2378634337 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb8f0037810 0x7fb8f0039cd0 secure :-1 s=READY pgs=3602 cs=0 l=1 rev1=1 crypto rx=0x7fb8f40ae000 tx=0x7fb8f4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:43.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.718+0000 7fb900ff9640 1 -- 172.21.15.114:0/2378634337 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb8f801a310 con 0x7fb9040a5d00 2024-09-16T05:24:43.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.718+0000 7fb900ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:43.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.718+0000 7fb900ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:24:43.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.942+0000 7fb90a315640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:24:43.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.942+0000 7fb90a315640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:24:43.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.942+0000 7fb90a315640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:43.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.942+0000 7fb90a315640 1 -- 172.21.15.114:0/2378634337 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fb90413def0 con 0x7fb9040a5d00 2024-09-16T05:24:43.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb900ff9640 1 -- 172.21.15.114:0/2378634337 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fb8f8004c80 con 0x7fb9040a5d00 2024-09-16T05:24:43.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb900ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:24:43.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb900ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:24:43.943 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.946+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.943 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.946+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=12,kv=12,logm=52} crc {auth=4058436830,config=2586702516,health=1699732930,kv=2757161437,logm=3418053362}) 2024-09-16T05:24:43.943 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.946+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=10,kv=11,logm=56} crc {auth=902803791,config=2586702516,health=2619282021,kv=1951643036,logm=3506291803}) 2024-09-16T05:24:43.943 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.946+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.943 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.946+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=12,kv=12,logm=52} crc {auth=4058436830,config=2586702516,health=1699732930,kv=2757161437,logm=3418053362}) 2024-09-16T05:24:43.943 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.946+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=11,kv=11,logm=54} crc {auth=4058436830,config=2586702516,health=4095903664,kv=800243116,logm=464025539}) 2024-09-16T05:24:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 10 librados: watch_flush enter 2024-09-16T05:24:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 10 librados: watch_flush exit 2024-09-16T05:24:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 20 client.104282.objecter shutdown clearing up homeless session... 2024-09-16T05:24:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 10 client.104282.objecter successfully canceled tick 2024-09-16T05:24:43.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 1 -- 172.21.15.114:0/2378634337 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb8f0037810 msgr2=0x7fb8f0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:43.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 1 --2- 172.21.15.114:0/2378634337 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb8f0037810 0x7fb8f0039cd0 secure :-1 s=READY pgs=3602 cs=0 l=1 rev1=1 crypto rx=0x7fb8f40ae000 tx=0x7fb8f4008040 comp rx=0 tx=0).stop 2024-09-16T05:24:43.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 10 monclient: shutdown 2024-09-16T05:24:43.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:43.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 1 -- 172.21.15.114:0/2378634337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9040a5d00 msgr2=0x7fb904136400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:43.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 1 --2- 172.21.15.114:0/2378634337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9040a5d00 0x7fb904136400 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb8f800c780 tx=0x7fb8f800cc40 comp rx=0 tx=0).stop 2024-09-16T05:24:43.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 1 -- 172.21.15.114:0/2378634337 shutdown_connections 2024-09-16T05:24:43.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 1 --2- 172.21.15.114:0/2378634337 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb8f0037810 0x7fb8f0039cd0 unknown :-1 s=CLOSED pgs=3602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:43.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 1 --2- 172.21.15.114:0/2378634337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb904137730 0x7fb904136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:43.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 1 --2- 172.21.15.114:0/2378634337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9040a5d00 0x7fb904136400 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:43.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 1 --2- 172.21.15.114:0/2378634337 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9040a4980 0x7fb9041d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:43.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 1 -- 172.21.15.114:0/2378634337 >> 172.21.15.114:0/2378634337 conn(0x7fb9040a9c70 msgr2=0x7fb9040a8600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:43.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 1 -- 172.21.15.114:0/2378634337 shutdown_connections 2024-09-16T05:24:43.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 1 -- 172.21.15.114:0/2378634337 wait complete. 2024-09-16T05:24:43.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:43.946+0000 7fb90a315640 1 librados: shutdown 2024-09-16T05:24:43.947 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.946+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.948 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.946+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505802895}) 2024-09-16T05:24:43.948 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.946+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2208528094}) 2024-09-16T05:24:43.948 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.946+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.948 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.946+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505802895}) 2024-09-16T05:24:43.948 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.946+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4121035045}) 2024-09-16T05:24:43.949 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.950+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.949 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.950+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876449205}) 2024-09-16T05:24:43.949 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.950+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3051470976}) 2024-09-16T05:24:43.949 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.950+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.949 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.950+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876449205}) 2024-09-16T05:24:43.949 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.950+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2474102748}) 2024-09-16T05:24:43.952 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.954+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.952 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.954+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2992962314}) 2024-09-16T05:24:43.952 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.954+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2208271293}) 2024-09-16T05:24:43.952 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.954+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.952 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.954+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2992962314}) 2024-09-16T05:24:43.952 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.954+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=264946677}) 2024-09-16T05:24:43.955 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.958+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.955 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.958+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2900508007}) 2024-09-16T05:24:43.955 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.958+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1402920383}) 2024-09-16T05:24:43.955 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.958+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.955 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.958+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2900508007}) 2024-09-16T05:24:43.955 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.958+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2762991730}) 2024-09-16T05:24:43.957 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.958+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.957 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.958+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=636890783}) 2024-09-16T05:24:43.957 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.958+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2974713660}) 2024-09-16T05:24:43.957 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.958+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.957 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.958+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=636890783}) 2024-09-16T05:24:43.957 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.958+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2339566472}) 2024-09-16T05:24:43.959 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.962+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.959 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.962+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4283435111}) 2024-09-16T05:24:43.959 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.962+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2514003644}) 2024-09-16T05:24:43.959 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.962+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.960 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.962+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4283435111}) 2024-09-16T05:24:43.960 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.962+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2826965488}) 2024-09-16T05:24:43.961 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.962+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.962 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.962+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=343199932}) 2024-09-16T05:24:43.962 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.962+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1606696948}) 2024-09-16T05:24:43.962 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.962+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.962 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.962+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=343199932}) 2024-09-16T05:24:43.962 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.962+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1347169026}) 2024-09-16T05:24:43.963 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.966+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.963 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.966+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291832949}) 2024-09-16T05:24:43.964 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.966+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3890484349}) 2024-09-16T05:24:43.964 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.966+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.964 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.966+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291832949}) 2024-09-16T05:24:43.964 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.966+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428572162}) 2024-09-16T05:24:43.965 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:24:43.965 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.966+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.965 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.966+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=393404831}) 2024-09-16T05:24:43.966 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.966+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2647947840}) 2024-09-16T05:24:43.966 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.966+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.966 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.966+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=393404831}) 2024-09-16T05:24:43.966 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.966+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4228449277}) 2024-09-16T05:24:43.967 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.970+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.967 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.970+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75847217}) 2024-09-16T05:24:43.967 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.970+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3180454150}) 2024-09-16T05:24:43.967 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.970+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.968 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.970+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75847217}) 2024-09-16T05:24:43.968 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.970+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=363530207}) 2024-09-16T05:24:43.969 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.970+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.970 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.970+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967098950}) 2024-09-16T05:24:43.970 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.970+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=971130803}) 2024-09-16T05:24:43.970 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.970+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.970 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.970+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967098950}) 2024-09-16T05:24:43.970 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.970+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1028632277}) 2024-09-16T05:24:43.971 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.974+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.971 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.974+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=663543545}) 2024-09-16T05:24:43.971 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.974+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1384666499}) 2024-09-16T05:24:43.971 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.974+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.972 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.974+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=663543545}) 2024-09-16T05:24:43.972 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.974+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=735608422}) 2024-09-16T05:24:43.973 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.974+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.973 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.974+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087617378}) 2024-09-16T05:24:43.973 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.974+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3941750666}) 2024-09-16T05:24:43.973 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.974+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.974 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.974+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087617378}) 2024-09-16T05:24:43.974 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.974+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3952158463}) 2024-09-16T05:24:43.975 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.978+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.976 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.978+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1458611457}) 2024-09-16T05:24:43.976 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.978+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=329138963}) 2024-09-16T05:24:43.976 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.978+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.976 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.978+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1458611457}) 2024-09-16T05:24:43.976 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.978+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3624831538}) 2024-09-16T05:24:43.978 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.978+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.978 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.978+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1585417421}) 2024-09-16T05:24:43.978 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.978+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3530374732}) 2024-09-16T05:24:43.978 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.978+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.978 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.978+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1585417421}) 2024-09-16T05:24:43.978 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.978+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2637041319}) 2024-09-16T05:24:43.980 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.982+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.980 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.982+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1333897170}) 2024-09-16T05:24:43.980 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.982+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=133618330}) 2024-09-16T05:24:43.980 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.982+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.980 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.982+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1333897170}) 2024-09-16T05:24:43.980 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.982+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2968081272}) 2024-09-16T05:24:43.981 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.982+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.981 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.982+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2474871026}) 2024-09-16T05:24:43.981 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.982+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1631707658}) 2024-09-16T05:24:43.981 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.982+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.982 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.982+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2474871026}) 2024-09-16T05:24:43.982 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.982+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1121798296}) 2024-09-16T05:24:43.983 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.986+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.983 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.986+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761632102}) 2024-09-16T05:24:43.983 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.986+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615032352}) 2024-09-16T05:24:43.983 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.986+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.983 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.986+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761632102}) 2024-09-16T05:24:43.983 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.986+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=467263447}) 2024-09-16T05:24:43.985 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.986+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.985 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.986+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892526471}) 2024-09-16T05:24:43.985 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.986+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3144804505}) 2024-09-16T05:24:43.985 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.986+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.986 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.986+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892526471}) 2024-09-16T05:24:43.986 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.986+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3508271570}) 2024-09-16T05:24:43.987 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.990+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.987 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.990+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953115463}) 2024-09-16T05:24:43.987 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.990+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1279323581}) 2024-09-16T05:24:43.988 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.990+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.988 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.990+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953115463}) 2024-09-16T05:24:43.988 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.990+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407838571}) 2024-09-16T05:24:43.989 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.990+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.989 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.990+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4141869627}) 2024-09-16T05:24:43.989 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.990+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2317409745}) 2024-09-16T05:24:43.989 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.990+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.989 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.990+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4141869627}) 2024-09-16T05:24:43.989 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.990+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042642427}) 2024-09-16T05:24:43.990 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.994+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.990 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.994+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4074505651}) 2024-09-16T05:24:43.991 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.994+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1153992715}) 2024-09-16T05:24:43.991 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.994+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.991 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.994+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4074505651}) 2024-09-16T05:24:43.991 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.994+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2915326568}) 2024-09-16T05:24:43.993 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.994+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.993 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.994+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=786570510}) 2024-09-16T05:24:43.993 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.994+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2252739287}) 2024-09-16T05:24:43.993 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.994+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.993 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.994+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=786570510}) 2024-09-16T05:24:43.993 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.994+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2518972894}) 2024-09-16T05:24:43.995 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.998+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.995 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.998+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2823344998}) 2024-09-16T05:24:43.995 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.998+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2827252591}) 2024-09-16T05:24:43.995 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.998+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.995 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.998+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2823344998}) 2024-09-16T05:24:43.995 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.998+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3931893992}) 2024-09-16T05:24:43.997 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.998+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.997 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.998+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283902498}) 2024-09-16T05:24:43.998 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.998+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3050955861}) 2024-09-16T05:24:43.998 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.998+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.998 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.998+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283902498}) 2024-09-16T05:24:43.998 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:43.998+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862494708}) 2024-09-16T05:24:43.998 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.002+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.999 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.002+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1709012339}) 2024-09-16T05:24:43.999 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.002+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1503135725}) 2024-09-16T05:24:43.999 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.002+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:43.999 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.002+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1709012339}) 2024-09-16T05:24:43.999 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.002+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1857081728}) 2024-09-16T05:24:44.000 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.002+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.001 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.002+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605533507}) 2024-09-16T05:24:44.001 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.002+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1313346221}) 2024-09-16T05:24:44.001 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.002+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.001 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.002+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605533507}) 2024-09-16T05:24:44.001 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.002+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1298485372}) 2024-09-16T05:24:44.003 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.003 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=188469412}) 2024-09-16T05:24:44.003 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3486514845}) 2024-09-16T05:24:44.003 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.003 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=188469412}) 2024-09-16T05:24:44.003 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3305003818}) 2024-09-16T05:24:44.004 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.004 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3161992336}) 2024-09-16T05:24:44.004 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1290396305}) 2024-09-16T05:24:44.004 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.005 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3161992336}) 2024-09-16T05:24:44.005 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732657504}) 2024-09-16T05:24:44.006 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.006 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2946636507}) 2024-09-16T05:24:44.006 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=767469951}) 2024-09-16T05:24:44.006 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.006 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2946636507}) 2024-09-16T05:24:44.006 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.006+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3079557936}) 2024-09-16T05:24:44.008 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.010+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.008 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.010+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1771936713}) 2024-09-16T05:24:44.008 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.010+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3320402519}) 2024-09-16T05:24:44.008 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.010+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.008 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.010+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1771936713}) 2024-09-16T05:24:44.008 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.010+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872223695}) 2024-09-16T05:24:44.010 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.010+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.010 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.010+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1828802550}) 2024-09-16T05:24:44.010 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.010+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792692032}) 2024-09-16T05:24:44.010 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.010+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.010 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.010+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1828802550}) 2024-09-16T05:24:44.010 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.010+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1559480138}) 2024-09-16T05:24:44.012 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.014+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.012 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.014+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2927700561}) 2024-09-16T05:24:44.012 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.014+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3538106698}) 2024-09-16T05:24:44.012 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.014+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.012 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.014+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2927700561}) 2024-09-16T05:24:44.012 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.014+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2747260326}) 2024-09-16T05:24:44.014 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.014+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.014 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.014+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1341427327}) 2024-09-16T05:24:44.014 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.014+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1783283433}) 2024-09-16T05:24:44.014 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.014+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.014 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.014+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1341427327}) 2024-09-16T05:24:44.014 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.014+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1240585661}) 2024-09-16T05:24:44.016 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.018+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.016 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.018+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446557574}) 2024-09-16T05:24:44.016 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.018+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1787113988}) 2024-09-16T05:24:44.016 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.018+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.016 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.018+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446557574}) 2024-09-16T05:24:44.017 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.018+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2670555262}) 2024-09-16T05:24:44.018 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.018+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.018 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.018+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4113454454}) 2024-09-16T05:24:44.018 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.018+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=400724957}) 2024-09-16T05:24:44.018 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.018+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.018 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.018+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4113454454}) 2024-09-16T05:24:44.018 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.018+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1065139569}) 2024-09-16T05:24:44.020 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.022+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.020 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.022+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876847319}) 2024-09-16T05:24:44.020 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.022+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2703356319}) 2024-09-16T05:24:44.020 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.022+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.020 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.022+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876847319}) 2024-09-16T05:24:44.020 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.022+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2455166166}) 2024-09-16T05:24:44.022 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.022+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.022 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.026+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50672031}) 2024-09-16T05:24:44.022 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.026+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3379039930}) 2024-09-16T05:24:44.022 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.026+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.022 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.026+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50672031}) 2024-09-16T05:24:44.022 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.026+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2237208221}) 2024-09-16T05:24:44.024 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.026+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.024 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.026+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3059006435}) 2024-09-16T05:24:44.024 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.026+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4200158377}) 2024-09-16T05:24:44.024 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.026+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.024 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.026+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3059006435}) 2024-09-16T05:24:44.025 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.026+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3096054922}) 2024-09-16T05:24:44.027 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.030+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.027 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.030+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613404207}) 2024-09-16T05:24:44.027 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.030+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1480827370}) 2024-09-16T05:24:44.027 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.030+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.027 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.030+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613404207}) 2024-09-16T05:24:44.028 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.030+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2020321669}) 2024-09-16T05:24:44.029 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.030+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.029 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.030+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1312161710}) 2024-09-16T05:24:44.029 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.030+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3863232116}) 2024-09-16T05:24:44.029 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.030+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.029 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.030+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1312161710}) 2024-09-16T05:24:44.030 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.030+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3749283626}) 2024-09-16T05:24:44.031 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.034+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.031 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.034+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434980151}) 2024-09-16T05:24:44.031 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.034+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2279915598}) 2024-09-16T05:24:44.031 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.034+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.031 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.034+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434980151}) 2024-09-16T05:24:44.031 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.034+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2419776826}) 2024-09-16T05:24:44.032 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.034+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.033 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.034+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1612899070}) 2024-09-16T05:24:44.033 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.034+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1141602347}) 2024-09-16T05:24:44.033 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.034+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.033 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.034+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1612899070}) 2024-09-16T05:24:44.033 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.034+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1669330935}) 2024-09-16T05:24:44.034 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.038+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.035 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.038+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1829466208}) 2024-09-16T05:24:44.035 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.038+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2622232863}) 2024-09-16T05:24:44.035 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.038+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.035 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.038+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1829466208}) 2024-09-16T05:24:44.035 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.038+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3783447479}) 2024-09-16T05:24:44.037 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.038+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.037 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.038+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1530079349}) 2024-09-16T05:24:44.037 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.038+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2953680301}) 2024-09-16T05:24:44.037 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.038+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.037 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.038+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1530079349}) 2024-09-16T05:24:44.037 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.038+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3545319236}) 2024-09-16T05:24:44.039 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.042+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.039 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.042+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335543354}) 2024-09-16T05:24:44.039 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.042+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2551555003}) 2024-09-16T05:24:44.039 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.042+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.039 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.042+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335543354}) 2024-09-16T05:24:44.039 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.042+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1748676627}) 2024-09-16T05:24:44.041 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.042+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.041 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.042+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142842944}) 2024-09-16T05:24:44.041 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.042+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4227899092}) 2024-09-16T05:24:44.041 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.042+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.041 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.042+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142842944}) 2024-09-16T05:24:44.041 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.042+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3976396113}) 2024-09-16T05:24:44.042 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.046+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.042 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.046+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2218716931}) 2024-09-16T05:24:44.043 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.046+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2505073529}) 2024-09-16T05:24:44.043 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.046+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.043 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.046+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2218716931}) 2024-09-16T05:24:44.043 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.046+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1012371340}) 2024-09-16T05:24:44.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.046+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.046+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3704176247}) 2024-09-16T05:24:44.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.046+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=438103873}) 2024-09-16T05:24:44.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.046+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.046+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3704176247}) 2024-09-16T05:24:44.046 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.046+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=10419956}) 2024-09-16T05:24:44.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.050+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.050+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3934673830}) 2024-09-16T05:24:44.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.050+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2557463727}) 2024-09-16T05:24:44.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.050+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.048 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.050+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3934673830}) 2024-09-16T05:24:44.049 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.050+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1506098763}) 2024-09-16T05:24:44.051 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.054+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.051 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.054+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3430812903}) 2024-09-16T05:24:44.051 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.054+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2314011649}) 2024-09-16T05:24:44.051 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.054+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.051 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.054+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3430812903}) 2024-09-16T05:24:44.052 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.054+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3020013160}) 2024-09-16T05:24:44.052 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.054+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.053 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.054+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2796674908}) 2024-09-16T05:24:44.053 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.054+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2722840458}) 2024-09-16T05:24:44.053 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.054+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.053 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.054+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2796674908}) 2024-09-16T05:24:44.053 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.054+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2698022296}) 2024-09-16T05:24:44.054 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.058+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.054 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.058+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3521812044}) 2024-09-16T05:24:44.055 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.058+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4186312805}) 2024-09-16T05:24:44.055 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.058+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.055 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.058+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3521812044}) 2024-09-16T05:24:44.056 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.058+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=45329477}) 2024-09-16T05:24:44.057 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.058+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.057 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.058+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245756552}) 2024-09-16T05:24:44.057 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.058+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1265323461}) 2024-09-16T05:24:44.057 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.058+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.058 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.058+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245756552}) 2024-09-16T05:24:44.058 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.058+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3514317340}) 2024-09-16T05:24:44.058 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.062+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.059 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.062+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044671690}) 2024-09-16T05:24:44.059 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.062+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=550891947}) 2024-09-16T05:24:44.059 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.062+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.059 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.062+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044671690}) 2024-09-16T05:24:44.059 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.062+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2250964755}) 2024-09-16T05:24:44.060 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.062+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.060 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.062+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606642103}) 2024-09-16T05:24:44.061 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.062+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3974907935}) 2024-09-16T05:24:44.061 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.062+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.061 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.062+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606642103}) 2024-09-16T05:24:44.061 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.062+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2515872955}) 2024-09-16T05:24:44.062 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.066+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.062 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.066+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=800737420}) 2024-09-16T05:24:44.062 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.066+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2481174524}) 2024-09-16T05:24:44.062 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.066+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.063 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.066+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=800737420}) 2024-09-16T05:24:44.063 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.066+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=250560307}) 2024-09-16T05:24:44.064 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.066+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.064 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.066+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1970941823}) 2024-09-16T05:24:44.064 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.066+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3753885698}) 2024-09-16T05:24:44.064 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.066+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.066+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1970941823}) 2024-09-16T05:24:44.065 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.066+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3063820577}) 2024-09-16T05:24:44.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.070+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.066 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.070+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=342913692}) 2024-09-16T05:24:44.067 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.070+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918763532}) 2024-09-16T05:24:44.067 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.070+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.067 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.070+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=342913692}) 2024-09-16T05:24:44.067 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.070+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2878650230}) 2024-09-16T05:24:44.068 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.070+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.068 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.070+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3487356491}) 2024-09-16T05:24:44.069 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.070+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1877793857}) 2024-09-16T05:24:44.069 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.070+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.069 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.070+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3487356491}) 2024-09-16T05:24:44.069 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.070+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=729420618}) 2024-09-16T05:24:44.071 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.074+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.071 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.074+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481975380}) 2024-09-16T05:24:44.072 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.074+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3312993098}) 2024-09-16T05:24:44.072 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.074+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.072 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.074+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481975380}) 2024-09-16T05:24:44.072 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.074+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3360877466}) 2024-09-16T05:24:44.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.074+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.074+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4036139926}) 2024-09-16T05:24:44.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.074+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2279445132}) 2024-09-16T05:24:44.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.074+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.074+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4036139926}) 2024-09-16T05:24:44.073 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.074+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=164405501}) 2024-09-16T05:24:44.075 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.078+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.075 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.078+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441596503}) 2024-09-16T05:24:44.075 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.078+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1428420422}) 2024-09-16T05:24:44.075 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.078+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.075 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.078+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441596503}) 2024-09-16T05:24:44.075 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.078+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4061299295}) 2024-09-16T05:24:44.077 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.078+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.077 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.078+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115119544}) 2024-09-16T05:24:44.077 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.078+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=548908959}) 2024-09-16T05:24:44.077 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.078+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.077 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.078+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115119544}) 2024-09-16T05:24:44.077 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.078+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1520263614}) 2024-09-16T05:24:44.079 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.082+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.079 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.082+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=253564293}) 2024-09-16T05:24:44.079 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.082+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3108882118}) 2024-09-16T05:24:44.080 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.082+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.080 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.082+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=253564293}) 2024-09-16T05:24:44.080 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.082+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3644661737}) 2024-09-16T05:24:44.081 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.082+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.082 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.082+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370830357}) 2024-09-16T05:24:44.082 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.082+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3924182345}) 2024-09-16T05:24:44.082 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.082+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.082 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.082+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370830357}) 2024-09-16T05:24:44.082 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.082+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1941032643}) 2024-09-16T05:24:44.083 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.086+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.083 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.086+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391579338}) 2024-09-16T05:24:44.083 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.086+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1389977068}) 2024-09-16T05:24:44.083 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.086+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.084 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.086+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391579338}) 2024-09-16T05:24:44.084 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.086+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1044638827}) 2024-09-16T05:24:44.085 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.086+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.085 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.086+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2490889084}) 2024-09-16T05:24:44.086 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.086+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1451935578}) 2024-09-16T05:24:44.086 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.086+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.086 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.086+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2490889084}) 2024-09-16T05:24:44.086 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.086+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4141435292}) 2024-09-16T05:24:44.088 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.090+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.088 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.090+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=21,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_snap=28} crc {logm=926401047,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=3497973547,mon_config_key=1589051853,monmap=3045472550,osd_metadata=487805116,osd_snap=776600368}) 2024-09-16T05:24:44.088 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.090+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=23,mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27} crc {logm=3168481699,mdsmap=2923618896,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=3497973547,mon_config_key=1589051853,monmap=3045472550,osd_metadata=487805116,osd_pg_creating=2600473707,osd_snap=301980072}) 2024-09-16T05:24:44.089 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.090+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.089 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.090+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=21,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_snap=28} crc {logm=926401047,mdsmap=2485929294,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=3497973547,mon_config_key=1589051853,monmap=3045472550,osd_metadata=487805116,osd_snap=776600368}) 2024-09-16T05:24:44.089 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.090+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=19,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31} crc {logm=1264790380,mdsmap=1078762840,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=3497973547,mon_config_key=971488657,monmap=3045472550,osd_metadata=487805116,osd_pg_creating=2600473707,osd_snap=3146476772}) 2024-09-16T05:24:44.091 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.094+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.092 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.094+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=3143929636,osdmap=475488454}) 2024-09-16T05:24:44.092 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.094+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=1086128581,osdmap=70558722}) 2024-09-16T05:24:44.092 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.094+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.092 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.094+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=3143929636,osdmap=475488454}) 2024-09-16T05:24:44.092 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.094+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=3143929636,osdmap=260754058}) 2024-09-16T05:24:44.094 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.094+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.094 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.094+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2748925590}) 2024-09-16T05:24:44.094 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.094+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3152567946}) 2024-09-16T05:24:44.094 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.094+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.094 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.094+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2748925590}) 2024-09-16T05:24:44.094 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.094+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3566029931}) 2024-09-16T05:24:44.097 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.098+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.097 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.098+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3403071281}) 2024-09-16T05:24:44.097 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.098+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2506345647}) 2024-09-16T05:24:44.097 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.098+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.097 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.098+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3403071281}) 2024-09-16T05:24:44.097 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.098+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1676078689}) 2024-09-16T05:24:44.101 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.102+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.101 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.102+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3826814203}) 2024-09-16T05:24:44.101 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.102+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=226062801}) 2024-09-16T05:24:44.101 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.102+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.101 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.102+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3826814203}) 2024-09-16T05:24:44.101 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.102+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2452193551}) 2024-09-16T05:24:44.102 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.102+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.102 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.102+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1755998547}) 2024-09-16T05:24:44.102 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.102+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3533837340}) 2024-09-16T05:24:44.102 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.102+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.102 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.102+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1755998547}) 2024-09-16T05:24:44.102 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.102+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1919631845}) 2024-09-16T05:24:44.104 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.106+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.104 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.106+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1282972494}) 2024-09-16T05:24:44.104 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.106+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3142642300}) 2024-09-16T05:24:44.104 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.106+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.104 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.106+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1282972494}) 2024-09-16T05:24:44.104 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.106+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=440481108}) 2024-09-16T05:24:44.107 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.110+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.107 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.110+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1916917871}) 2024-09-16T05:24:44.108 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.110+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1777143177}) 2024-09-16T05:24:44.108 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.110+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.108 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.110+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1916917871}) 2024-09-16T05:24:44.109 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.110+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3696961082}) 2024-09-16T05:24:44.112 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.114+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.112 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.114+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=815226448}) 2024-09-16T05:24:44.112 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.114+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2741037612}) 2024-09-16T05:24:44.112 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.114+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.112 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.114+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=815226448}) 2024-09-16T05:24:44.112 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.114+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1200402867}) 2024-09-16T05:24:44.116 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.118+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.116 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.118+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3751020239}) 2024-09-16T05:24:44.116 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.118+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3918145959}) 2024-09-16T05:24:44.116 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.118+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.116 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.118+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3751020239}) 2024-09-16T05:24:44.117 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.118+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=979952579}) 2024-09-16T05:24:44.121 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.122+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.121 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.122+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2073716102}) 2024-09-16T05:24:44.121 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.122+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3902461968}) 2024-09-16T05:24:44.121 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.122+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:24:44.121 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.122+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2073716102}) 2024-09-16T05:24:44.121 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:24:44.122+0000 7f64ea3b7640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=244808706}) 2024-09-16T05:24:44.966 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:24:44.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:24:45.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 10 monclient: get_monmap_and_config 2024-09-16T05:24:45.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 10 monclient: build_initial_monmap 2024-09-16T05:24:45.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 10 monclient: monmap: 2024-09-16T05:24:45.096 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:45.096 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.096 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:24:45.102002+0000 2024-09-16T05:24:45.096 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:24:45.102002+0000 2024-09-16T05:24:45.096 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:45.097 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.097 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:24:45.097 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:24:45.097 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:24:45.097 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:24:45.097 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:24:45.097 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 1 Processor -- start 2024-09-16T05:24:45.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 1 -- start start 2024-09-16T05:24:45.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 10 monclient: init 2024-09-16T05:24:45.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:45.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-16T05:24:45.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffa34137480 addr v1:172.21.15.149:6789/0 2024-09-16T05:24:45.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa340a57b0 addr v1:172.21.15.114:6789/0 2024-09-16T05:24:45.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ffa3413da70 0x7ffa3413fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 10 monclient(hunting): picked mon.noname-c con 0x7ffa3413da70 addr v2:172.21.15.149:3300/0 2024-09-16T05:24:45.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa3409cfa0 con 0x7ffa340a57b0 2024-09-16T05:24:45.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa340a52f0 con 0x7ffa34137480 2024-09-16T05:24:45.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7ffa34137f70 con 0x7ffa3413da70 2024-09-16T05:24:45.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:45.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3bd23640 10 monclient(hunting): authenticate will time out at 1969.731357s 2024-09-16T05:24:45.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa392be640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ffa340a57b0 0x7ffa3413c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37982/0 (socket says 172.21.15.114:37982) 2024-09-16T05:24:45.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa392be640 1 -- 172.21.15.114:0/2223233734 learned_addr learned my addr 172.21.15.114:0/2223233734 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:24:45.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3a2c0640 1 --2- 172.21.15.114:0/2223233734 >> v2:172.21.15.149:3300/0 conn(0x7ffa3413da70 0x7ffa3413fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa38abd640 1 -- 172.21.15.114:0/2223233734 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 327748163 0 0) 0x7ffa30001120 con 0x7ffa340a57b0 2024-09-16T05:24:45.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa38abd640 10 monclient(hunting): _negotiate 2024-09-16T05:24:45.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa38abd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa38abd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa38abd640 10 monclient(hunting): my global_id is 74796 2024-09-16T05:24:45.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa38abd640 1 -- 172.21.15.114:0/2223233734 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa20003530 con 0x7ffa340a57b0 2024-09-16T05:24:45.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3a2c0640 10 monclient(hunting): get_auth_request con 0x7ffa3413da70 auth_method 0 2024-09-16T05:24:45.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3a2c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3a2c0640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.098+0000 7ffa3a2c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3a2c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3a2c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3a2c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 -- 172.21.15.114:0/2223233734 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2076133734 0 0) 0x7ffa20003530 con 0x7ffa340a57b0 2024-09-16T05:24:45.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 -- 172.21.15.114:0/2223233734 >> v2:172.21.15.149:3300/0 conn(0x7ffa3413da70 msgr2=0x7ffa3413fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:45.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 --2- 172.21.15.114:0/2223233734 >> v2:172.21.15.149:3300/0 conn(0x7ffa3413da70 0x7ffa3413fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 -- 172.21.15.114:0/2223233734 >> v1:172.21.15.149:6789/0 conn(0x7ffa34137480 legacy=0x7ffa34137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: _finish_hunting 0 2024-09-16T05:24:45.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:24:45.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:45.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:24:45.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 -- 172.21.15.114:0/2223233734 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa340a5570 con 0x7ffa340a57b0 2024-09-16T05:24:45.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: _finish_auth 0 2024-09-16T05:24:45.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: _check_auth_tickets 2024-09-16T05:24:45.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:45.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 -- 172.21.15.114:0/2223233734 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7ffa30001850 con 0x7ffa340a57b0 2024-09-16T05:24:45.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:24:45.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: dump: 2024-09-16T05:24:45.104 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.104 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.104 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.104 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.104 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.104 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.104 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.104 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.104 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.104 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:24:45.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:45.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 -- 172.21.15.114:0/2223233734 >> v1:172.21.15.114:6789/0 conn(0x7ffa340a57b0 legacy=0x7ffa3413c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:24:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 --2- 172.21.15.114:0/2223233734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa200039c0 0x7ffa20003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient(hunting): picked mon.a con 0x7ffa200039c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 --2- 172.21.15.114:0/2223233734 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa20004810 0x7ffa20004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient(hunting): picked mon.c con 0x7ffa20004810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 --2- 172.21.15.114:0/2223233734 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa200052f0 0x7ffa2000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient(hunting): picked mon.b con 0x7ffa200052f0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:24:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 -- 172.21.15.114:0/2223233734 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffa34137f70 con 0x7ffa200039c0 2024-09-16T05:24:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 -- 172.21.15.114:0/2223233734 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2000c1c0 con 0x7ffa200052f0 2024-09-16T05:24:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 -- 172.21.15.114:0/2223233734 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ffa2000c3c0 con 0x7ffa20004810 2024-09-16T05:24:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:24:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 1 --2- 172.21.15.114:0/2223233734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa200039c0 0x7ffa20003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa392be640 1 --2- 172.21.15.114:0/2223233734 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa20004810 0x7ffa20004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 10 monclient(hunting): get_auth_request con 0x7ffa200039c0 auth_method 0 2024-09-16T05:24:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:24:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa392be640 10 monclient(hunting): get_auth_request con 0x7ffa20004810 auth_method 0 2024-09-16T05:24:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa392be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa392be640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa392be640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:24:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:24:45.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa392be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa392be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa392be640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:24:45.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3a2c0640 1 --2- 172.21.15.114:0/2223233734 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa200052f0 0x7ffa2000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 10 monclient(hunting): handle_auth_done global_id 74796 payload 293 2024-09-16T05:24:45.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 1 -- 172.21.15.114:0/2223233734 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa20004810 msgr2=0x7ffa20004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:45.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 1 --2- 172.21.15.114:0/2223233734 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa20004810 0x7ffa20004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 1 -- 172.21.15.114:0/2223233734 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa200052f0 msgr2=0x7ffa2000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:45.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 1 --2- 172.21.15.114:0/2223233734 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa200052f0 0x7ffa2000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 10 monclient: _finish_hunting 0 2024-09-16T05:24:45.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 1 monclient: found mon.a 2024-09-16T05:24:45.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:45.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:24:45.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 1 -- 172.21.15.114:0/2223233734 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa20003530 con 0x7ffa200039c0 2024-09-16T05:24:45.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 10 monclient: _finish_auth 0 2024-09-16T05:24:45.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 10 monclient: _check_auth_tickets 2024-09-16T05:24:45.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:45.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3a2c0640 10 monclient: get_auth_request con 0x7ffa200052f0 auth_method 0 2024-09-16T05:24:45.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3a2c0640 1 --2- 172.21.15.114:0/2223233734 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa200052f0 0x7ffa2000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:24:45.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3bd23640 5 monclient: authenticate success, global_id 74796 2024-09-16T05:24:45.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3bd23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:24:45.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3a2c0640 1 -- 172.21.15.114:0/2223233734 reap_dead start 2024-09-16T05:24:45.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa39abf640 1 --2- 172.21.15.114:0/2223233734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa200039c0 0x7ffa20003de0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ffa24001750 tx=0x7ffa2401c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca8de56729be7c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 -- 172.21.15.114:0/2223233734 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa240013a0 con 0x7ffa200039c0 2024-09-16T05:24:45.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:24:45.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: dump: 2024-09-16T05:24:45.115 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.115 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.115 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.115 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.115 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.115 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.115 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.115 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.115 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.115 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3bd23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:24:45.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 -- 172.21.15.114:0/2223233734 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa240052f0 con 0x7ffa200039c0 2024-09-16T05:24:45.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:45.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3bd23640 10 monclient: get_monmap_and_config success 2024-09-16T05:24:45.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3bd23640 4 set_mon_vals no callback set 2024-09-16T05:24:45.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 1 -- 172.21.15.114:0/2223233734 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24003840 con 0x7ffa200039c0 2024-09-16T05:24:45.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:24:45.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa38abd640 10 monclient: dump: 2024-09-16T05:24:45.116 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.116 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.116 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.116 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.116 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.116 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.117 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.117 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.117 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.117 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3bd23640 10 monclient: shutdown 2024-09-16T05:24:45.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3bd23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:45.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3bd23640 1 -- 172.21.15.114:0/2223233734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa200039c0 msgr2=0x7ffa20003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3bd23640 1 --2- 172.21.15.114:0/2223233734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa200039c0 0x7ffa20003de0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ffa24001750 tx=0x7ffa2401c370 comp rx=0 tx=0).stop 2024-09-16T05:24:45.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3bd23640 1 -- 172.21.15.114:0/2223233734 shutdown_connections 2024-09-16T05:24:45.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3bd23640 1 --2- 172.21.15.114:0/2223233734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa200039c0 0x7ffa20003de0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3bd23640 1 -- 172.21.15.114:0/2223233734 >> 172.21.15.114:0/2223233734 conn(0x7ffa340a9c40 msgr2=0x7ffa340aa050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:45.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3bd23640 1 -- 172.21.15.114:0/2223233734 shutdown_connections 2024-09-16T05:24:45.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.102+0000 7ffa3bd23640 1 -- 172.21.15.114:0/2223233734 wait complete. 2024-09-16T05:24:45.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient: build_initial_monmap 2024-09-16T05:24:45.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient: monmap: 2024-09-16T05:24:45.118 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:45.118 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:24:45.118 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:24:45.118 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:24:45.118 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:45.118 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.118 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:24:45.118 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:24:45.118 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:24:45.118 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 librados: starting msgr at 2024-09-16T05:24:45.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 librados: starting objecter 2024-09-16T05:24:45.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 Processor -- start 2024-09-16T05:24:45.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 -- start start 2024-09-16T05:24:45.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 librados: setting wanted keys 2024-09-16T05:24:45.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 librados: calling monclient init 2024-09-16T05:24:45.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient: init 2024-09-16T05:24:45.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:45.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:24:45.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa200039c0 0x7ffa341cf220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient(hunting): picked mon.noname-c con 0x7ffa200039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:45.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa34136270 0x7ffa3411fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient(hunting): picked mon.noname-a con 0x7ffa34136270 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:45.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa341366b0 0x7ffa34120400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient(hunting): picked mon.noname-b con 0x7ffa341366b0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:24:45.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffa34137df0 con 0x7ffa34136270 2024-09-16T05:24:45.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7ffa34137c70 con 0x7ffa341366b0 2024-09-16T05:24:45.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ffa34072910 con 0x7ffa200039c0 2024-09-16T05:24:45.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:45.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient(hunting): authenticate will time out at 1969.737050s 2024-09-16T05:24:45.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa200039c0 0x7ffa341cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa34136270 0x7ffa3411fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa200039c0 0x7ffa341cf220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56376/0 (socket says 172.21.15.114:56376) 2024-09-16T05:24:45.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa34136270 0x7ffa3411fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40196/0 (socket says 172.21.15.114:40196) 2024-09-16T05:24:45.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 1 -- 172.21.15.114:0/2921232459 learned_addr learned my addr 172.21.15.114:0/2921232459 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:45.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient(hunting): get_auth_request con 0x7ffa200039c0 auth_method 0 2024-09-16T05:24:45.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient(hunting): get_auth_request con 0x7ffa34136270 auth_method 0 2024-09-16T05:24:45.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3a2c0640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa341366b0 0x7ffa34120400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient(hunting): handle_auth_done global_id 74799 payload 824 2024-09-16T05:24:45.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 1 -- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa200039c0 msgr2=0x7ffa341cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa200039c0 0x7ffa341cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 1 -- 172.21.15.114:0/2921232459 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa341366b0 msgr2=0x7ffa34120400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa341366b0 0x7ffa34120400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient: _finish_hunting 0 2024-09-16T05:24:45.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 1 monclient: found mon.noname-a 2024-09-16T05:24:45.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:45.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:24:45.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 1 -- 172.21.15.114:0/2921232459 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa341d0130 con 0x7ffa34136270 2024-09-16T05:24:45.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3a2c0640 10 monclient: get_auth_request con 0x7ffa341366b0 auth_method 0 2024-09-16T05:24:45.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa200039c0 0x7ffa341cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:24:45.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3a2c0640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa341366b0 0x7ffa34120400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:24:45.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa34136270 0x7ffa3411fec0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ffa30000c10 tx=0x7ffa30004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 client.?.objecter ms_handle_connect 0x7ffa34136270 2024-09-16T05:24:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:24:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 1 -- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa34136270 msgr2=0x7ffa3411fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:24:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 1 -- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa34136270 msgr2=0x7ffa3411fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:24:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa34136270 0x7ffa3411fec0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ffa30000c10 tx=0x7ffa30004f90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:24:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa34136270 0x7ffa3411fec0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ffa30000c10 tx=0x7ffa30004f90 comp rx=0 tx=0).stop 2024-09-16T05:24:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 1 -- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa34136270 msgr2=0x7ffa3411fec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:24:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa34136270 0x7ffa3411fec0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:24:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa200039c0 0x7ffa18006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7ffa200039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa18007840 0x7ffa18009cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7ffa18007840 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa341366b0 0x7ffa1800c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7ffa341366b0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:24:45.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 1 -- 172.21.15.114:0/2921232459 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1800d1d0 con 0x7ffa18007840 2024-09-16T05:24:45.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 1 -- 172.21.15.114:0/2921232459 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1800d380 con 0x7ffa341366b0 2024-09-16T05:24:45.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 1 -- 172.21.15.114:0/2921232459 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ffa1800d580 con 0x7ffa200039c0 2024-09-16T05:24:45.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:45.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa200039c0 0x7ffa18006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa18007840 0x7ffa18009cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient(hunting): get_auth_request con 0x7ffa200039c0 auth_method 0 2024-09-16T05:24:45.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:24:45.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient(hunting): get_auth_request con 0x7ffa18007840 auth_method 0 2024-09-16T05:24:45.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:24:45.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:24:45.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:24:45.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient(hunting): handle_auth_done global_id 74799 payload 293 2024-09-16T05:24:45.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 1 -- 172.21.15.114:0/2921232459 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa341366b0 msgr2=0x7ffa1800c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa341366b0 0x7ffa1800c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 1 -- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa18007840 msgr2=0x7ffa18009cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa18007840 0x7ffa18009cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient: _finish_hunting 0 2024-09-16T05:24:45.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 1 monclient: found mon.noname-c 2024-09-16T05:24:45.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:45.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:24:45.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 1 -- 172.21.15.114:0/2921232459 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1800d840 con 0x7ffa200039c0 2024-09-16T05:24:45.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa392be640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa18007840 0x7ffa18009cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:24:45.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 client.?.objecter ms_handle_connect 0x7ffa200039c0 2024-09-16T05:24:45.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:24:45.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa39abf640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa200039c0 0x7ffa18006e50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffa2401c1a0 tx=0x7ffa30001cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 1 -- 172.21.15.114:0/2921232459 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa240b9070 con 0x7ffa200039c0 2024-09-16T05:24:45.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:24:45.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient: dump: 2024-09-16T05:24:45.129 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.129 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.129 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.129 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.129 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.129 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.129 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.129 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.129 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.129 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:24:45.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:24:45.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:45.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 5 monclient: authenticate success, global_id 74799 2024-09-16T05:24:45.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient: _renew_subs 2024-09-16T05:24:45.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 1 -- 172.21.15.114:0/2921232459 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa240039b0 con 0x7ffa200039c0 2024-09-16T05:24:45.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:45.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 -- 172.21.15.114:0/2921232459 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa34123670 con 0x7ffa200039c0 2024-09-16T05:24:45.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:45.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 1 -- 172.21.15.114:0/2921232459 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24007a40 con 0x7ffa200039c0 2024-09-16T05:24:45.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:45.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 monclient: dump: 2024-09-16T05:24:45.130 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.130 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.130 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.130 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.131 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.131 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.131 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.131 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.131 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.131 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2bfff640 4 set_mon_vals no callback set 2024-09-16T05:24:45.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 client.74799.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:24:45.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient: _renew_subs 2024-09-16T05:24:45.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:45.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 -- 172.21.15.114:0/2921232459 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa34123950 con 0x7ffa200039c0 2024-09-16T05:24:45.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 librados: init done 2024-09-16T05:24:45.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 1 -- 172.21.15.114:0/2921232459 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ffa240bc070 con 0x7ffa200039c0 2024-09-16T05:24:45.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:45.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:45.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:45.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa3bd23640 1 -- 172.21.15.114:0/2921232459 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa3413ac00 con 0x7ffa200039c0 2024-09-16T05:24:45.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 1 --2- 172.21.15.114:0/2921232459 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffa34136270 0x7ffa18041c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 1 -- 172.21.15.114:0/2921232459 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(689..689 src has 1..689) ==== 5592+0+0 (secure 0 0 0) 0x7ffa240e2960 con 0x7ffa200039c0 2024-09-16T05:24:45.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 10 client.74799.objecter ms_dispatch 0x7ffa34000ba0 osd_map(689..689 src has 1..689) 2024-09-16T05:24:45.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 3 client.74799.objecter handle_osd_map got epochs [689,689] > 0 2024-09-16T05:24:45.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.106+0000 7ffa2a7fc640 3 client.74799.objecter handle_osd_map decoding full epoch 689 2024-09-16T05:24:45.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.110+0000 7ffa2a7fc640 20 client.74799.objecter dump_active .. 0 homeless 2024-09-16T05:24:45.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.110+0000 7ffa392be640 1 --2- 172.21.15.114:0/2921232459 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffa34136270 0x7ffa18041c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.110+0000 7ffa392be640 10 monclient: get_auth_request con 0x7ffa34136270 auth_method 0 2024-09-16T05:24:45.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.110+0000 7ffa2a7fc640 10 client.74799.objecter ms_handle_connect 0x7ffa34136270 2024-09-16T05:24:45.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.110+0000 7ffa392be640 1 --2- 172.21.15.114:0/2921232459 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffa34136270 0x7ffa18041c50 secure :-1 s=READY pgs=3603 cs=0 l=1 rev1=1 crypto rx=0x7ffa300019c0 tx=0x7ffa30022040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.114+0000 7ffa2a7fc640 1 -- 172.21.15.114:0/2921232459 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ffa240c5070 con 0x7ffa200039c0 2024-09-16T05:24:45.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.114+0000 7ffa2a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:45.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.114+0000 7ffa2a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:24:45.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.354+0000 7ffa3bd23640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:24:45.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.354+0000 7ffa3bd23640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:24:45.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.354+0000 7ffa3bd23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:45.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.354+0000 7ffa3bd23640 1 -- 172.21.15.114:0/2921232459 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffa340a4a80 con 0x7ffa200039c0 2024-09-16T05:24:45.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.354+0000 7ffa2a7fc640 1 -- 172.21.15.114:0/2921232459 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7ffa240160b0 con 0x7ffa200039c0 2024-09-16T05:24:45.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.354+0000 7ffa2a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:24:45.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.354+0000 7ffa2a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:24:45.352 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:24:45.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 10 librados: watch_flush enter 2024-09-16T05:24:45.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 10 librados: watch_flush exit 2024-09-16T05:24:45.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 20 client.74799.objecter shutdown clearing up homeless session... 2024-09-16T05:24:45.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 10 client.74799.objecter successfully canceled tick 2024-09-16T05:24:45.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 1 -- 172.21.15.114:0/2921232459 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffa34136270 msgr2=0x7ffa18041c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 1 --2- 172.21.15.114:0/2921232459 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffa34136270 0x7ffa18041c50 secure :-1 s=READY pgs=3603 cs=0 l=1 rev1=1 crypto rx=0x7ffa300019c0 tx=0x7ffa30022040 comp rx=0 tx=0).stop 2024-09-16T05:24:45.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 10 monclient: shutdown 2024-09-16T05:24:45.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:45.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 1 -- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa200039c0 msgr2=0x7ffa18006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa200039c0 0x7ffa18006e50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffa2401c1a0 tx=0x7ffa30001cf0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 1 -- 172.21.15.114:0/2921232459 shutdown_connections 2024-09-16T05:24:45.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 1 --2- 172.21.15.114:0/2921232459 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffa34136270 0x7ffa18041c50 unknown :-1 s=CLOSED pgs=3603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa18007840 0x7ffa18009cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa341366b0 0x7ffa1800c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 1 --2- 172.21.15.114:0/2921232459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa200039c0 0x7ffa18006e50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 1 -- 172.21.15.114:0/2921232459 >> 172.21.15.114:0/2921232459 conn(0x7ffa340a9c40 msgr2=0x7ffa340a8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:45.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 1 -- 172.21.15.114:0/2921232459 shutdown_connections 2024-09-16T05:24:45.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 1 -- 172.21.15.114:0/2921232459 wait complete. 2024-09-16T05:24:45.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.358+0000 7ffa3bd23640 1 librados: shutdown 2024-09-16T05:24:45.371 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:24:45.372 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:24:45.372 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:24:45.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:24:45.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 10 monclient: get_monmap_and_config 2024-09-16T05:24:45.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 10 monclient: build_initial_monmap 2024-09-16T05:24:45.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 10 monclient: monmap: 2024-09-16T05:24:45.487 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:45.487 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.487 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:24:45.492322+0000 2024-09-16T05:24:45.487 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:24:45.492322+0000 2024-09-16T05:24:45.487 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:45.487 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.487 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:24:45.487 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:24:45.487 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:24:45.487 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:24:45.487 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:24:45.487 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 1 Processor -- start 2024-09-16T05:24:45.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 1 -- start start 2024-09-16T05:24:45.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 10 monclient: init 2024-09-16T05:24:45.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:45.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-16T05:24:45.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa0240a0ec0 0x7fa02409dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 10 monclient(hunting): picked mon.noname-a con 0x7fa0240a0ec0 addr v2:172.21.15.114:3300/0 2024-09-16T05:24:45.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa02409e000 addr v1:172.21.15.149:6789/0 2024-09-16T05:24:45.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fa024139640 0x7fa02413ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 10 monclient(hunting): picked mon.noname-c con 0x7fa024139640 addr v2:172.21.15.149:3300/0 2024-09-16T05:24:45.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa02409a760 con 0x7fa02409e000 2024-09-16T05:24:45.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa0240994b0 con 0x7fa0240a0ec0 2024-09-16T05:24:45.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fa0240a1870 con 0x7fa024139640 2024-09-16T05:24:45.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:45.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa02b149640 10 monclient(hunting): authenticate will time out at 1970.121580s 2024-09-16T05:24:45.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa0240a0ec0 0x7fa02409dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa0240a0ec0 0x7fa02409dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40214/0 (socket says 172.21.15.114:40214) 2024-09-16T05:24:45.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 1 -- 172.21.15.114:0/2623376763 learned_addr learned my addr 172.21.15.114:0/2623376763 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:45.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 10 monclient(hunting): get_auth_request con 0x7fa0240a0ec0 auth_method 0 2024-09-16T05:24:45.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa0296e6640 1 --2- 172.21.15.114:0/2623376763 >> v2:172.21.15.149:3300/0 conn(0x7fa024139640 0x7fa02413ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 10 monclient(hunting): handle_auth_done global_id 74802 payload 274 2024-09-16T05:24:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 1 -- 172.21.15.114:0/2623376763 >> v2:172.21.15.149:3300/0 conn(0x7fa024139640 msgr2=0x7fa02413ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 1 --2- 172.21.15.114:0/2623376763 >> v2:172.21.15.149:3300/0 conn(0x7fa024139640 0x7fa02413ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 1 -- 172.21.15.114:0/2623376763 >> v1:172.21.15.149:6789/0 conn(0x7fa02409e000 legacy=0x7fa02409e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 10 monclient: _finish_hunting 0 2024-09-16T05:24:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 1 monclient: found mon.noname-a 2024-09-16T05:24:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:45.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:24:45.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa028ee5640 1 -- 172.21.15.114:0/2623376763 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa024099790 con 0x7fa0240a0ec0 2024-09-16T05:24:45.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa0296e6640 10 monclient: get_auth_request con 0x7fa024139640 auth_method 0 2024-09-16T05:24:45.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.490+0000 7fa0296e6640 1 --2- 172.21.15.114:0/2623376763 >> v2:172.21.15.149:3300/0 conn(0x7fa024139640 0x7fa02413ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:24:45.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa028ee5640 1 --2- 172.21.15.114:0/2623376763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240a0ec0 0x7fa02409dac0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa0240a2fd0 tx=0x7fa01401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ce137d618e56fd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0237fe640 1 -- 172.21.15.114:0/2623376763 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0140b0070 con 0x7fa0240a0ec0 2024-09-16T05:24:45.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:24:45.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0237fe640 10 monclient: dump: 2024-09-16T05:24:45.500 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.500 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.500 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.500 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.500 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.500 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.500 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.500 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.500 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.501 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0237fe640 10 monclient: _finish_auth 0 2024-09-16T05:24:45.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0237fe640 10 monclient: _check_auth_tickets 2024-09-16T05:24:45.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:45.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 5 monclient: authenticate success, global_id 74802 2024-09-16T05:24:45.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:24:45.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0237fe640 1 -- 172.21.15.114:0/2623376763 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0140ab600 con 0x7fa0240a0ec0 2024-09-16T05:24:45.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0237fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:45.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient: get_monmap_and_config success 2024-09-16T05:24:45.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0237fe640 1 -- 172.21.15.114:0/2623376763 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0140ab9d0 con 0x7fa0240a0ec0 2024-09-16T05:24:45.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 4 set_mon_vals no callback set 2024-09-16T05:24:45.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:24:45.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0237fe640 10 monclient: dump: 2024-09-16T05:24:45.502 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.502 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.502 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.502 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.502 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.502 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.502 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.502 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.502 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.502 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient: shutdown 2024-09-16T05:24:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 -- 172.21.15.114:0/2623376763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240a0ec0 msgr2=0x7fa02409dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 --2- 172.21.15.114:0/2623376763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240a0ec0 0x7fa02409dac0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa0240a2fd0 tx=0x7fa01401c410 comp rx=0 tx=0).stop 2024-09-16T05:24:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 -- 172.21.15.114:0/2623376763 shutdown_connections 2024-09-16T05:24:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 --2- 172.21.15.114:0/2623376763 >> v2:172.21.15.149:3300/0 conn(0x7fa024139640 0x7fa02413ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 --2- 172.21.15.114:0/2623376763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240a0ec0 0x7fa02409dac0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 -- 172.21.15.114:0/2623376763 >> 172.21.15.114:0/2623376763 conn(0x7fa0240a1fb0 msgr2=0x7fa0240a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 -- 172.21.15.114:0/2623376763 shutdown_connections 2024-09-16T05:24:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 -- 172.21.15.114:0/2623376763 wait complete. 2024-09-16T05:24:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient: build_initial_monmap 2024-09-16T05:24:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient: monmap: 2024-09-16T05:24:45.504 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:45.504 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:24:45.504 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:24:45.504 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:24:45.504 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:45.504 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.504 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:24:45.504 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:24:45.504 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:24:45.504 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 librados: starting msgr at 2024-09-16T05:24:45.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 librados: starting objecter 2024-09-16T05:24:45.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 Processor -- start 2024-09-16T05:24:45.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 -- start start 2024-09-16T05:24:45.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 librados: setting wanted keys 2024-09-16T05:24:45.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 librados: calling monclient init 2024-09-16T05:24:45.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient: init 2024-09-16T05:24:45.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:45.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:24:45.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa02409e000 0x7fa024138880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient(hunting): picked mon.noname-a con 0x7fa02409e000 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:45.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa0240a0ec0 0x7fa024138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient(hunting): picked mon.noname-b con 0x7fa0240a0ec0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:24:45.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa024139640 0x7fa02408f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient(hunting): picked mon.noname-c con 0x7fa024139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:45.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa02413e910 con 0x7fa02409e000 2024-09-16T05:24:45.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fa02413e790 con 0x7fa0240a0ec0 2024-09-16T05:24:45.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa02413ea90 con 0x7fa024139640 2024-09-16T05:24:45.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:45.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient(hunting): authenticate will time out at 1970.126552s 2024-09-16T05:24:45.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa028ee5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa02409e000 0x7fa024138880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa024139640 0x7fa02408f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa028ee5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa02409e000 0x7fa024138880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40218/0 (socket says 172.21.15.114:40218) 2024-09-16T05:24:45.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa028ee5640 1 -- 172.21.15.114:0/1394263457 learned_addr learned my addr 172.21.15.114:0/1394263457 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:45.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa028ee5640 10 monclient(hunting): get_auth_request con 0x7fa02409e000 auth_method 0 2024-09-16T05:24:45.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa028ee5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa028ee5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa028ee5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 10 monclient(hunting): get_auth_request con 0x7fa024139640 auth_method 0 2024-09-16T05:24:45.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa028ee5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa028ee5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa028ee5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa023fff640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa0240a0ec0 0x7fa024138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 10 monclient(hunting): handle_auth_done global_id 104288 payload 824 2024-09-16T05:24:45.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 1 -- 172.21.15.114:0/1394263457 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa0240a0ec0 msgr2=0x7fa024138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa0240a0ec0 0x7fa024138dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 1 -- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa02409e000 msgr2=0x7fa024138880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa02409e000 0x7fa024138880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 10 monclient: _finish_hunting 0 2024-09-16T05:24:45.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 1 monclient: found mon.noname-c 2024-09-16T05:24:45.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:45.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:24:45.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 1 -- 172.21.15.114:0/1394263457 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa024093270 con 0x7fa024139640 2024-09-16T05:24:45.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa023fff640 10 monclient: get_auth_request con 0x7fa0240a0ec0 auth_method 0 2024-09-16T05:24:45.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa028ee5640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa02409e000 0x7fa024138880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:24:45.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa023fff640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa0240a0ec0 0x7fa024138dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:24:45.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0217fa640 10 client.?.objecter ms_handle_connect 0x7fa024139640 2024-09-16T05:24:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0217fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:24:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0296e6640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa024139640 0x7fa02408f7f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa01c010300 tx=0x7fa01c0107c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0217fa640 1 -- 172.21.15.114:0/1394263457 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c00e2e0 con 0x7fa024139640 2024-09-16T05:24:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0217fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:24:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0217fa640 10 monclient: dump: 2024-09-16T05:24:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.513 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.513 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.513 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.513 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.513 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0217fa640 10 monclient: _finish_auth 0 2024-09-16T05:24:45.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0217fa640 10 monclient: _check_auth_tickets 2024-09-16T05:24:45.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:45.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0217fa640 1 -- 172.21.15.114:0/1394263457 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa01c00e480 con 0x7fa024139640 2024-09-16T05:24:45.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0217fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:45.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0217fa640 1 -- 172.21.15.114:0/1394263457 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c00c970 con 0x7fa024139640 2024-09-16T05:24:45.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 5 monclient: authenticate success, global_id 104288 2024-09-16T05:24:45.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient: _renew_subs 2024-09-16T05:24:45.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:45.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 -- 172.21.15.114:0/1394263457 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa024091700 con 0x7fa024139640 2024-09-16T05:24:45.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0227fc640 4 set_mon_vals no callback set 2024-09-16T05:24:45.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:45.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa0217fa640 10 monclient: dump: 2024-09-16T05:24:45.514 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.514 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.514 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.514 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.514 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.514 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.514 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.515 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.515 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.515 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 client.104288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:24:45.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient: _renew_subs 2024-09-16T05:24:45.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:45.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 -- 172.21.15.114:0/1394263457 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa02409b2b0 con 0x7fa024139640 2024-09-16T05:24:45.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.494+0000 7fa02b149640 1 librados: init done 2024-09-16T05:24:45.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa02b149640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:45.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa02b149640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240904f0 0x7fa02402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa02b149640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa02b149640 1 -- 172.21.15.114:0/1394263457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa024091fe0 con 0x7fa0240904f0 2024-09-16T05:24:45.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa028ee5640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240904f0 0x7fa02402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa028ee5640 10 monclient: get_auth_request con 0x7fa0240904f0 auth_method 0 2024-09-16T05:24:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa028ee5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa028ee5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa028ee5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa028ee5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa028ee5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 1 -- 172.21.15.114:0/1394263457 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa01c00cb10 con 0x7fa024139640 2024-09-16T05:24:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa028ee5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa028ee5640 10 monclient(hunting): handle_auth_done global_id 74808 payload 824 2024-09-16T05:24:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa028ee5640 1 -- 172.21.15.114:0/1394263457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa02413ba30 con 0x7fa0240904f0 2024-09-16T05:24:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 1 --2- 172.21.15.114:0/1394263457 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa010037840 0x7fa010039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 1 -- 172.21.15.114:0/1394263457 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(690..690 src has 1..690) ==== 5965+0+0 (secure 0 0 0) 0x7fa01c04d870 con 0x7fa024139640 2024-09-16T05:24:45.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 10 client.104288.objecter ms_dispatch 0x7fa024000ba0 osd_map(690..690 src has 1..690) 2024-09-16T05:24:45.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 3 client.104288.objecter handle_osd_map got epochs [690,690] > 0 2024-09-16T05:24:45.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 3 client.104288.objecter handle_osd_map decoding full epoch 690 2024-09-16T05:24:45.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 20 client.104288.objecter dump_active .. 0 homeless 2024-09-16T05:24:45.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 10 client.104288.objecter ms_handle_connect 0x7fa0240904f0 2024-09-16T05:24:45.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 10 client.104288.objecter resend_mon_ops 2024-09-16T05:24:45.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa028ee5640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240904f0 0x7fa02402c650 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa01401a040 tx=0x7fa01401ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 1 -- 172.21.15.114:0/1394263457 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0140bb020 con 0x7fa0240904f0 2024-09-16T05:24:45.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:24:45.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 10 monclient: dump: 2024-09-16T05:24:45.518 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.518 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.518 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.518 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.518 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.518 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.518 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.518 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.519 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.519 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa023fff640 1 --2- 172.21.15.114:0/1394263457 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa010037840 0x7fa010039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa023fff640 10 monclient: get_auth_request con 0x7fa010037840 auth_method 0 2024-09-16T05:24:45.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 1 -- 172.21.15.114:0/1394263457 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa0140b6670 con 0x7fa0240904f0 2024-09-16T05:24:45.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:45.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:24:45.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 1 -- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240904f0 msgr2=0x7fa02402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240904f0 0x7fa02402c650 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa01401a040 tx=0x7fa01401ac50 comp rx=0 tx=0).stop 2024-09-16T05:24:45.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 1 -- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240904f0 msgr2=0x7fa02402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:24:45.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240904f0 0x7fa02402c650 secure :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa01401a040 tx=0x7fa01401ac50 comp rx=0 tx=0).stop 2024-09-16T05:24:45.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa0217fa640 10 client.104288.objecter ms_handle_connect 0x7fa010037840 2024-09-16T05:24:45.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.498+0000 7fa023fff640 1 --2- 172.21.15.114:0/1394263457 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa010037840 0x7fa010039d00 secure :-1 s=READY pgs=3604 cs=0 l=1 rev1=1 crypto rx=0x7fa024099120 tx=0x7fa018008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.502+0000 7fa02b149640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:24:45.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.502+0000 7fa02b149640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240908d0 0x7fa02402cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.502+0000 7fa02b149640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.502+0000 7fa02b149640 1 -- 172.21.15.114:0/1394263457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa024069390 con 0x7fa0240908d0 2024-09-16T05:24:45.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.502+0000 7fa028ee5640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240908d0 0x7fa02402cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.502+0000 7fa028ee5640 10 monclient: get_auth_request con 0x7fa0240908d0 auth_method 0 2024-09-16T05:24:45.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.502+0000 7fa028ee5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.502+0000 7fa028ee5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.502+0000 7fa028ee5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.502+0000 7fa028ee5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.502+0000 7fa028ee5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.502+0000 7fa028ee5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.502+0000 7fa028ee5640 10 monclient(hunting): handle_auth_done global_id 74811 payload 824 2024-09-16T05:24:45.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.502+0000 7fa028ee5640 1 -- 172.21.15.114:0/1394263457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa02409c6a0 con 0x7fa0240908d0 2024-09-16T05:24:45.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa0217fa640 10 client.104288.objecter ms_handle_connect 0x7fa0240908d0 2024-09-16T05:24:45.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa0217fa640 10 client.104288.objecter resend_mon_ops 2024-09-16T05:24:45.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa028ee5640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240908d0 0x7fa02402cb90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa0140a5010 tx=0x7fa0140a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa0217fa640 1 -- 172.21.15.114:0/1394263457 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0140bb030 con 0x7fa0240908d0 2024-09-16T05:24:45.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa0217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa0217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:24:45.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa0217fa640 10 monclient: dump: 2024-09-16T05:24:45.524 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.524 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.524 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.524 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.525 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa0217fa640 1 -- 172.21.15.114:0/1394263457 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7fa0140c2c70 con 0x7fa0240908d0 2024-09-16T05:24:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa0217fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:24:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa0217fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:24:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa0217fa640 1 -- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240908d0 msgr2=0x7fa02402cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa0217fa640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240908d0 0x7fa02402cb90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa0140a5010 tx=0x7fa0140a8000 comp rx=0 tx=0).stop 2024-09-16T05:24:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa0217fa640 1 -- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240908d0 msgr2=0x7fa02402cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:24:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa0217fa640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0240908d0 0x7fa02402cb90 secure :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa0140a5010 tx=0x7fa0140a8000 comp rx=0 tx=0).stop 2024-09-16T05:24:45.526 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:24:45.526 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:24:45.526 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:24:45.526 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:24:45.526 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 68, 2024-09-16T05:24:45.527 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 644561, 2024-09-16T05:24:45.527 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:24:45.527 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:24:45.527 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:24:45.527 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:24:45.527 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:45.527 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 5, 2024-09-16T05:24:45.527 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:24:45.527 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:24:45.527 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:24:45.527 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:45.527 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:45.528 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:45.528 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:45.528 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:45.528 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:45.528 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:45.528 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:45.528 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:45.528 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:45.528 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:45.528 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:45.528 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:24:45.528 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:24:45.528 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:45.528 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:45.529 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:45.529 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:45.529 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:45.529 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:45.529 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:45.529 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:45.529 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:45.529 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:45.529 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:45.529 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.529 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.529 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:24:45.530 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:24:45.530 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:24:45.530 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:24:45.530 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:24:45.530 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:24:45.530 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:24:45.530 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:24:45.530 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:24:45.530 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:24:45.530 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:24:45.530 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:24:45.530 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:24:45.530 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:24:45.531 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:24:45.531 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:24:45.531 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:24:45.531 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:45.531 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:45.531 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:45.531 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:45.531 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:45.531 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:45.532 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:45.532 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:45.532 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:45.532 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:45.532 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:45.532 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:45.533 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:24:45.533 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.537 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:24:45.537 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.538 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:24:45.538 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:24:45.538 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:45.538 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:45.538 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.538 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:45.538 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:24:45.538 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.538 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.538 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.538 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:45.539 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:24:45.539 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.539 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.539 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.539 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.539 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:24:45.539 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:24:45.539 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:45.539 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:45.539 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:45.539 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.539 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.540 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:24:45.540 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:24:45.540 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:45.540 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:45.540 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.540 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:45.540 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:24:45.540 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.540 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.540 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.540 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:45.540 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:24:45.540 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:24:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:24:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:24:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:24:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:45.542 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:45.542 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.542 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:45.542 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:24:45.542 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.542 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.542 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.542 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:45.542 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:24:45.542 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.542 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.542 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.542 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.543 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:24:45.543 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:24:45.543 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:45.543 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:45.543 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:45.543 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.543 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.543 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.543 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:24:45.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:24:45.544 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.544 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:45.544 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:45.544 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:24:45.544 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.544 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:45.544 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:24:45.544 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.544 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:45.544 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:45.544 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:24:45.544 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.544 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:45.545 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:24:45.545 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.545 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:45.545 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:45.545 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:24:45.545 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.545 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.545 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.545 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:24:45.545 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:24:45.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 10 librados: watch_flush enter 2024-09-16T05:24:45.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 10 librados: watch_flush exit 2024-09-16T05:24:45.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 20 client.104288.objecter shutdown clearing up homeless session... 2024-09-16T05:24:45.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 10 client.104288.objecter successfully canceled tick 2024-09-16T05:24:45.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 1 -- 172.21.15.114:0/1394263457 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa010037840 msgr2=0x7fa010039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 1 --2- 172.21.15.114:0/1394263457 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa010037840 0x7fa010039d00 secure :-1 s=READY pgs=3604 cs=0 l=1 rev1=1 crypto rx=0x7fa024099120 tx=0x7fa018008040 comp rx=0 tx=0).stop 2024-09-16T05:24:45.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 10 monclient: shutdown 2024-09-16T05:24:45.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 1 -- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa024139640 msgr2=0x7fa02408f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa0296e6640 1 -- 172.21.15.114:0/1394263457 reap_dead start 2024-09-16T05:24:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa024139640 0x7fa02408f7f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa01c010300 tx=0x7fa01c0107c0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 1 -- 172.21.15.114:0/1394263457 shutdown_connections 2024-09-16T05:24:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 1 --2- 172.21.15.114:0/1394263457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa024139640 0x7fa02408f7f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 1 -- 172.21.15.114:0/1394263457 >> 172.21.15.114:0/1394263457 conn(0x7fa0240a1fb0 msgr2=0x7fa02406f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 1 -- 172.21.15.114:0/1394263457 shutdown_connections 2024-09-16T05:24:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 1 -- 172.21.15.114:0/1394263457 wait complete. 2024-09-16T05:24:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.506+0000 7fa02b149640 1 librados: shutdown 2024-09-16T05:24:45.548 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:24:45.549 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:24:45.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:24:45.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 10 monclient: get_monmap_and_config 2024-09-16T05:24:45.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 10 monclient: build_initial_monmap 2024-09-16T05:24:45.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 10 monclient: monmap: 2024-09-16T05:24:45.663 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:45.663 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.664 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:24:45.669285+0000 2024-09-16T05:24:45.664 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:24:45.669285+0000 2024-09-16T05:24:45.664 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:45.664 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.664 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:24:45.664 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:24:45.664 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:24:45.664 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:24:45.664 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:24:45.664 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 1 Processor -- start 2024-09-16T05:24:45.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 1 -- start start 2024-09-16T05:24:45.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 10 monclient: init 2024-09-16T05:24:45.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:45.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T05:24:45.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9654137730 addr v1:172.21.15.114:6789/0 2024-09-16T05:24:45.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f96540a5d00 0x7f965413c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 10 monclient(hunting): picked mon.noname-a con 0x7f96540a5d00 addr v2:172.21.15.114:3300/0 2024-09-16T05:24:45.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f965413ca00 addr v1:172.21.15.149:6789/0 2024-09-16T05:24:45.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96540a55e0 con 0x7f9654137730 2024-09-16T05:24:45.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96540a3710 con 0x7f965413ca00 2024-09-16T05:24:45.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f965409c450 con 0x7f96540a5d00 2024-09-16T05:24:45.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:45.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f965b735640 10 monclient(hunting): authenticate will time out at 1970.299163s 2024-09-16T05:24:45.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f96540a5d00 0x7f965413c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f96540a5d00 0x7f965413c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40244/0 (socket says 172.21.15.114:40244) 2024-09-16T05:24:45.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 1 -- 172.21.15.114:0/641029900 learned_addr learned my addr 172.21.15.114:0/641029900 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:45.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 10 monclient(hunting): get_auth_request con 0x7f96540a5d00 auth_method 0 2024-09-16T05:24:45.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 10 monclient(hunting): handle_auth_done global_id 74814 payload 274 2024-09-16T05:24:45.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 1 -- 172.21.15.114:0/641029900 >> v1:172.21.15.149:6789/0 conn(0x7f965413ca00 legacy=0x7f965413edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:45.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 1 -- 172.21.15.114:0/641029900 >> v1:172.21.15.114:6789/0 conn(0x7f9654137730 legacy=0x7f9654137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 10 monclient: _finish_hunting 0 2024-09-16T05:24:45.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 1 monclient: found mon.noname-a 2024-09-16T05:24:45.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:45.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:24:45.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.666+0000 7f9658cd0640 1 -- 172.21.15.114:0/641029900 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965409be00 con 0x7f96540a5d00 2024-09-16T05:24:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f9658cd0640 1 --2- 172.21.15.114:0/641029900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96540a5d00 0x7f965413c4c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9644009870 tx=0x7f964401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc673be3c9852d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f964bfff640 1 -- 172.21.15.114:0/641029900 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96440b0070 con 0x7f96540a5d00 2024-09-16T05:24:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f964bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f964bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:24:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f964bfff640 10 monclient: dump: 2024-09-16T05:24:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.676 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.676 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.676 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.676 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f964bfff640 10 monclient: _finish_auth 0 2024-09-16T05:24:45.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f964bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:24:45.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f964bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:45.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 5 monclient: authenticate success, global_id 74814 2024-09-16T05:24:45.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:24:45.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f964bfff640 1 -- 172.21.15.114:0/641029900 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96440ab600 con 0x7f96540a5d00 2024-09-16T05:24:45.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f964bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:45.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 10 monclient: get_monmap_and_config success 2024-09-16T05:24:45.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 4 set_mon_vals no callback set 2024-09-16T05:24:45.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f964bfff640 1 -- 172.21.15.114:0/641029900 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96440ab9d0 con 0x7f96540a5d00 2024-09-16T05:24:45.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f964bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f964bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:24:45.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f964bfff640 10 monclient: dump: 2024-09-16T05:24:45.677 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.677 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.677 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.677 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.677 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.677 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.677 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.677 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.677 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.678 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 10 monclient: shutdown 2024-09-16T05:24:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 1 -- 172.21.15.114:0/641029900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96540a5d00 msgr2=0x7f965413c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 1 --2- 172.21.15.114:0/641029900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96540a5d00 0x7f965413c4c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9644009870 tx=0x7f964401c3e0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 1 -- 172.21.15.114:0/641029900 shutdown_connections 2024-09-16T05:24:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 1 --2- 172.21.15.114:0/641029900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96540a5d00 0x7f965413c4c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 1 -- 172.21.15.114:0/641029900 >> 172.21.15.114:0/641029900 conn(0x7f96540a9c70 msgr2=0x7f96540aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 1 -- 172.21.15.114:0/641029900 shutdown_connections 2024-09-16T05:24:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 1 -- 172.21.15.114:0/641029900 wait complete. 2024-09-16T05:24:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 10 monclient: build_initial_monmap 2024-09-16T05:24:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 10 monclient: monmap: 2024-09-16T05:24:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:24:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:24:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:24:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:24:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:24:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:24:45.679 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 1 librados: starting msgr at 2024-09-16T05:24:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 1 librados: starting objecter 2024-09-16T05:24:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 1 Processor -- start 2024-09-16T05:24:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 1 -- start start 2024-09-16T05:24:45.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 1 librados: setting wanted keys 2024-09-16T05:24:45.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 1 librados: calling monclient init 2024-09-16T05:24:45.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 10 monclient: init 2024-09-16T05:24:45.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:45.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:24:45.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f96540a5d00 0x7f96541361d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.670+0000 7f965b735640 10 monclient(hunting): picked mon.noname-b con 0x7f96540a5d00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:24:45.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9654137730 0x7f965409e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 10 monclient(hunting): picked mon.noname-c con 0x7f9654137730 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:45.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f965413ca00 0x7f965409ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 10 monclient(hunting): picked mon.noname-a con 0x7f965413ca00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:45.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9654142eb0 con 0x7f965413ca00 2024-09-16T05:24:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f9654142d30 con 0x7f96540a5d00 2024-09-16T05:24:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9654143030 con 0x7f9654137730 2024-09-16T05:24:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 10 monclient(hunting): authenticate will time out at 1970.304048s 2024-09-16T05:24:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9654137730 0x7f965409e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9654137730 0x7f965409e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56398/0 (socket says 172.21.15.114:56398) 2024-09-16T05:24:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9659cd2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f965413ca00 0x7f965409ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 1 -- 172.21.15.114:0/2846923441 learned_addr learned my addr 172.21.15.114:0/2846923441 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 10 monclient(hunting): get_auth_request con 0x7f9654137730 auth_method 0 2024-09-16T05:24:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9659cd2640 10 monclient(hunting): get_auth_request con 0x7f965413ca00 auth_method 0 2024-09-16T05:24:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9659cd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9659cd2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9659cd2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9659cd2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9659cd2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9659cd2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 10 monclient(hunting): handle_auth_done global_id 104291 payload 824 2024-09-16T05:24:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f96594d1640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f96540a5d00 0x7f96541361d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 1 -- 172.21.15.114:0/2846923441 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f96540a5d00 msgr2=0x7f96541361d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f96540a5d00 0x7f96541361d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 1 -- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f965413ca00 msgr2=0x7f965409ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f965413ca00 0x7f965409ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 10 monclient: _finish_hunting 0 2024-09-16T05:24:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 1 monclient: found mon.noname-c 2024-09-16T05:24:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:24:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 1 -- 172.21.15.114:0/2846923441 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96540a1530 con 0x7f9654137730 2024-09-16T05:24:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9659cd2640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f965413ca00 0x7f965409ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:24:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9658cd0640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9654137730 0x7f965409e9f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f96440ae040 tx=0x7f9644019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 client.?.objecter ms_handle_connect 0x7f9654137730 2024-09-16T05:24:45.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:24:45.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 1 -- 172.21.15.114:0/2846923441 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96440b0040 con 0x7f9654137730 2024-09-16T05:24:45.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:24:45.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 monclient: dump: 2024-09-16T05:24:45.684 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.684 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.684 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.684 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.684 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.684 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.684 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.684 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.685 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 monclient: _finish_auth 0 2024-09-16T05:24:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:24:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 1 -- 172.21.15.114:0/2846923441 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96440b9070 con 0x7f9654137730 2024-09-16T05:24:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 5 monclient: authenticate success, global_id 104291 2024-09-16T05:24:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 10 monclient: _renew_subs 2024-09-16T05:24:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 1 -- 172.21.15.114:0/2846923441 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96540a26a0 con 0x7f9654137730 2024-09-16T05:24:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 1 -- 172.21.15.114:0/2846923441 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96440b4880 con 0x7f9654137730 2024-09-16T05:24:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 monclient: dump: 2024-09-16T05:24:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.686 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 10 client.104291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:24:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 10 monclient: _renew_subs 2024-09-16T05:24:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:45.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 1 -- 172.21.15.114:0/2846923441 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f96540a9270 con 0x7f9654137730 2024-09-16T05:24:45.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 1 librados: init done 2024-09-16T05:24:45.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f964b7fe640 4 set_mon_vals no callback set 2024-09-16T05:24:45.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:45.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965409fc30 0x7f965402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f965b735640 1 -- 172.21.15.114:0/2846923441 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9654073840 con 0x7f965409fc30 2024-09-16T05:24:45.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 1 -- 172.21.15.114:0/2846923441 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f96440ab4c0 con 0x7f9654137730 2024-09-16T05:24:45.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f96594d1640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965409fc30 0x7f965402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f96594d1640 10 monclient: get_auth_request con 0x7f965409fc30 auth_method 0 2024-09-16T05:24:45.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f96594d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f96594d1640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f96594d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f96594d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 1 --2- 172.21.15.114:0/2846923441 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9638037810 0x7f9638039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f96594d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f96594d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 1 -- 172.21.15.114:0/2846923441 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(690..690 src has 1..690) ==== 5965+0+0 (secure 0 0 0) 0x7f96440e2d20 con 0x7f9654137730 2024-09-16T05:24:45.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 client.104291.objecter ms_dispatch 0x7f9654000ba0 osd_map(690..690 src has 1..690) 2024-09-16T05:24:45.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 3 client.104291.objecter handle_osd_map got epochs [690,690] > 0 2024-09-16T05:24:45.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 3 client.104291.objecter handle_osd_map decoding full epoch 690 2024-09-16T05:24:45.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f96594d1640 10 monclient(hunting): handle_auth_done global_id 104294 payload 824 2024-09-16T05:24:45.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 20 client.104291.objecter dump_active .. 0 homeless 2024-09-16T05:24:45.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f96594d1640 1 -- 172.21.15.114:0/2846923441 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9654137b10 con 0x7f965409fc30 2024-09-16T05:24:45.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 client.104291.objecter ms_handle_connect 0x7f965409fc30 2024-09-16T05:24:45.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 client.104291.objecter resend_mon_ops 2024-09-16T05:24:45.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f96594d1640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965409fc30 0x7f965402c650 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f965000d380 tx=0x7f965000d840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9659cd2640 1 --2- 172.21.15.114:0/2846923441 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9638037810 0x7f9638039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9659cd2640 10 monclient: get_auth_request con 0x7f9638037810 auth_method 0 2024-09-16T05:24:45.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 1 -- 172.21.15.114:0/2846923441 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650018070 con 0x7f965409fc30 2024-09-16T05:24:45.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:45.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 monclient: dump: 2024-09-16T05:24:45.689 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.689 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.689 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.689 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.689 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.689 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.689 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.690 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.690 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.690 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 1 -- 172.21.15.114:0/2846923441 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9650003620 con 0x7f965409fc30 2024-09-16T05:24:45.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:45.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:24:45.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 1 -- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965409fc30 msgr2=0x7f965402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965409fc30 0x7f965402c650 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f965000d380 tx=0x7f965000d840 comp rx=0 tx=0).stop 2024-09-16T05:24:45.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 1 -- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965409fc30 msgr2=0x7f965402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:24:45.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965409fc30 0x7f965402c650 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9649ffb640 10 client.104291.objecter ms_handle_connect 0x7f9638037810 2024-09-16T05:24:45.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.674+0000 7f9659cd2640 1 --2- 172.21.15.114:0/2846923441 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9638037810 0x7f9638039cd0 secure :-1 s=READY pgs=3605 cs=0 l=1 rev1=1 crypto rx=0x7f96540a1b50 tx=0x7f964c003d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:24:45.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965402cb90 0x7f965402cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 1 -- 172.21.15.114:0/2846923441 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f965413b440 con 0x7f965402cb90 2024-09-16T05:24:45.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f96594d1640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965402cb90 0x7f965402cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f96594d1640 10 monclient: get_auth_request con 0x7f965402cb90 auth_method 0 2024-09-16T05:24:45.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f96594d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f96594d1640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f96594d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f96594d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f96594d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f96594d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f96594d1640 10 monclient(hunting): handle_auth_done global_id 104297 payload 824 2024-09-16T05:24:45.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f96594d1640 1 -- 172.21.15.114:0/2846923441 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9654068650 con 0x7f965402cb90 2024-09-16T05:24:45.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f9649ffb640 10 client.104291.objecter ms_handle_connect 0x7f965402cb90 2024-09-16T05:24:45.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f9649ffb640 10 client.104291.objecter resend_mon_ops 2024-09-16T05:24:45.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f96594d1640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965402cb90 0x7f965402cfa0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f965000dfd0 tx=0x7f9650000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f9649ffb640 1 -- 172.21.15.114:0/2846923441 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650018040 con 0x7f965402cb90 2024-09-16T05:24:45.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f9649ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f9649ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:45.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f9649ffb640 10 monclient: dump: 2024-09-16T05:24:45.694 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.694 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.694 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.694 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.694 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.694 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.694 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.695 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.695 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.695 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f9649ffb640 1 -- 172.21.15.114:0/2846923441 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f9650013470 con 0x7f965402cb90 2024-09-16T05:24:45.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f9649ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:24:45.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f9649ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:24:45.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f9649ffb640 1 -- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965402cb90 msgr2=0x7f965402cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f9649ffb640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965402cb90 0x7f965402cfa0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f965000dfd0 tx=0x7f9650000ee0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f9649ffb640 1 -- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965402cb90 msgr2=0x7f965402cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:24:45.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f9649ffb640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f965402cb90 0x7f965402cfa0 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f965000dfd0 tx=0x7f9650000ee0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.695 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:24:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:24:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:24:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:24:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 68, 2024-09-16T05:24:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 261846, 2024-09-16T05:24:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:24:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:24:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:24:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:24:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 6, 2024-09-16T05:24:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:24:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:24:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:24:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:24:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:24:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:24:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:24:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:24:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:24:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:24:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:24:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:24:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:24:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:24:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:24:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:24:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:24:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:24:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:24:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:24:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:24:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:24:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:24:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:24:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:24:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:24:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:24:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:24:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:24:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:24:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:24:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:24:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:24:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:24:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:24:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:24:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:24:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:24:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:24:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:24:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:24:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:24:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:24:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:24:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:24:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:24:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:24:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:24:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 4 2024-09-16T05:24:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:24:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:24:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.709 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:24:45.709 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:24:45.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 10 librados: watch_flush enter 2024-09-16T05:24:45.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 10 librados: watch_flush exit 2024-09-16T05:24:45.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 20 client.104291.objecter shutdown clearing up homeless session... 2024-09-16T05:24:45.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 10 client.104291.objecter successfully canceled tick 2024-09-16T05:24:45.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 1 -- 172.21.15.114:0/2846923441 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9638037810 msgr2=0x7f9638039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 1 --2- 172.21.15.114:0/2846923441 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9638037810 0x7f9638039cd0 secure :-1 s=READY pgs=3605 cs=0 l=1 rev1=1 crypto rx=0x7f96540a1b50 tx=0x7f964c003d50 comp rx=0 tx=0).stop 2024-09-16T05:24:45.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 10 monclient: shutdown 2024-09-16T05:24:45.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:45.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 1 -- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9654137730 msgr2=0x7f965409e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9654137730 0x7f965409e9f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f96440ae040 tx=0x7f9644019180 comp rx=0 tx=0).stop 2024-09-16T05:24:45.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f9659cd2640 1 -- 172.21.15.114:0/2846923441 reap_dead start 2024-09-16T05:24:45.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 1 -- 172.21.15.114:0/2846923441 shutdown_connections 2024-09-16T05:24:45.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 1 --2- 172.21.15.114:0/2846923441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9654137730 0x7f965409e9f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 1 -- 172.21.15.114:0/2846923441 >> 172.21.15.114:0/2846923441 conn(0x7f96540a9c70 msgr2=0x7f965413e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:45.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 1 -- 172.21.15.114:0/2846923441 shutdown_connections 2024-09-16T05:24:45.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 1 -- 172.21.15.114:0/2846923441 wait complete. 2024-09-16T05:24:45.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.682+0000 7f965b735640 1 librados: shutdown 2024-09-16T05:24:45.712 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:24:45.712 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:24:45.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:24:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 10 monclient: get_monmap_and_config 2024-09-16T05:24:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 10 monclient: build_initial_monmap 2024-09-16T05:24:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 10 monclient: monmap: 2024-09-16T05:24:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:24:45.836752+0000 2024-09-16T05:24:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:24:45.836752+0000 2024-09-16T05:24:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:24:45.832 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:24:45.832 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:24:45.832 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:24:45.832 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:24:45.832 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 1 Processor -- start 2024-09-16T05:24:45.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 1 -- start start 2024-09-16T05:24:45.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 10 monclient: init 2024-09-16T05:24:45.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:45.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-16T05:24:45.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f966809b6c0 0x7f96680a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 10 monclient(hunting): picked mon.noname-c con 0x7f966809b6c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:24:45.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f96680a1400 0x7f96680a1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 10 monclient(hunting): picked mon.noname-a con 0x7f96680a1400 addr v2:172.21.15.114:3300/0 2024-09-16T05:24:45.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9668139640 addr v1:172.21.15.114:6789/0 2024-09-16T05:24:45.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f966809d4e0 con 0x7f9668139640 2024-09-16T05:24:45.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f96680a1d50 con 0x7f96680a1400 2024-09-16T05:24:45.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f966809ece0 con 0x7f966809b6c0 2024-09-16T05:24:45.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:45.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966cfa1640 10 monclient(hunting): authenticate will time out at 1970.466359s 2024-09-16T05:24:45.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f96680a1400 0x7f96680a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f96680a1400 0x7f96680a1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40258/0 (socket says 172.21.15.114:40258) 2024-09-16T05:24:45.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 1 -- 172.21.15.114:0/1188132856 learned_addr learned my addr 172.21.15.114:0/1188132856 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:45.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 10 monclient(hunting): get_auth_request con 0x7f96680a1400 auth_method 0 2024-09-16T05:24:45.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966659c640 1 --2- 172.21.15.114:0/1188132856 >> v2:172.21.15.149:3300/0 conn(0x7f966809b6c0 0x7f96680a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966659c640 10 monclient(hunting): get_auth_request con 0x7f966809b6c0 auth_method 0 2024-09-16T05:24:45.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f966659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 10 monclient(hunting): handle_auth_done global_id 74820 payload 274 2024-09-16T05:24:45.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 1 -- 172.21.15.114:0/1188132856 >> v2:172.21.15.149:3300/0 conn(0x7f966809b6c0 msgr2=0x7f96680a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:45.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 1 --2- 172.21.15.114:0/1188132856 >> v2:172.21.15.149:3300/0 conn(0x7f966809b6c0 0x7f96680a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 1 -- 172.21.15.114:0/1188132856 >> v1:172.21.15.114:6789/0 conn(0x7f9668139640 legacy=0x7f966813ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:24:45.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 1 monclient: found mon.noname-a 2024-09-16T05:24:45.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:45.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:24:45.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.834+0000 7f9665d9b640 1 -- 172.21.15.114:0/1188132856 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f966809f430 con 0x7f96680a1400 2024-09-16T05:24:45.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f9665d9b640 1 --2- 172.21.15.114:0/1188132856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96680a1400 0x7f96680a1810 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f96680a2f70 tx=0x7f965001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5511f8c2497ecd95 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966559a640 1 -- 172.21.15.114:0/1188132856 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96500b0070 con 0x7f96680a1400 2024-09-16T05:24:45.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:24:45.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966559a640 10 monclient: dump: 2024-09-16T05:24:45.844 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.844 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.844 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.844 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.844 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.844 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.844 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.844 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.844 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.844 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966559a640 10 monclient: _finish_auth 0 2024-09-16T05:24:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966559a640 10 monclient: _check_auth_tickets 2024-09-16T05:24:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966559a640 1 -- 172.21.15.114:0/1188132856 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96500ab600 con 0x7f96680a1400 2024-09-16T05:24:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966559a640 1 -- 172.21.15.114:0/1188132856 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96500ab9d0 con 0x7f96680a1400 2024-09-16T05:24:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 5 monclient: authenticate success, global_id 74820 2024-09-16T05:24:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 10 monclient: get_monmap_and_config success 2024-09-16T05:24:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 4 set_mon_vals no callback set 2024-09-16T05:24:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:24:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966559a640 10 monclient: dump: 2024-09-16T05:24:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.846 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 10 monclient: shutdown 2024-09-16T05:24:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:45.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 -- 172.21.15.114:0/1188132856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96680a1400 msgr2=0x7f96680a1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 --2- 172.21.15.114:0/1188132856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96680a1400 0x7f96680a1810 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f96680a2f70 tx=0x7f965001c470 comp rx=0 tx=0).stop 2024-09-16T05:24:45.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 -- 172.21.15.114:0/1188132856 shutdown_connections 2024-09-16T05:24:45.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 --2- 172.21.15.114:0/1188132856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96680a1400 0x7f96680a1810 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 --2- 172.21.15.114:0/1188132856 >> v2:172.21.15.149:3300/0 conn(0x7f966809b6c0 0x7f96680a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 -- 172.21.15.114:0/1188132856 >> 172.21.15.114:0/1188132856 conn(0x7f96680a1fb0 msgr2=0x7f96680a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:45.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 -- 172.21.15.114:0/1188132856 shutdown_connections 2024-09-16T05:24:45.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 -- 172.21.15.114:0/1188132856 wait complete. 2024-09-16T05:24:45.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 10 monclient: build_initial_monmap 2024-09-16T05:24:45.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 10 monclient: monmap: 2024-09-16T05:24:45.847 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:45.847 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:24:45.847 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:24:45.848 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:24:45.848 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:45.848 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.848 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:24:45.848 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:24:45.848 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:24:45.848 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 librados: starting msgr at 2024-09-16T05:24:45.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 librados: starting objecter 2024-09-16T05:24:45.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 Processor -- start 2024-09-16T05:24:45.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 -- start start 2024-09-16T05:24:45.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 librados: setting wanted keys 2024-09-16T05:24:45.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 librados: calling monclient init 2024-09-16T05:24:45.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 10 monclient: init 2024-09-16T05:24:45.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:45.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:24:45.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f966809b6c0 0x7f96681ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 10 monclient(hunting): picked mon.noname-a con 0x7f966809b6c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:45.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96680a1400 0x7f9668098670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 10 monclient(hunting): picked mon.noname-c con 0x7f96680a1400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:45.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668139640 0x7f9668098bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 10 monclient(hunting): picked mon.noname-b con 0x7f9668139640 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:24:45.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f966813f970 con 0x7f966809b6c0 2024-09-16T05:24:45.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f966809cad0 con 0x7f9668139640 2024-09-16T05:24:45.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f966813faf0 con 0x7f96680a1400 2024-09-16T05:24:45.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:45.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966cfa1640 10 monclient(hunting): authenticate will time out at 1970.471889s 2024-09-16T05:24:45.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f966809b6c0 0x7f96681ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f966809b6c0 0x7f96681ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40264/0 (socket says 172.21.15.114:40264) 2024-09-16T05:24:45.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966659c640 1 -- 172.21.15.114:0/2007366972 learned_addr learned my addr 172.21.15.114:0/2007366972 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:45.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966659c640 10 monclient(hunting): get_auth_request con 0x7f966809b6c0 auth_method 0 2024-09-16T05:24:45.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f966659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.838+0000 7f9665d9b640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96680a1400 0x7f9668098670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 10 monclient(hunting): get_auth_request con 0x7f96680a1400 auth_method 0 2024-09-16T05:24:45.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966659c640 10 monclient(hunting): handle_auth_done global_id 74823 payload 824 2024-09-16T05:24:45.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9666d9d640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668139640 0x7f9668098bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966659c640 1 -- 172.21.15.114:0/2007366972 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96680a1400 msgr2=0x7f9668098670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966659c640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96680a1400 0x7f9668098670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966659c640 1 -- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668139640 msgr2=0x7f9668098bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966659c640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668139640 0x7f9668098bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966659c640 10 monclient: _finish_hunting 0 2024-09-16T05:24:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966659c640 1 monclient: found mon.noname-a 2024-09-16T05:24:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:24:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966659c640 1 -- 172.21.15.114:0/2007366972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9668139c20 con 0x7f966809b6c0 2024-09-16T05:24:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966659c640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f966809b6c0 0x7f96681ccfa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9660007e40 tx=0x7f966000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 client.?.objecter ms_handle_connect 0x7f966809b6c0 2024-09-16T05:24:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:24:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 1 -- 172.21.15.114:0/2007366972 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660014620 con 0x7f966809b6c0 2024-09-16T05:24:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:24:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 monclient: dump: 2024-09-16T05:24:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.853 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 monclient: _finish_auth 0 2024-09-16T05:24:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 monclient: _check_auth_tickets 2024-09-16T05:24:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966cfa1640 5 monclient: authenticate success, global_id 74823 2024-09-16T05:24:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 1 -- 172.21.15.114:0/2007366972 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9660019070 con 0x7f966809b6c0 2024-09-16T05:24:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966cfa1640 10 monclient: _renew_subs 2024-09-16T05:24:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 1 -- 172.21.15.114:0/2007366972 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660014a00 con 0x7f966809b6c0 2024-09-16T05:24:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966cfa1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:24:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966cfa1640 1 -- 172.21.15.114:0/2007366972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f966808eb00 con 0x7f966809b6c0 2024-09-16T05:24:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:24:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 monclient: dump: 2024-09-16T05:24:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.855 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.855 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.855 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.855 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.855 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.855 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.855 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9664d99640 4 set_mon_vals no callback set 2024-09-16T05:24:45.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966cfa1640 10 client.74823.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:24:45.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966cfa1640 10 monclient: _renew_subs 2024-09-16T05:24:45.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966cfa1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:24:45.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966cfa1640 1 -- 172.21.15.114:0/2007366972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f966808e580 con 0x7f966809b6c0 2024-09-16T05:24:45.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966cfa1640 1 librados: init done 2024-09-16T05:24:45.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966cfa1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:45.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966cfa1640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668072710 0x7f966802c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966cfa1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f966cfa1640 1 -- 172.21.15.114:0/2007366972 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f9668069390 con 0x7f9668072710 2024-09-16T05:24:45.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 1 -- 172.21.15.114:0/2007366972 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9660014ba0 con 0x7f966809b6c0 2024-09-16T05:24:45.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 1 --2- 172.21.15.114:0/2007366972 >> v2:172.21.15.149:6812/1185531135 conn(0x7f964c037810 0x7f964c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668072710 0x7f966802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 1 -- 172.21.15.114:0/2007366972 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(690..690 src has 1..690) ==== 5965+0+0 (secure 0 0 0) 0x7f9660049830 con 0x7f966809b6c0 2024-09-16T05:24:45.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 client.74823.objecter ms_dispatch 0x7f9668000ba0 osd_map(690..690 src has 1..690) 2024-09-16T05:24:45.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 3 client.74823.objecter handle_osd_map got epochs [690,690] > 0 2024-09-16T05:24:45.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 3 client.74823.objecter handle_osd_map decoding full epoch 690 2024-09-16T05:24:45.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 10 monclient: get_auth_request con 0x7f9668072710 auth_method 0 2024-09-16T05:24:45.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 20 client.74823.objecter dump_active .. 0 homeless 2024-09-16T05:24:45.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9666d9d640 1 --2- 172.21.15.114:0/2007366972 >> v2:172.21.15.149:6812/1185531135 conn(0x7f964c037810 0x7f964c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9666d9d640 10 monclient: get_auth_request con 0x7f964c037810 auth_method 0 2024-09-16T05:24:45.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 10 monclient(hunting): handle_auth_done global_id 124117 payload 824 2024-09-16T05:24:45.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 1 -- 172.21.15.114:0/2007366972 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f966813ce00 con 0x7f9668072710 2024-09-16T05:24:45.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 client.74823.objecter ms_handle_connect 0x7f9668072710 2024-09-16T05:24:45.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 client.74823.objecter resend_mon_ops 2024-09-16T05:24:45.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9665d9b640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668072710 0x7f966802c650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f966809d220 tx=0x7f9650019210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 client.74823.objecter ms_handle_connect 0x7f964c037810 2024-09-16T05:24:45.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f9666d9d640 1 --2- 172.21.15.114:0/2007366972 >> v2:172.21.15.149:6812/1185531135 conn(0x7f964c037810 0x7f964c039cd0 secure :-1 s=READY pgs=3606 cs=0 l=1 rev1=1 crypto rx=0x7f9658000f70 tx=0x7f9658009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 1 -- 172.21.15.114:0/2007366972 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96500ba030 con 0x7f9668072710 2024-09-16T05:24:45.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:24:45.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 monclient: dump: 2024-09-16T05:24:45.858 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.858 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.858 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.858 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.858 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.858 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.858 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.858 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.859 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.859 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 1 -- 172.21.15.114:0/2007366972 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9650003620 con 0x7f9668072710 2024-09-16T05:24:45.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:45.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:24:45.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 1 -- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668072710 msgr2=0x7f966802c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668072710 0x7f966802c650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f966809d220 tx=0x7f9650019210 comp rx=0 tx=0).stop 2024-09-16T05:24:45.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 1 -- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668072710 msgr2=0x7f966802c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:24:45.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.842+0000 7f965effd640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668072710 0x7f966802c650 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f966cfa1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:24:45.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f966cfa1640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668072af0 0x7f966802cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:45.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f966cfa1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:45.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f966cfa1640 1 -- 172.21.15.114:0/2007366972 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f966806f0a0 con 0x7f9668072af0 2024-09-16T05:24:45.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f9665d9b640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668072af0 0x7f966802cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:45.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f9665d9b640 10 monclient: get_auth_request con 0x7f9668072af0 auth_method 0 2024-09-16T05:24:45.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f9665d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:45.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f9665d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:45.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f9665d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:45.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f9665d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:45.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f9665d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:45.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f9665d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:45.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f9665d9b640 10 monclient(hunting): handle_auth_done global_id 124120 payload 824 2024-09-16T05:24:45.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f9665d9b640 1 -- 172.21.15.114:0/2007366972 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f96680a0e40 con 0x7f9668072af0 2024-09-16T05:24:45.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f9665d9b640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668072af0 0x7f966802cb90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9650019080 tx=0x7f96500a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:45.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f965effd640 10 client.74823.objecter ms_handle_connect 0x7f9668072af0 2024-09-16T05:24:45.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f965effd640 10 client.74823.objecter resend_mon_ops 2024-09-16T05:24:45.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f965effd640 1 -- 172.21.15.114:0/2007366972 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96500ba040 con 0x7f9668072af0 2024-09-16T05:24:45.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f965effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:45.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f965effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:24:45.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.850+0000 7f965effd640 10 monclient: dump: 2024-09-16T05:24:45.862 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:45.863 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:45.863 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.863 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:45.863 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:45.863 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:45.863 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:45.863 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:45.863 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:45.863 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:45.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f965effd640 1 -- 172.21.15.114:0/2007366972 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f96500b6470 con 0x7f9668072af0 2024-09-16T05:24:45.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f965effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:24:45.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f965effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:24:45.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f965effd640 1 -- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668072af0 msgr2=0x7f966802cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f965effd640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668072af0 0x7f966802cb90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9650019080 tx=0x7f96500a8000 comp rx=0 tx=0).stop 2024-09-16T05:24:45.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f965effd640 1 -- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668072af0 msgr2=0x7f966802cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:24:45.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f965effd640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9668072af0 0x7f966802cb90 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9650019080 tx=0x7f96500a8000 comp rx=0 tx=0).stop 2024-09-16T05:24:45.864 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:24:45.864 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:24:45.864 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:24:45.864 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:24:45.864 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 68, 2024-09-16T05:24:45.864 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 7875, 2024-09-16T05:24:45.864 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:24:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:24:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:24:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:24:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 6, 2024-09-16T05:24:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:24:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:24:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:24:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:24:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:24:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:24:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:24:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:24:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:24:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:24:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:24:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:24:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:24:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:24:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:24:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:24:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:24:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:24:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:24:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:24:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:24:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:24:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:24:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:24:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:24:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:24:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:24:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:24:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:24:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:24:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:24:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:24:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:24:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:24:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:24:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:24:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:24:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:24:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:24:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:24:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:24:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:24:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:24:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:24:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:24:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:24:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:24:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:24:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:24:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:24:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:24:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:24:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:24:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:24:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:24:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:24:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:24:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:24:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:24:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:24:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:24:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:24:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:24:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:24:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:24:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:24:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:24:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:24:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:24:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:24:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:24:45.877 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:24:45.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:24:45.877 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:24:45.877 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:24:45.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 10 librados: watch_flush enter 2024-09-16T05:24:45.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 10 librados: watch_flush exit 2024-09-16T05:24:45.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 20 client.74823.objecter shutdown clearing up homeless session... 2024-09-16T05:24:45.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 10 client.74823.objecter successfully canceled tick 2024-09-16T05:24:45.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 1 -- 172.21.15.114:0/2007366972 >> v2:172.21.15.149:6812/1185531135 conn(0x7f964c037810 msgr2=0x7f964c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 1 --2- 172.21.15.114:0/2007366972 >> v2:172.21.15.149:6812/1185531135 conn(0x7f964c037810 0x7f964c039cd0 secure :-1 s=READY pgs=3606 cs=0 l=1 rev1=1 crypto rx=0x7f9658000f70 tx=0x7f9658009040 comp rx=0 tx=0).stop 2024-09-16T05:24:45.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 10 monclient: shutdown 2024-09-16T05:24:45.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:45.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 1 -- 172.21.15.114:0/2007366972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f966809b6c0 msgr2=0x7f96681ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:45.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f9666d9d640 1 -- 172.21.15.114:0/2007366972 reap_dead start 2024-09-16T05:24:45.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f966809b6c0 0x7f96681ccfa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9660007e40 tx=0x7f966000e4b0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 1 -- 172.21.15.114:0/2007366972 shutdown_connections 2024-09-16T05:24:45.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 1 --2- 172.21.15.114:0/2007366972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f966809b6c0 0x7f96681ccfa0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:45.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 1 -- 172.21.15.114:0/2007366972 >> 172.21.15.114:0/2007366972 conn(0x7f96680a1fb0 msgr2=0x7f9668138970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:45.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 1 -- 172.21.15.114:0/2007366972 shutdown_connections 2024-09-16T05:24:45.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 1 -- 172.21.15.114:0/2007366972 wait complete. 2024-09-16T05:24:45.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:45.854+0000 7f966cfa1640 1 librados: shutdown 2024-09-16T05:24:45.880 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:24:45.880 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T05:24:45.880 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T05:24:45.880 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T05:24:45.880 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T05:24:45.881 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T05:24:45.881 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:24:45.899 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T05:24:45.899 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:24:45.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:24:46.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 10 monclient: get_monmap_and_config 2024-09-16T05:24:46.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 10 monclient: build_initial_monmap 2024-09-16T05:24:46.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 10 monclient: monmap: 2024-09-16T05:24:46.060 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:46.060 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:46.060 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:24:46.065075+0000 2024-09-16T05:24:46.060 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:24:46.065075+0000 2024-09-16T05:24:46.060 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:46.060 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:46.060 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:24:46.060 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:24:46.060 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:24:46.060 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:24:46.060 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:24:46.061 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:46.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 1 Processor -- start 2024-09-16T05:24:46.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 1 -- start start 2024-09-16T05:24:46.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 10 monclient: init 2024-09-16T05:24:46.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:46.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-16T05:24:46.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f47a8071da0 addr v1:172.21.15.149:6789/0 2024-09-16T05:24:46.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a80a1ab0 0x7f47a8072890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:46.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 10 monclient(hunting): picked mon.noname-b con 0x7f47a80a1ab0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:46.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f47a8099cc0 0x7f47a813e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:46.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 10 monclient(hunting): picked mon.noname-a con 0x7f47a8099cc0 addr v2:172.21.15.114:3300/0 2024-09-16T05:24:46.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47a80a1870 con 0x7f47a8071da0 2024-09-16T05:24:46.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:46.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f47a8073e20 con 0x7f47a8099cc0 2024-09-16T05:24:46.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:46.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f47a80a1120 con 0x7f47a80a1ab0 2024-09-16T05:24:46.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:46.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47afbe8640 10 monclient(hunting): authenticate will time out at 1970.694681s 2024-09-16T05:24:46.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ae185640 1 -- >> v2:172.21.15.114:3300/0 conn(0x7f47a8099cc0 msgr2=0x7f47a813e890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:24:46.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ae185640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f47a8099cc0 0x7f47a813e890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:24:46.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a80a1ab0 0x7f47a8072890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:46.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a80a1ab0 0x7f47a8072890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56476/0 (socket says 172.21.15.114:56476) 2024-09-16T05:24:46.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 1 -- 172.21.15.114:0/2726556836 learned_addr learned my addr 172.21.15.114:0/2726556836 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:46.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 10 monclient(hunting): get_auth_request con 0x7f47a80a1ab0 auth_method 0 2024-09-16T05:24:46.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:46.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:46.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:46.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:46.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:46.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:46.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 10 monclient(hunting): handle_auth_done global_id 104303 payload 274 2024-09-16T05:24:46.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 1 -- 172.21.15.114:0/2726556836 >> v2:172.21.15.114:3300/0 conn(0x7f47a8099cc0 msgr2=0x7f47a813e890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:24:46.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 1 --2- 172.21.15.114:0/2726556836 >> v2:172.21.15.114:3300/0 conn(0x7f47a8099cc0 0x7f47a813e890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:46.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 1 -- 172.21.15.114:0/2726556836 >> v1:172.21.15.149:6789/0 conn(0x7f47a8071da0 legacy=0x7f47a8072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:46.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 10 monclient: _finish_hunting 0 2024-09-16T05:24:46.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 1 monclient: found mon.noname-b 2024-09-16T05:24:46.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:46.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:24:46.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.062+0000 7f47ad183640 1 -- 172.21.15.114:0/2726556836 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a8073420 con 0x7f47a80a1ab0 2024-09-16T05:24:46.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ad183640 1 --2- 172.21.15.114:0/2726556836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a80a1ab0 0x7f47a8072890 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4798006fd0 tx=0x7f479801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1c5ce2b3bb0f0ae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:46.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ac982640 1 -- 172.21.15.114:0/2726556836 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47980b0070 con 0x7f47a80a1ab0 2024-09-16T05:24:46.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ac982640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:46.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ac982640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:24:46.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ac982640 10 monclient: dump: 2024-09-16T05:24:46.070 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:46.070 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:46.070 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:46.070 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:46.070 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:46.070 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:46.070 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:46.071 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:46.071 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:46.071 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:46.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ac982640 10 monclient: _finish_auth 0 2024-09-16T05:24:46.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ac982640 10 monclient: _check_auth_tickets 2024-09-16T05:24:46.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ac982640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:46.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 5 monclient: authenticate success, global_id 104303 2024-09-16T05:24:46.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:24:46.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ac982640 1 -- 172.21.15.114:0/2726556836 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47980ab600 con 0x7f47a80a1ab0 2024-09-16T05:24:46.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ac982640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:46.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 10 monclient: get_monmap_and_config success 2024-09-16T05:24:46.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 4 set_mon_vals no callback set 2024-09-16T05:24:46.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ac982640 1 -- 172.21.15.114:0/2726556836 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47980ab9d0 con 0x7f47a80a1ab0 2024-09-16T05:24:46.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ac982640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:46.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ac982640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:46.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ac982640 10 monclient: dump: 2024-09-16T05:24:46.073 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:46.073 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:46.074 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:46.074 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:46.074 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:46.074 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:46.074 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:46.074 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:46.075 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:46.075 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:46.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 10 monclient: shutdown 2024-09-16T05:24:46.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:46.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 -- 172.21.15.114:0/2726556836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a80a1ab0 msgr2=0x7f47a8072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 --2- 172.21.15.114:0/2726556836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a80a1ab0 0x7f47a8072890 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4798006fd0 tx=0x7f479801c3e0 comp rx=0 tx=0).stop 2024-09-16T05:24:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 -- 172.21.15.114:0/2726556836 shutdown_connections 2024-09-16T05:24:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 --2- 172.21.15.114:0/2726556836 >> v2:172.21.15.114:3300/0 conn(0x7f47a8099cc0 0x7f47a813e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 --2- 172.21.15.114:0/2726556836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a80a1ab0 0x7f47a8072890 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 -- 172.21.15.114:0/2726556836 >> 172.21.15.114:0/2726556836 conn(0x7f47a80a1fb0 msgr2=0x7f47a80a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 -- 172.21.15.114:0/2726556836 shutdown_connections 2024-09-16T05:24:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 -- 172.21.15.114:0/2726556836 wait complete. 2024-09-16T05:24:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 10 monclient: build_initial_monmap 2024-09-16T05:24:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 10 monclient: monmap: 2024-09-16T05:24:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:24:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:24:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:24:46.077 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:46.077 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:46.077 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:24:46.077 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:24:46.077 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:24:46.077 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:46.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 librados: starting msgr at 2024-09-16T05:24:46.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 librados: starting objecter 2024-09-16T05:24:46.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 Processor -- start 2024-09-16T05:24:46.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 -- start start 2024-09-16T05:24:46.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 librados: setting wanted keys 2024-09-16T05:24:46.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 librados: calling monclient init 2024-09-16T05:24:46.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 10 monclient: init 2024-09-16T05:24:46.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:46.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:24:46.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47a8071da0 0x7f47a813ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:46.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 10 monclient(hunting): picked mon.noname-a con 0x7f47a8071da0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:46.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a8099cc0 0x7f47a808d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:46.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 10 monclient(hunting): picked mon.noname-c con 0x7f47a8099cc0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:46.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f47a80a1ab0 0x7f47a808d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:46.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ad984640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47a8071da0 msgr2=0x7f47a813ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:24:46.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 10 monclient(hunting): picked mon.noname-b con 0x7f47a80a1ab0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:24:46.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:46.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f47a8141800 con 0x7f47a8071da0 2024-09-16T05:24:46.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47ad984640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47a8071da0 0x7f47a813ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:24:46.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:46.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f47a80a12e0 con 0x7f47a80a1ab0 2024-09-16T05:24:46.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:46.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f47a8141980 con 0x7f47a8099cc0 2024-09-16T05:24:46.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:46.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.066+0000 7f47afbe8640 10 monclient(hunting): authenticate will time out at 1970.699967s 2024-09-16T05:24:46.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a8099cc0 0x7f47a808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:46.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a8099cc0 0x7f47a808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56486/0 (socket says 172.21.15.114:56486) 2024-09-16T05:24:46.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 1 -- 172.21.15.114:0/2704998583 learned_addr learned my addr 172.21.15.114:0/2704998583 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:46.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 10 monclient(hunting): get_auth_request con 0x7f47a8099cc0 auth_method 0 2024-09-16T05:24:46.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:46.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:46.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:46.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:46.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:46.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:46.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 10 monclient(hunting): handle_auth_done global_id 104306 payload 824 2024-09-16T05:24:46.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ae185640 1 --2- 172.21.15.114:0/2704998583 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f47a80a1ab0 0x7f47a808d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:46.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 1 -- 172.21.15.114:0/2704998583 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f47a80a1ab0 msgr2=0x7f47a808d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:46.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 1 --2- 172.21.15.114:0/2704998583 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f47a80a1ab0 0x7f47a808d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:46.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 1 -- 172.21.15.114:0/2704998583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47a8071da0 msgr2=0x7f47a813ddd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:24:46.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 1 --2- 172.21.15.114:0/2704998583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47a8071da0 0x7f47a813ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:46.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 10 monclient: _finish_hunting 0 2024-09-16T05:24:46.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 1 monclient: found mon.noname-c 2024-09-16T05:24:46.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:46.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:24:46.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 1 -- 172.21.15.114:0/2704998583 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a808e140 con 0x7f47a8099cc0 2024-09-16T05:24:46.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad183640 1 --2- 172.21.15.114:0/2704998583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a8099cc0 0x7f47a808d000 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f479801aa90 tx=0x7f4798019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:46.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 10 client.?.objecter ms_handle_connect 0x7f47a8099cc0 2024-09-16T05:24:46.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:24:46.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 1 -- 172.21.15.114:0/2704998583 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47980ba030 con 0x7f47a8099cc0 2024-09-16T05:24:46.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:46.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:24:46.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 10 monclient: dump: 2024-09-16T05:24:46.087 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:46.087 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:46.087 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:46.087 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:46.087 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:46.087 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:46.087 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:46.087 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:46.087 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:46.088 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:46.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:24:46.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:24:46.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:46.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 1 -- 172.21.15.114:0/2704998583 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47980044e0 con 0x7f47a8099cc0 2024-09-16T05:24:46.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47afbe8640 5 monclient: authenticate success, global_id 104306 2024-09-16T05:24:46.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47afbe8640 10 monclient: _renew_subs 2024-09-16T05:24:46.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47afbe8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:46.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47afbe8640 1 -- 172.21.15.114:0/2704998583 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47a80a42a0 con 0x7f47a8099cc0 2024-09-16T05:24:46.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:46.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 1 -- 172.21.15.114:0/2704998583 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47980b0040 con 0x7f47a8099cc0 2024-09-16T05:24:46.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:46.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:46.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 10 monclient: dump: 2024-09-16T05:24:46.089 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:46.089 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:46.089 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:46.089 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:46.089 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:46.089 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:46.089 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:46.089 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:46.089 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:46.089 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:46.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479ffff640 4 set_mon_vals no callback set 2024-09-16T05:24:46.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47afbe8640 10 client.104306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:24:46.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47afbe8640 10 monclient: _renew_subs 2024-09-16T05:24:46.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47afbe8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:46.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47afbe8640 1 -- 172.21.15.114:0/2704998583 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f47a8090430 con 0x7f47a8099cc0 2024-09-16T05:24:46.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47afbe8640 1 librados: init done 2024-09-16T05:24:46.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 1 -- 172.21.15.114:0/2704998583 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4798003620 con 0x7f47a8099cc0 2024-09-16T05:24:46.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47afbe8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:46.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47afbe8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:46.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47afbe8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:46.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 1 --2- 172.21.15.114:0/2704998583 >> v2:172.21.15.149:6812/1185531135 conn(0x7f478c0377f0 0x7f478c039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:46.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47afbe8640 1 -- 172.21.15.114:0/2704998583 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47a8072180 con 0x7f47a8099cc0 2024-09-16T05:24:46.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 1 -- 172.21.15.114:0/2704998583 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(690..690 src has 1..690) ==== 5965+0+0 (secure 0 0 0) 0x7f47980e3110 con 0x7f47a8099cc0 2024-09-16T05:24:46.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 10 client.104306.objecter ms_dispatch 0x7f47a8000ba0 osd_map(690..690 src has 1..690) 2024-09-16T05:24:46.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 3 client.104306.objecter handle_osd_map got epochs [690,690] > 0 2024-09-16T05:24:46.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 3 client.104306.objecter handle_osd_map decoding full epoch 690 2024-09-16T05:24:46.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 20 client.104306.objecter dump_active .. 0 homeless 2024-09-16T05:24:46.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad984640 1 --2- 172.21.15.114:0/2704998583 >> v2:172.21.15.149:6812/1185531135 conn(0x7f478c0377f0 0x7f478c039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:46.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad984640 10 monclient: get_auth_request con 0x7f478c0377f0 auth_method 0 2024-09-16T05:24:46.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f479e7fc640 10 client.104306.objecter ms_handle_connect 0x7f478c0377f0 2024-09-16T05:24:46.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.070+0000 7f47ad984640 1 --2- 172.21.15.114:0/2704998583 >> v2:172.21.15.149:6812/1185531135 conn(0x7f478c0377f0 0x7f478c039cb0 secure :-1 s=READY pgs=3607 cs=0 l=1 rev1=1 crypto rx=0x7f47a4003780 tx=0x7f47a4007a20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:46.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.074+0000 7f479e7fc640 1 -- 172.21.15.114:0/2704998583 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f47980c4020 con 0x7f47a8099cc0 2024-09-16T05:24:46.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.074+0000 7f479e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:46.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.074+0000 7f479e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:24:46.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.294+0000 7f47afbe8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:24:46.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.294+0000 7f47afbe8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:24:46.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.294+0000 7f47afbe8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:46.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.294+0000 7f47afbe8640 1 -- 172.21.15.114:0/2704998583 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f47a813af80 con 0x7f47a8099cc0 2024-09-16T05:24:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f479e7fc640 1 -- 172.21.15.114:0/2704998583 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f47a80a42a0 con 0x7f47a8099cc0 2024-09-16T05:24:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f479e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:24:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f479e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:24:46.295 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:24:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 10 librados: watch_flush enter 2024-09-16T05:24:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 10 librados: watch_flush exit 2024-09-16T05:24:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 20 client.104306.objecter shutdown clearing up homeless session... 2024-09-16T05:24:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 10 client.104306.objecter successfully canceled tick 2024-09-16T05:24:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 1 -- 172.21.15.114:0/2704998583 >> v2:172.21.15.149:6812/1185531135 conn(0x7f478c0377f0 msgr2=0x7f478c039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 1 --2- 172.21.15.114:0/2704998583 >> v2:172.21.15.149:6812/1185531135 conn(0x7f478c0377f0 0x7f478c039cb0 secure :-1 s=READY pgs=3607 cs=0 l=1 rev1=1 crypto rx=0x7f47a4003780 tx=0x7f47a4007a20 comp rx=0 tx=0).stop 2024-09-16T05:24:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 10 monclient: shutdown 2024-09-16T05:24:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 1 -- 172.21.15.114:0/2704998583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a8099cc0 msgr2=0x7f47a808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 1 --2- 172.21.15.114:0/2704998583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a8099cc0 0x7f47a808d000 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f479801aa90 tx=0x7f4798019180 comp rx=0 tx=0).stop 2024-09-16T05:24:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 1 -- 172.21.15.114:0/2704998583 shutdown_connections 2024-09-16T05:24:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 1 --2- 172.21.15.114:0/2704998583 >> v2:172.21.15.149:6812/1185531135 conn(0x7f478c0377f0 0x7f478c039cb0 unknown :-1 s=CLOSED pgs=3607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 1 --2- 172.21.15.114:0/2704998583 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f47a80a1ab0 0x7f47a808d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:46.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 1 --2- 172.21.15.114:0/2704998583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47a8099cc0 0x7f47a808d000 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:46.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 1 --2- 172.21.15.114:0/2704998583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47a8071da0 0x7f47a813ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:46.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 1 -- 172.21.15.114:0/2704998583 >> 172.21.15.114:0/2704998583 conn(0x7f47a80a1fb0 msgr2=0x7f47a8137a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:46.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 1 -- 172.21.15.114:0/2704998583 shutdown_connections 2024-09-16T05:24:46.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 1 -- 172.21.15.114:0/2704998583 wait complete. 2024-09-16T05:24:46.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:46.298+0000 7f47afbe8640 1 librados: shutdown 2024-09-16T05:24:49.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:24:49.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 10 monclient: get_monmap_and_config 2024-09-16T05:24:49.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 10 monclient: build_initial_monmap 2024-09-16T05:24:49.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 10 monclient: monmap: 2024-09-16T05:24:49.444 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:49.444 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:49.444 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:24:49.449589+0000 2024-09-16T05:24:49.444 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:24:49.449589+0000 2024-09-16T05:24:49.444 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:49.444 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:49.444 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:24:49.444 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:24:49.445 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:24:49.445 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:24:49.445 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:24:49.445 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:49.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 1 Processor -- start 2024-09-16T05:24:49.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 1 -- start start 2024-09-16T05:24:49.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 10 monclient: init 2024-09-16T05:24:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-16T05:24:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f24cc0a0ec0 0x7f24cc09dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 10 monclient(hunting): picked mon.noname-c con 0x7f24cc0a0ec0 addr v2:172.21.15.149:3300/0 2024-09-16T05:24:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f24cc09e000 addr v1:172.21.15.149:6789/0 2024-09-16T05:24:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24cc139640 0x7f24cc13ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 10 monclient(hunting): picked mon.noname-b con 0x7f24cc139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24cc09a760 con 0x7f24cc09e000 2024-09-16T05:24:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f24cc0a12a0 con 0x7f24cc0a0ec0 2024-09-16T05:24:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f24cc0994b0 con 0x7f24cc139640 2024-09-16T05:24:49.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24d1d79640 10 monclient(hunting): authenticate will time out at 1974.078766s 2024-09-16T05:24:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24cc139640 0x7f24cc13ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24cc139640 0x7f24cc13ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:44652/0 (socket says 172.21.15.114:44652) 2024-09-16T05:24:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 1 -- 172.21.15.114:0/2499608985 learned_addr learned my addr 172.21.15.114:0/2499608985 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 10 monclient(hunting): get_auth_request con 0x7f24cc139640 auth_method 0 2024-09-16T05:24:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cb7fe640 1 --2- 172.21.15.114:0/2499608985 >> v2:172.21.15.149:3300/0 conn(0x7f24cc0a0ec0 0x7f24cc09dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:49.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cb7fe640 10 monclient(hunting): get_auth_request con 0x7f24cc0a0ec0 auth_method 0 2024-09-16T05:24:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 10 monclient(hunting): handle_auth_done global_id 104309 payload 274 2024-09-16T05:24:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 1 -- 172.21.15.114:0/2499608985 >> v2:172.21.15.149:3300/0 conn(0x7f24cc0a0ec0 msgr2=0x7f24cc09dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 1 --2- 172.21.15.114:0/2499608985 >> v2:172.21.15.149:3300/0 conn(0x7f24cc0a0ec0 0x7f24cc09dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 1 -- 172.21.15.114:0/2499608985 >> v1:172.21.15.149:6789/0 conn(0x7f24cc09e000 legacy=0x7f24cc09e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 10 monclient: _finish_hunting 0 2024-09-16T05:24:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 1 monclient: found mon.noname-b 2024-09-16T05:24:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:24:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cbfff640 1 -- 172.21.15.114:0/2499608985 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24cc0a1870 con 0x7f24cc139640 2024-09-16T05:24:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.446+0000 7f24cb7fe640 1 --2- 172.21.15.114:0/2499608985 >> v2:172.21.15.149:3300/0 conn(0x7f24cc0a0ec0 0x7f24cc09dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:24:49.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24cbfff640 1 --2- 172.21.15.114:0/2499608985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24cc139640 0x7f24cc13ba30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009e50 tx=0x7f24c401c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0a4c3e3bfe1c2d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24ca7fc640 1 -- 172.21.15.114:0/2499608985 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c40a5750 con 0x7f24cc139640 2024-09-16T05:24:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24ca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:24:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24ca7fc640 10 monclient: dump: 2024-09-16T05:24:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:49.453 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:49.453 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24ca7fc640 10 monclient: _finish_auth 0 2024-09-16T05:24:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24ca7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:24:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 5 monclient: authenticate success, global_id 104309 2024-09-16T05:24:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24ca7fc640 1 -- 172.21.15.114:0/2499608985 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24c40a58f0 con 0x7f24cc139640 2024-09-16T05:24:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:24:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 10 monclient: get_monmap_and_config success 2024-09-16T05:24:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 4 set_mon_vals no callback set 2024-09-16T05:24:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24ca7fc640 1 -- 172.21.15.114:0/2499608985 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c40a5cc0 con 0x7f24cc139640 2024-09-16T05:24:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24ca7fc640 10 monclient: dump: 2024-09-16T05:24:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:49.455 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 10 monclient: shutdown 2024-09-16T05:24:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 1 -- 172.21.15.114:0/2499608985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24cc139640 msgr2=0x7f24cc13ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:49.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 1 --2- 172.21.15.114:0/2499608985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24cc139640 0x7f24cc13ba30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009e50 tx=0x7f24c401c980 comp rx=0 tx=0).stop 2024-09-16T05:24:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 1 -- 172.21.15.114:0/2499608985 shutdown_connections 2024-09-16T05:24:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 1 --2- 172.21.15.114:0/2499608985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24cc139640 0x7f24cc13ba30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 1 --2- 172.21.15.114:0/2499608985 >> v2:172.21.15.149:3300/0 conn(0x7f24cc0a0ec0 0x7f24cc09dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 1 -- 172.21.15.114:0/2499608985 >> 172.21.15.114:0/2499608985 conn(0x7f24cc0a1fb0 msgr2=0x7f24cc0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 1 -- 172.21.15.114:0/2499608985 shutdown_connections 2024-09-16T05:24:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 1 -- 172.21.15.114:0/2499608985 wait complete. 2024-09-16T05:24:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 10 monclient: build_initial_monmap 2024-09-16T05:24:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 10 monclient: monmap: 2024-09-16T05:24:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:24:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:24:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:24:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:49.456 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:24:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:24:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:24:49.457 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 1 librados: starting msgr at 2024-09-16T05:24:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 1 librados: starting objecter 2024-09-16T05:24:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 1 Processor -- start 2024-09-16T05:24:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 1 -- start start 2024-09-16T05:24:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 1 librados: setting wanted keys 2024-09-16T05:24:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 1 librados: calling monclient init 2024-09-16T05:24:49.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 10 monclient: init 2024-09-16T05:24:49.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.450+0000 7f24d1d79640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:24:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24cc09e000 0x7f24cc138f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 monclient(hunting): picked mon.noname-b con 0x7f24cc09e000 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:24:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24cc0a0ec0 0x7f24cc08f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 monclient(hunting): picked mon.noname-a con 0x7f24cc0a0ec0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24cc139640 0x7f24cc08f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 monclient(hunting): picked mon.noname-c con 0x7f24cc139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc13e910 con 0x7f24cc0a0ec0 2024-09-16T05:24:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc13e790 con 0x7f24cc09e000 2024-09-16T05:24:49.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f24cc13ea90 con 0x7f24cc139640 2024-09-16T05:24:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 monclient(hunting): authenticate will time out at 1974.084087s 2024-09-16T05:24:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24caffd640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24cc0a0ec0 msgr2=0x7f24cc08f2e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:24:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24caffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24cc0a0ec0 0x7f24cc08f2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:24:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24cc139640 0x7f24cc08f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24cc139640 0x7f24cc08f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:44660/0 (socket says 172.21.15.114:44660) 2024-09-16T05:24:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 1 -- 172.21.15.114:0/870291610 learned_addr learned my addr 172.21.15.114:0/870291610 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 10 monclient(hunting): get_auth_request con 0x7f24cc139640 auth_method 0 2024-09-16T05:24:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cb7fe640 1 --2- 172.21.15.114:0/870291610 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24cc09e000 0x7f24cc138f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 10 monclient(hunting): handle_auth_done global_id 104312 payload 824 2024-09-16T05:24:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 1 -- 172.21.15.114:0/870291610 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24cc09e000 msgr2=0x7f24cc138f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 1 --2- 172.21.15.114:0/870291610 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24cc09e000 0x7f24cc138f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 1 -- 172.21.15.114:0/870291610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24cc0a0ec0 msgr2=0x7f24cc08f2e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:24:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 1 --2- 172.21.15.114:0/870291610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24cc0a0ec0 0x7f24cc08f2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 10 monclient: _finish_hunting 0 2024-09-16T05:24:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 1 monclient: found mon.noname-c 2024-09-16T05:24:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:24:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 1 -- 172.21.15.114:0/870291610 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24cc0a4eb0 con 0x7f24cc139640 2024-09-16T05:24:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cb7fe640 10 monclient: get_auth_request con 0x7f24cc09e000 auth_method 0 2024-09-16T05:24:49.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cb7fe640 1 --2- 172.21.15.114:0/870291610 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24cc09e000 0x7f24cc138f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:24:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cbfff640 1 --2- 172.21.15.114:0/870291610 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24cc139640 0x7f24cc08f820 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f24c401a5b0 tx=0x7f24c4019720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 10 client.?.objecter ms_handle_connect 0x7f24cc139640 2024-09-16T05:24:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:24:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 1 -- 172.21.15.114:0/870291610 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c40bc070 con 0x7f24cc139640 2024-09-16T05:24:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:24:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 10 monclient: dump: 2024-09-16T05:24:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:49.466 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 10 monclient: _finish_auth 0 2024-09-16T05:24:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 10 monclient: _check_auth_tickets 2024-09-16T05:24:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 1 -- 172.21.15.114:0/870291610 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24c40b8640 con 0x7f24cc139640 2024-09-16T05:24:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 1 -- 172.21.15.114:0/870291610 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c40b8a10 con 0x7f24cc139640 2024-09-16T05:24:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 10 monclient: dump: 2024-09-16T05:24:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:49.466 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:49.467 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24c9ffb640 4 set_mon_vals no callback set 2024-09-16T05:24:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 5 monclient: authenticate success, global_id 104312 2024-09-16T05:24:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 monclient: _renew_subs 2024-09-16T05:24:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 1 -- 172.21.15.114:0/870291610 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24cc092100 con 0x7f24cc139640 2024-09-16T05:24:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 client.104312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:24:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 monclient: _renew_subs 2024-09-16T05:24:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 1 -- 172.21.15.114:0/870291610 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f24cc092b60 con 0x7f24cc139640 2024-09-16T05:24:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 1 librados: init done 2024-09-16T05:24:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 1 -- 172.21.15.114:0/870291610 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f24c40b8bb0 con 0x7f24cc139640 2024-09-16T05:24:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24d1d79640 1 -- 172.21.15.114:0/870291610 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24cc09dac0 con 0x7f24cc139640 2024-09-16T05:24:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 1 --2- 172.21.15.114:0/870291610 >> v2:172.21.15.149:6812/1185531135 conn(0x7f24b8037820 0x7f24b8039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 1 -- 172.21.15.114:0/870291610 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(690..690 src has 1..690) ==== 5965+0+0 (secure 0 0 0) 0x7f24c40e3a60 con 0x7f24cc139640 2024-09-16T05:24:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 10 client.104312.objecter ms_dispatch 0x7f24cc000ba0 osd_map(690..690 src has 1..690) 2024-09-16T05:24:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 3 client.104312.objecter handle_osd_map got epochs [690,690] > 0 2024-09-16T05:24:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 3 client.104312.objecter handle_osd_map decoding full epoch 690 2024-09-16T05:24:49.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 20 client.104312.objecter dump_active .. 0 homeless 2024-09-16T05:24:49.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cb7fe640 1 --2- 172.21.15.114:0/870291610 >> v2:172.21.15.149:6812/1185531135 conn(0x7f24b8037820 0x7f24b8039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:49.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cb7fe640 10 monclient: get_auth_request con 0x7f24b8037820 auth_method 0 2024-09-16T05:24:49.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24b3fff640 10 client.104312.objecter ms_handle_connect 0x7f24b8037820 2024-09-16T05:24:49.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.454+0000 7f24cb7fe640 1 --2- 172.21.15.114:0/870291610 >> v2:172.21.15.149:6812/1185531135 conn(0x7f24b8037820 0x7f24b8039ce0 secure :-1 s=READY pgs=3608 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009980 tx=0x7f24bc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:49.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.458+0000 7f24b3fff640 1 -- 172.21.15.114:0/870291610 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f24c40c5070 con 0x7f24cc139640 2024-09-16T05:24:49.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.458+0000 7f24b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:49.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.458+0000 7f24b3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:24:49.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.682+0000 7f24d1d79640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:24:49.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.682+0000 7f24d1d79640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:24:49.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.682+0000 7f24d1d79640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:49.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.682+0000 7f24d1d79640 1 -- 172.21.15.114:0/870291610 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f24cc0727d0 con 0x7f24cc139640 2024-09-16T05:24:49.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24b3fff640 1 -- 172.21.15.114:0/870291610 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f24cc0a4eb0 con 0x7f24cc139640 2024-09-16T05:24:49.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:24:49.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24b3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:24:49.683 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:24:49.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 10 librados: watch_flush enter 2024-09-16T05:24:49.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 10 librados: watch_flush exit 2024-09-16T05:24:49.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 20 client.104312.objecter shutdown clearing up homeless session... 2024-09-16T05:24:49.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 10 client.104312.objecter successfully canceled tick 2024-09-16T05:24:49.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 1 -- 172.21.15.114:0/870291610 >> v2:172.21.15.149:6812/1185531135 conn(0x7f24b8037820 msgr2=0x7f24b8039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:49.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 1 --2- 172.21.15.114:0/870291610 >> v2:172.21.15.149:6812/1185531135 conn(0x7f24b8037820 0x7f24b8039ce0 secure :-1 s=READY pgs=3608 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009980 tx=0x7f24bc008040 comp rx=0 tx=0).stop 2024-09-16T05:24:49.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 10 monclient: shutdown 2024-09-16T05:24:49.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:49.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 1 -- 172.21.15.114:0/870291610 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24cc139640 msgr2=0x7f24cc08f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:49.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 1 --2- 172.21.15.114:0/870291610 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24cc139640 0x7f24cc08f820 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f24c401a5b0 tx=0x7f24c4019720 comp rx=0 tx=0).stop 2024-09-16T05:24:49.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 1 -- 172.21.15.114:0/870291610 shutdown_connections 2024-09-16T05:24:49.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 1 --2- 172.21.15.114:0/870291610 >> v2:172.21.15.149:6812/1185531135 conn(0x7f24b8037820 0x7f24b8039ce0 unknown :-1 s=CLOSED pgs=3608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:49.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 1 --2- 172.21.15.114:0/870291610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24cc0a0ec0 0x7f24cc08f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:49.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 1 --2- 172.21.15.114:0/870291610 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24cc139640 0x7f24cc08f820 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:49.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 1 --2- 172.21.15.114:0/870291610 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24cc09e000 0x7f24cc138f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:49.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 1 -- 172.21.15.114:0/870291610 >> 172.21.15.114:0/870291610 conn(0x7f24cc0a1fb0 msgr2=0x7f24cc06f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:49.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 1 -- 172.21.15.114:0/870291610 shutdown_connections 2024-09-16T05:24:49.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 1 -- 172.21.15.114:0/870291610 wait complete. 2024-09-16T05:24:49.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:49.686+0000 7f24d1d79640 1 librados: shutdown 2024-09-16T05:24:52.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:24:52.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 10 monclient: get_monmap_and_config 2024-09-16T05:24:52.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 10 monclient: build_initial_monmap 2024-09-16T05:24:52.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 10 monclient: monmap: 2024-09-16T05:24:52.836 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:52.836 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:52.836 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:24:52.841594+0000 2024-09-16T05:24:52.836 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:24:52.841594+0000 2024-09-16T05:24:52.836 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:52.836 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:52.837 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:24:52.837 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:24:52.837 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:24:52.837 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:24:52.837 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:24:52.837 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:52.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 1 Processor -- start 2024-09-16T05:24:52.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 1 -- start start 2024-09-16T05:24:52.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 10 monclient: init 2024-09-16T05:24:52.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:52.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T05:24:52.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcecc098500 addr v1:172.21.15.114:6789/0 2024-09-16T05:24:52.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcecc1368a0 0x7fcecc138c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:52.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 10 monclient(hunting): picked mon.noname-b con 0x7fcecc1368a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fcecc1391d0 0x7fcecc13b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 10 monclient(hunting): picked mon.noname-c con 0x7fcecc1391d0 addr v2:172.21.15.149:3300/0 2024-09-16T05:24:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcecc09db80 con 0x7fcecc098500 2024-09-16T05:24:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fcecc09e070 con 0x7fcecc1391d0 2024-09-16T05:24:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fcecc098ff0 con 0x7fcecc1368a0 2024-09-16T05:24:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced2aa7640 10 monclient(hunting): authenticate will time out at 1977.470747s 2024-09-16T05:24:52.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced0843640 1 -- >> v1:172.21.15.114:6789/0 conn(0x7fcecc098500 legacy=0x7fcecc0988e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T05:24:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcecc1368a0 0x7fcecc138c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcecc1368a0 0x7fcecc138c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:44670/0 (socket says 172.21.15.114:44670) 2024-09-16T05:24:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 1 -- 172.21.15.114:0/111790254 learned_addr learned my addr 172.21.15.114:0/111790254 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 10 monclient(hunting): get_auth_request con 0x7fcecc1368a0 auth_method 0 2024-09-16T05:24:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:52.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:52.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:52.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:52.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced1044640 1 --2- 172.21.15.114:0/111790254 >> v2:172.21.15.149:3300/0 conn(0x7fcecc1391d0 0x7fcecc13b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:52.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced1044640 10 monclient(hunting): get_auth_request con 0x7fcecc1391d0 auth_method 0 2024-09-16T05:24:52.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced1044640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:52.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced1044640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fced1044640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 10 monclient(hunting): handle_auth_done global_id 104315 payload 274 2024-09-16T05:24:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 1 -- 172.21.15.114:0/111790254 >> v2:172.21.15.149:3300/0 conn(0x7fcecc1391d0 msgr2=0x7fcecc13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:24:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 1 --2- 172.21.15.114:0/111790254 >> v2:172.21.15.149:3300/0 conn(0x7fcecc1391d0 0x7fcecc13b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 1 -- 172.21.15.114:0/111790254 >> v1:172.21.15.114:6789/0 conn(0x7fcecc098500 legacy=0x7fcecc0988e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:24:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 10 monclient: _finish_hunting 0 2024-09-16T05:24:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 1 monclient: found mon.noname-b 2024-09-16T05:24:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:52.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:24:52.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.838+0000 7fcecbfff640 1 -- 172.21.15.114:0/111790254 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcecc09e2d0 con 0x7fcecc1368a0 2024-09-16T05:24:52.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecbfff640 1 --2- 172.21.15.114:0/111790254 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcecc1368a0 0x7fcecc138c90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fceb4006fd0 tx=0x7fceb401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d45b6ffa2cd4c46 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:52.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecb7fe640 1 -- 172.21.15.114:0/111790254 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb40b0070 con 0x7fcecc1368a0 2024-09-16T05:24:52.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:52.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:24:52.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecb7fe640 10 monclient: dump: 2024-09-16T05:24:52.853 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:52.853 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:52.854 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecb7fe640 10 monclient: _finish_auth 0 2024-09-16T05:24:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecb7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:24:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecb7fe640 1 -- 172.21.15.114:0/111790254 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb40ab600 con 0x7fcecc1368a0 2024-09-16T05:24:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 5 monclient: authenticate success, global_id 104315 2024-09-16T05:24:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:24:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 10 monclient: get_monmap_and_config success 2024-09-16T05:24:52.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecb7fe640 1 -- 172.21.15.114:0/111790254 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb40ab9d0 con 0x7fcecc1368a0 2024-09-16T05:24:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 4 set_mon_vals no callback set 2024-09-16T05:24:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecb7fe640 10 monclient: dump: 2024-09-16T05:24:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:52.856 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:52.857 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 10 monclient: shutdown 2024-09-16T05:24:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 -- 172.21.15.114:0/111790254 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcecc1368a0 msgr2=0x7fcecc138c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 --2- 172.21.15.114:0/111790254 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcecc1368a0 0x7fcecc138c90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fceb4006fd0 tx=0x7fceb401c3c0 comp rx=0 tx=0).stop 2024-09-16T05:24:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 -- 172.21.15.114:0/111790254 shutdown_connections 2024-09-16T05:24:52.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 --2- 172.21.15.114:0/111790254 >> v2:172.21.15.149:3300/0 conn(0x7fcecc1391d0 0x7fcecc13b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:52.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 --2- 172.21.15.114:0/111790254 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcecc1368a0 0x7fcecc138c90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:52.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 -- 172.21.15.114:0/111790254 >> 172.21.15.114:0/111790254 conn(0x7fcecc0a1fb0 msgr2=0x7fcecc0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:52.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 -- 172.21.15.114:0/111790254 shutdown_connections 2024-09-16T05:24:52.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 -- 172.21.15.114:0/111790254 wait complete. 2024-09-16T05:24:52.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 10 monclient: build_initial_monmap 2024-09-16T05:24:52.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 10 monclient: monmap: 2024-09-16T05:24:52.858 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:52.858 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:24:52.858 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:24:52.858 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:24:52.858 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:52.859 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:52.859 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:24:52.859 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:24:52.859 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:24:52.859 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:52.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 librados: starting msgr at 2024-09-16T05:24:52.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 librados: starting objecter 2024-09-16T05:24:52.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 Processor -- start 2024-09-16T05:24:52.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 -- start start 2024-09-16T05:24:52.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 librados: setting wanted keys 2024-09-16T05:24:52.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 librados: calling monclient init 2024-09-16T05:24:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 10 monclient: init 2024-09-16T05:24:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:24:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcecc098500 0x7fcecc097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 10 monclient(hunting): picked mon.noname-b con 0x7fcecc098500 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:24:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcecc1368a0 0x7fcecc13fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 10 monclient(hunting): picked mon.noname-c con 0x7fcecc1368a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcecc1391d0 0x7fcecc140170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 10 monclient(hunting): picked mon.noname-a con 0x7fcecc1391d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:24:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcecc099fa0 con 0x7fcecc1391d0 2024-09-16T05:24:52.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:52.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fcecc099e20 con 0x7fcecc098500 2024-09-16T05:24:52.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:52.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fcecc13bb00 con 0x7fcecc1368a0 2024-09-16T05:24:52.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:52.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced2aa7640 10 monclient(hunting): authenticate will time out at 1977.475807s 2024-09-16T05:24:52.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced1044640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcecc1391d0 msgr2=0x7fcecc140170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:24:52.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fced1044640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcecc1391d0 0x7fcecc140170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:24:52.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcecc1368a0 0x7fcecc13fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:52.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcecc1368a0 0x7fcecc13fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:44678/0 (socket says 172.21.15.114:44678) 2024-09-16T05:24:52.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecbfff640 1 -- 172.21.15.114:0/2284523253 learned_addr learned my addr 172.21.15.114:0/2284523253 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:52.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecbfff640 10 monclient(hunting): get_auth_request con 0x7fcecc1368a0 auth_method 0 2024-09-16T05:24:52.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:52.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:52.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.842+0000 7fcecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:52.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcecbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:24:52.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcecbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:24:52.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcecbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:24:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced0843640 1 --2- 172.21.15.114:0/2284523253 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcecc098500 0x7fcecc097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced0843640 10 monclient(hunting): get_auth_request con 0x7fcecc098500 auth_method 0 2024-09-16T05:24:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced0843640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced0843640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced0843640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcecbfff640 10 monclient(hunting): handle_auth_done global_id 104318 payload 824 2024-09-16T05:24:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcecbfff640 1 -- 172.21.15.114:0/2284523253 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcecc098500 msgr2=0x7fcecc097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:52.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcecbfff640 1 --2- 172.21.15.114:0/2284523253 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcecc098500 0x7fcecc097bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcecbfff640 1 -- 172.21.15.114:0/2284523253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcecc1391d0 msgr2=0x7fcecc140170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:24:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcecbfff640 1 --2- 172.21.15.114:0/2284523253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcecc1391d0 0x7fcecc140170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcecbfff640 10 monclient: _finish_hunting 0 2024-09-16T05:24:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcecbfff640 1 monclient: found mon.noname-c 2024-09-16T05:24:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcecbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:24:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcecbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:24:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcecbfff640 1 -- 172.21.15.114:0/2284523253 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcecc08e390 con 0x7fcecc1368a0 2024-09-16T05:24:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcecbfff640 1 --2- 172.21.15.114:0/2284523253 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcecc1368a0 0x7fcecc13fc30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fceb4006fd0 tx=0x7fceb4019160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 10 client.?.objecter ms_handle_connect 0x7fcecc1368a0 2024-09-16T05:24:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:24:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 1 -- 172.21.15.114:0/2284523253 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb40b0040 con 0x7fcecc1368a0 2024-09-16T05:24:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:24:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 10 monclient: dump: 2024-09-16T05:24:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:52.865 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:52.866 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 10 monclient: _finish_auth 0 2024-09-16T05:24:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 10 monclient: _check_auth_tickets 2024-09-16T05:24:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:24:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced2aa7640 5 monclient: authenticate success, global_id 104318 2024-09-16T05:24:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 1 -- 172.21.15.114:0/2284523253 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb4005ba0 con 0x7fcecc1368a0 2024-09-16T05:24:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced2aa7640 10 monclient: _renew_subs 2024-09-16T05:24:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced2aa7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced2aa7640 1 -- 172.21.15.114:0/2284523253 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcecc138ca0 con 0x7fcecc1368a0 2024-09-16T05:24:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:24:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 1 -- 172.21.15.114:0/2284523253 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb40ab7b0 con 0x7fcecc1368a0 2024-09-16T05:24:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:24:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:24:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 10 monclient: dump: 2024-09-16T05:24:52.867 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:24:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:24:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:24:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:24:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:24:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:24:52.868 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcecaffd640 4 set_mon_vals no callback set 2024-09-16T05:24:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced2aa7640 10 client.104318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:24:52.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced2aa7640 10 monclient: _renew_subs 2024-09-16T05:24:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced2aa7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced2aa7640 1 -- 172.21.15.114:0/2284523253 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcecc08d000 con 0x7fcecc1368a0 2024-09-16T05:24:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced2aa7640 1 librados: init done 2024-09-16T05:24:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced2aa7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced2aa7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced2aa7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 1 -- 172.21.15.114:0/2284523253 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fceb40ab950 con 0x7fcecc1368a0 2024-09-16T05:24:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced2aa7640 1 -- 172.21.15.114:0/2284523253 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcecc0988e0 con 0x7fcecc1368a0 2024-09-16T05:24:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 1 --2- 172.21.15.114:0/2284523253 >> v2:172.21.15.149:6812/1185531135 conn(0x7fceac0377c0 0x7fceac039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 1 -- 172.21.15.114:0/2284523253 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(690..690 src has 1..690) ==== 5965+0+0 (secure 0 0 0) 0x7fceb40e2490 con 0x7fcecc1368a0 2024-09-16T05:24:52.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 10 client.104318.objecter ms_dispatch 0x7fcecc000ba0 osd_map(690..690 src has 1..690) 2024-09-16T05:24:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 3 client.104318.objecter handle_osd_map got epochs [690,690] > 0 2024-09-16T05:24:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 3 client.104318.objecter handle_osd_map decoding full epoch 690 2024-09-16T05:24:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 20 client.104318.objecter dump_active .. 0 homeless 2024-09-16T05:24:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced0843640 1 --2- 172.21.15.114:0/2284523253 >> v2:172.21.15.149:6812/1185531135 conn(0x7fceac0377c0 0x7fceac039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced0843640 10 monclient: get_auth_request con 0x7fceac0377c0 auth_method 0 2024-09-16T05:24:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fcec97fa640 10 client.104318.objecter ms_handle_connect 0x7fceac0377c0 2024-09-16T05:24:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.846+0000 7fced0843640 1 --2- 172.21.15.114:0/2284523253 >> v2:172.21.15.149:6812/1185531135 conn(0x7fceac0377c0 0x7fceac039c80 secure :-1 s=READY pgs=3609 cs=0 l=1 rev1=1 crypto rx=0x7fcebc006fd0 tx=0x7fcebc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:24:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.850+0000 7fcec97fa640 1 -- 172.21.15.114:0/2284523253 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fceb40160d0 con 0x7fcecc1368a0 2024-09-16T05:24:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.850+0000 7fcec97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:24:52.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:52.850+0000 7fcec97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:24:53.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:24:53.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:24:53.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:24:53.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 1 -- 172.21.15.114:0/2284523253 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcecc0008e0 con 0x7fcecc1368a0 2024-09-16T05:24:53.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fcec97fa640 1 -- 172.21.15.114:0/2284523253 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fceb40b2320 con 0x7fcecc1368a0 2024-09-16T05:24:53.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fcec97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:24:53.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fcec97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:24:53.060 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:24:53.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 10 librados: watch_flush enter 2024-09-16T05:24:53.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 10 librados: watch_flush exit 2024-09-16T05:24:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 20 client.104318.objecter shutdown clearing up homeless session... 2024-09-16T05:24:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 10 client.104318.objecter successfully canceled tick 2024-09-16T05:24:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 1 -- 172.21.15.114:0/2284523253 >> v2:172.21.15.149:6812/1185531135 conn(0x7fceac0377c0 msgr2=0x7fceac039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 1 --2- 172.21.15.114:0/2284523253 >> v2:172.21.15.149:6812/1185531135 conn(0x7fceac0377c0 0x7fceac039c80 secure :-1 s=READY pgs=3609 cs=0 l=1 rev1=1 crypto rx=0x7fcebc006fd0 tx=0x7fcebc008040 comp rx=0 tx=0).stop 2024-09-16T05:24:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 10 monclient: shutdown 2024-09-16T05:24:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:24:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 1 -- 172.21.15.114:0/2284523253 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcecc1368a0 msgr2=0x7fcecc13fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 1 --2- 172.21.15.114:0/2284523253 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcecc1368a0 0x7fcecc13fc30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fceb4006fd0 tx=0x7fceb4019160 comp rx=0 tx=0).stop 2024-09-16T05:24:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 1 -- 172.21.15.114:0/2284523253 shutdown_connections 2024-09-16T05:24:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 1 --2- 172.21.15.114:0/2284523253 >> v2:172.21.15.149:6812/1185531135 conn(0x7fceac0377c0 0x7fceac039c80 unknown :-1 s=CLOSED pgs=3609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 1 --2- 172.21.15.114:0/2284523253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcecc1391d0 0x7fcecc140170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:53.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 1 --2- 172.21.15.114:0/2284523253 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcecc1368a0 0x7fcecc13fc30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:53.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 1 --2- 172.21.15.114:0/2284523253 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcecc098500 0x7fcecc097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:53.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 1 -- 172.21.15.114:0/2284523253 >> 172.21.15.114:0/2284523253 conn(0x7fcecc0a1fb0 msgr2=0x7fcecc138480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:24:53.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 1 -- 172.21.15.114:0/2284523253 shutdown_connections 2024-09-16T05:24:53.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 1 -- 172.21.15.114:0/2284523253 wait complete. 2024-09-16T05:24:53.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:53.062+0000 7fced2aa7640 1 librados: shutdown 2024-09-16T05:24:56.079 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:24:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 10 monclient: get_monmap_and_config 2024-09-16T05:24:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 10 monclient: build_initial_monmap 2024-09-16T05:24:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 10 monclient: monmap: 2024-09-16T05:24:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:24:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:24:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:24:56.213858+0000 2024-09-16T05:24:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:24:56.213858+0000 2024-09-16T05:24:56.208 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:24:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:24:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:24:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:24:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:24:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:24:56.209 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:24:56.209 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:24:56.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 1 Processor -- start 2024-09-16T05:24:56.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 1 -- start start 2024-09-16T05:24:56.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 10 monclient: init 2024-09-16T05:24:56.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 10 monclient: _reopen_session rank -1 2024-09-16T05:24:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-16T05:24:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19bc137730 addr v1:172.21.15.114:6789/0 2024-09-16T05:24:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19bc0a5d00 addr v1:172.21.15.149:6789/0 2024-09-16T05:24:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 0x7f19bc13ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 10 monclient(hunting): picked mon.noname-c con 0x7f19bc13dbe0 addr v2:172.21.15.149:3300/0 2024-09-16T05:24:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19bc0a55e0 con 0x7f19bc137730 2024-09-16T05:24:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19bc0a3710 con 0x7f19bc0a5d00 2024-09-16T05:24:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f19bc09c450 con 0x7f19bc13dbe0 2024-09-16T05:24:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19c178c640 10 monclient(hunting): authenticate will time out at 1980.843082s 2024-09-16T05:24:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19baffd640 1 -- >> v1:172.21.15.114:6789/0 conn(0x7f19bc137730 legacy=0x7f19bc137b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T05:24:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19bb7fe640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 0x7f19bc13ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19bb7fe640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 0x7f19bc13ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.114:44960/0 (socket says 172.21.15.114:44960) 2024-09-16T05:24:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19ba7fc640 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f19bc0a5d00 0x7f19bc13c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.114:59660/0 (socket says 172.21.15.114:59660) 2024-09-16T05:24:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19bb7fe640 1 -- 172.21.15.114:0/1033212319 learned_addr learned my addr 172.21.15.114:0/1033212319 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:24:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19bb7fe640 10 monclient(hunting): get_auth_request con 0x7f19bc13dbe0 auth_method 0 2024-09-16T05:24:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19bb7fe640 1 -- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 msgr2=0x7f19bc13ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:24:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19bb7fe640 1 -- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 msgr2=0x7f19bc13ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:24:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19bb7fe640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 0x7f19bc13ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:24:56.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.210+0000 7f19bb7fe640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 0x7f19bc13ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.414+0000 7f19baffd640 1 -- 172.21.15.114:0/1033212319 >> v1:172.21.15.114:6789/0 conn(0x7f19bc137730 legacy=0x7f19bc137b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.414+0000 7f19bb7fe640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 0x7f19bc13ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.414+0000 7f19bb7fe640 10 monclient(hunting): get_auth_request con 0x7f19bc13dbe0 auth_method 0 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.414+0000 7f19bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.414+0000 7f19bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.414+0000 7f19bb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.414+0000 7f19bb7fe640 1 -- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 msgr2=0x7f19bc13ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.414+0000 7f19bb7fe640 1 -- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 msgr2=0x7f19bc13ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:24:56.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.414+0000 7f19bb7fe640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 0x7f19bc13ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:24:56.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.414+0000 7f19bb7fe640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 0x7f19bc13ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:24:56.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.818+0000 7f19baffd640 1 -- 172.21.15.114:0/1033212319 >> v1:172.21.15.114:6789/0 conn(0x7f19bc137730 legacy=0x7f19bc137b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T05:24:56.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.818+0000 7f19bb7fe640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 0x7f19bc13ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:56.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.818+0000 7f19bb7fe640 10 monclient(hunting): get_auth_request con 0x7f19bc13dbe0 auth_method 0 2024-09-16T05:24:56.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.818+0000 7f19bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:56.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.818+0000 7f19bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:56.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.818+0000 7f19bb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:24:56.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.818+0000 7f19bb7fe640 1 -- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 msgr2=0x7f19bc13ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:24:56.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.818+0000 7f19bb7fe640 1 -- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 msgr2=0x7f19bc13ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:24:56.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.818+0000 7f19bb7fe640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 0x7f19bc13ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:24:56.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:56.818+0000 7f19bb7fe640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 0x7f19bc13ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:24:57.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:57.622+0000 7f19baffd640 1 -- 172.21.15.114:0/1033212319 >> v1:172.21.15.114:6789/0 conn(0x7f19bc137730 legacy=0x7f19bc137b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T05:24:57.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:57.622+0000 7f19bb7fe640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 0x7f19bc13ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:57.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:57.622+0000 7f19bb7fe640 10 monclient(hunting): get_auth_request con 0x7f19bc13dbe0 auth_method 0 2024-09-16T05:24:57.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:57.622+0000 7f19bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:57.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:57.622+0000 7f19bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:57.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:57.622+0000 7f19bb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:24:57.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:57.622+0000 7f19bb7fe640 1 -- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 msgr2=0x7f19bc13ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:24:57.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:57.622+0000 7f19bb7fe640 1 -- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 msgr2=0x7f19bc13ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:24:57.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:57.622+0000 7f19bb7fe640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 0x7f19bc13ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:24:57.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:57.622+0000 7f19bb7fe640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 0x7f19bc13ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:24:59.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 10 monclient(hunting): tick 2024-09-16T05:24:59.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:24:59.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 1 monclient(hunting): continuing hunt 2024-09-16T05:24:59.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:24:59.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 1 -- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 msgr2=0x7f19bc13ffd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:24:59.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.149:3300/0 conn(0x7f19bc13dbe0 0x7f19bc13ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:24:59.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 1 -- 172.21.15.114:0/1033212319 >> v1:172.21.15.149:6789/0 conn(0x7f19bc0a5d00 legacy=0x7f19bc13c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:24:59.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 1 -- 172.21.15.114:0/1033212319 >> v1:172.21.15.114:6789/0 conn(0x7f19bc137730 legacy=0x7f19bc137b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:24:59.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 10 monclient: _add_conns ranks=[3,2] 2024-09-16T05:24:59.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.114:3300/0 conn(0x7f19a80036d0 0x7f19a8005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:59.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f19a80036d0 addr v2:172.21.15.114:3300/0 2024-09-16T05:24:59.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 1 --2- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 0x7f19a8008660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:24:59.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f19a80061c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:24:59.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:59.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 1 -- 172.21.15.114:0/1033212319 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f19bc09c450 con 0x7f19a80036d0 2024-09-16T05:24:59.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 10 monclient(hunting): start opening mon connection 2024-09-16T05:24:59.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 1 -- 172.21.15.114:0/1033212319 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f19a8008d90 con 0x7f19a80061c0 2024-09-16T05:24:59.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19b8ff9640 10 monclient(hunting): _renew_subs 2024-09-16T05:24:59.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19baffd640 1 -- 172.21.15.114:0/1033212319 >> v2:172.21.15.114:3300/0 conn(0x7f19a80036d0 msgr2=0x7f19a8005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:24:59.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19baffd640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.114:3300/0 conn(0x7f19a80036d0 0x7f19a8005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:24:59.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19ba7fc640 1 --2- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 0x7f19a8008660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:59.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19ba7fc640 10 monclient(hunting): get_auth_request con 0x7f19a80061c0 auth_method 0 2024-09-16T05:24:59.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:59.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:59.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.210+0000 7f19ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:24:59.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.214+0000 7f19ba7fc640 1 -- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 msgr2=0x7f19a8008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:24:59.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.214+0000 7f19ba7fc640 1 -- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 msgr2=0x7f19a8008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:24:59.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.214+0000 7f19ba7fc640 1 --2- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 0x7f19a8008660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:24:59.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.214+0000 7f19ba7fc640 1 --2- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 0x7f19a8008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:24:59.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.414+0000 7f19baffd640 1 -- 172.21.15.114:0/1033212319 >> v2:172.21.15.114:3300/0 conn(0x7f19a80036d0 msgr2=0x7f19a8005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:24:59.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.414+0000 7f19baffd640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.114:3300/0 conn(0x7f19a80036d0 0x7f19a8005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:24:59.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.418+0000 7f19ba7fc640 1 --2- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 0x7f19a8008660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:59.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.418+0000 7f19ba7fc640 10 monclient(hunting): get_auth_request con 0x7f19a80061c0 auth_method 0 2024-09-16T05:24:59.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.418+0000 7f19ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:59.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.418+0000 7f19ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:59.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.418+0000 7f19ba7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:24:59.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.418+0000 7f19ba7fc640 1 -- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 msgr2=0x7f19a8008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:24:59.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.418+0000 7f19ba7fc640 1 -- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 msgr2=0x7f19a8008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:24:59.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.418+0000 7f19ba7fc640 1 --2- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 0x7f19a8008660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:24:59.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.418+0000 7f19ba7fc640 1 --2- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 0x7f19a8008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:24:59.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.818+0000 7f19baffd640 1 -- 172.21.15.114:0/1033212319 >> v2:172.21.15.114:3300/0 conn(0x7f19a80036d0 msgr2=0x7f19a8005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:24:59.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.818+0000 7f19baffd640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.114:3300/0 conn(0x7f19a80036d0 0x7f19a8005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:24:59.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.822+0000 7f19ba7fc640 1 --2- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 0x7f19a8008660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:24:59.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.822+0000 7f19ba7fc640 10 monclient(hunting): get_auth_request con 0x7f19a80061c0 auth_method 0 2024-09-16T05:24:59.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.822+0000 7f19ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:24:59.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.822+0000 7f19ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:24:59.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.822+0000 7f19ba7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:24:59.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.822+0000 7f19ba7fc640 1 -- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 msgr2=0x7f19a8008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:24:59.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.822+0000 7f19ba7fc640 1 -- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 msgr2=0x7f19a8008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:24:59.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.822+0000 7f19ba7fc640 1 --2- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 0x7f19a8008660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:24:59.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:24:59.822+0000 7f19ba7fc640 1 --2- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 0x7f19a8008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:25:00.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.622+0000 7f19baffd640 1 -- 172.21.15.114:0/1033212319 >> v2:172.21.15.114:3300/0 conn(0x7f19a80036d0 msgr2=0x7f19a8005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:25:00.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.622+0000 7f19baffd640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.114:3300/0 conn(0x7f19a80036d0 0x7f19a8005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:25:00.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 1 --2- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 0x7f19a8008660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:00.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 10 monclient(hunting): get_auth_request con 0x7f19a80061c0 auth_method 0 2024-09-16T05:25:00.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:25:00.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:25:00.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:25:00.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:25:00.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:25:00.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:25:00.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 10 monclient(hunting): handle_auth_done global_id 104321 payload 274 2024-09-16T05:25:00.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 1 -- 172.21.15.114:0/1033212319 >> v2:172.21.15.114:3300/0 conn(0x7f19a80036d0 msgr2=0x7f19a8005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:25:00.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 1 --2- 172.21.15.114:0/1033212319 >> v2:172.21.15.114:3300/0 conn(0x7f19a80036d0 0x7f19a8005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:00.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:25:00.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 1 monclient: found mon.noname-b 2024-09-16T05:25:00.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:25:00.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:25:00.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.626+0000 7f19ba7fc640 1 -- 172.21.15.114:0/1033212319 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19bc09be00 con 0x7f19a80061c0 2024-09-16T05:25:00.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19ba7fc640 1 --2- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 0x7f19a8008660 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f19b0007fd0 tx=0x7f19b001c9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45b75d527bc37f75 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:25:00.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19b9ffb640 1 -- 172.21.15.114:0/1033212319 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b00ab070 con 0x7f19a80061c0 2024-09-16T05:25:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19b9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:25:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19b9ffb640 10 monclient: dump: 2024-09-16T05:25:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:00.630 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19b9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:25:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19b9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:25:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:25:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19b9ffb640 1 -- 172.21.15.114:0/1033212319 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19b00a5790 con 0x7f19a80061c0 2024-09-16T05:25:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19b9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:25:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19c178c640 5 monclient: authenticate success, global_id 104321 2024-09-16T05:25:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19b9ffb640 1 -- 172.21.15.114:0/1033212319 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b00a5b40 con 0x7f19a80061c0 2024-09-16T05:25:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19c178c640 10 monclient: get_monmap_and_config success 2024-09-16T05:25:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19c178c640 4 set_mon_vals no callback set 2024-09-16T05:25:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19b9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:25:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.630+0000 7f19b9ffb640 10 monclient: dump: 2024-09-16T05:25:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:00.632 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:00.632 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:00.632 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:00.632 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:00.632 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:00.632 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:00.632 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:00.632 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:00.632 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:00.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 10 monclient: shutdown 2024-09-16T05:25:00.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:25:00.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 1 -- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 msgr2=0x7f19a8008660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:00.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 1 --2- 172.21.15.114:0/1033212319 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19a80061c0 0x7f19a8008660 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f19b0007fd0 tx=0x7f19b001c9d0 comp rx=0 tx=0).stop 2024-09-16T05:25:00.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19bb7fe640 1 -- 172.21.15.114:0/1033212319 reap_dead start 2024-09-16T05:25:00.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 1 -- 172.21.15.114:0/1033212319 shutdown_connections 2024-09-16T05:25:00.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 1 -- 172.21.15.114:0/1033212319 >> 172.21.15.114:0/1033212319 conn(0x7f19bc0a9c70 msgr2=0x7f19bc0aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:25:00.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 1 -- 172.21.15.114:0/1033212319 shutdown_connections 2024-09-16T05:25:00.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 1 -- 172.21.15.114:0/1033212319 wait complete. 2024-09-16T05:25:00.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 10 monclient: build_initial_monmap 2024-09-16T05:25:00.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 10 monclient: monmap: 2024-09-16T05:25:00.634 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:25:00.634 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:25:00.634 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:25:00.634 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:25:00.634 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:25:00.635 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:00.635 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:25:00.635 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:25:00.635 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:25:00.635 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:00.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 1 librados: starting msgr at 2024-09-16T05:25:00.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 1 librados: starting objecter 2024-09-16T05:25:00.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 1 Processor -- start 2024-09-16T05:25:00.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 1 -- start start 2024-09-16T05:25:00.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 1 librados: setting wanted keys 2024-09-16T05:25:00.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 1 librados: calling monclient init 2024-09-16T05:25:00.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.634+0000 7f19c178c640 10 monclient: init 2024-09-16T05:25:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient: _reopen_session rank -1 2024-09-16T05:25:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:25:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19bc132240 0x7f19bc132620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient(hunting): picked mon.noname-a con 0x7f19bc132240 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:25:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19bc132b60 0x7f19bc13b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient(hunting): picked mon.noname-b con 0x7f19bc132b60 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:25:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19bc13b610 0x7f19bc13baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient(hunting): picked mon.noname-c con 0x7f19bc13b610 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:25:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc142eb0 con 0x7f19bc132240 2024-09-16T05:25:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc142d30 con 0x7f19bc132b60 2024-09-16T05:25:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f19bc143030 con 0x7f19bc13b610 2024-09-16T05:25:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient(hunting): _renew_subs 2024-09-16T05:25:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19baffd640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19bc132240 msgr2=0x7f19bc132620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:25:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient(hunting): authenticate will time out at 1985.268254s 2024-09-16T05:25:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19baffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19bc132240 0x7f19bc132620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:25:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19bc13b610 0x7f19bc13baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19bc13b610 0x7f19bc13baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51070/0 (socket says 172.21.15.114:51070) 2024-09-16T05:25:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 1 -- 172.21.15.114:0/1069325389 learned_addr learned my addr 172.21.15.114:0/1069325389 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:25:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 10 monclient(hunting): get_auth_request con 0x7f19bc13b610 auth_method 0 2024-09-16T05:25:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:25:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:25:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:25:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:25:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:25:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19ba7fc640 1 --2- 172.21.15.114:0/1069325389 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19bc132b60 0x7f19bc13b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:25:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19ba7fc640 10 monclient(hunting): get_auth_request con 0x7f19bc132b60 auth_method 0 2024-09-16T05:25:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:25:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:25:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:25:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19ba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:25:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19ba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:25:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19ba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:25:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 10 monclient(hunting): handle_auth_done global_id 104324 payload 824 2024-09-16T05:25:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 1 -- 172.21.15.114:0/1069325389 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19bc132b60 msgr2=0x7f19bc13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 1 --2- 172.21.15.114:0/1069325389 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19bc132b60 0x7f19bc13b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 1 -- 172.21.15.114:0/1069325389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19bc132240 msgr2=0x7f19bc132620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:25:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 1 --2- 172.21.15.114:0/1069325389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19bc132240 0x7f19bc132620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:25:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 1 monclient: found mon.noname-c 2024-09-16T05:25:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:25:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:25:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 1 -- 172.21.15.114:0/1069325389 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19bc073840 con 0x7f19bc13b610 2024-09-16T05:25:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19bb7fe640 1 --2- 172.21.15.114:0/1069325389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19bc13b610 0x7f19bc13baa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f19a8006190 tx=0x7f19b4007d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:25:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 10 client.?.objecter ms_handle_connect 0x7f19bc13b610 2024-09-16T05:25:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:25:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 1 -- 172.21.15.114:0/1069325389 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b401b070 con 0x7f19bc13b610 2024-09-16T05:25:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:25:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 10 monclient: dump: 2024-09-16T05:25:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:00.644 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 10 monclient: _finish_auth 0 2024-09-16T05:25:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:25:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:25:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 5 monclient: authenticate success, global_id 104324 2024-09-16T05:25:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient: _renew_subs 2024-09-16T05:25:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 1 -- 172.21.15.114:0/1069325389 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19b4013a10 con 0x7f19bc13b610 2024-09-16T05:25:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:25:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 1 -- 172.21.15.114:0/1069325389 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19bc13cd30 con 0x7f19bc13b610 2024-09-16T05:25:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:25:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 1 -- 172.21.15.114:0/1069325389 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b4013e00 con 0x7f19bc13b610 2024-09-16T05:25:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:25:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 10 monclient: dump: 2024-09-16T05:25:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:00.646 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19b97fa640 4 set_mon_vals no callback set 2024-09-16T05:25:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 client.104324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:25:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient: _renew_subs 2024-09-16T05:25:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:25:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 1 -- 172.21.15.114:0/1069325389 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f19bc09e530 con 0x7f19bc13b610 2024-09-16T05:25:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 1 librados: init done 2024-09-16T05:25:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:25:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:25:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:25:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19c178c640 1 -- 172.21.15.114:0/1069325389 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19bc137b10 con 0x7f19bc13b610 2024-09-16T05:25:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.638+0000 7f19a37fe640 1 -- 172.21.15.114:0/1069325389 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f19b401e070 con 0x7f19bc13b610 2024-09-16T05:25:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.642+0000 7f19a37fe640 1 --2- 172.21.15.114:0/1069325389 >> v2:172.21.15.149:6812/1185531135 conn(0x7f19a8038620 0x7f19a803aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.642+0000 7f19a37fe640 1 -- 172.21.15.114:0/1069325389 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(691..691 src has 1..691) ==== 5978+0+0 (secure 0 0 0) 0x7f19b4049960 con 0x7f19bc13b610 2024-09-16T05:25:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.642+0000 7f19a37fe640 10 client.104324.objecter ms_dispatch 0x7f19bc000ba0 osd_map(691..691 src has 1..691) 2024-09-16T05:25:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.642+0000 7f19a37fe640 3 client.104324.objecter handle_osd_map got epochs [691,691] > 0 2024-09-16T05:25:00.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.642+0000 7f19a37fe640 3 client.104324.objecter handle_osd_map decoding full epoch 691 2024-09-16T05:25:00.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.642+0000 7f19a37fe640 20 client.104324.objecter dump_active .. 0 homeless 2024-09-16T05:25:00.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.642+0000 7f19baffd640 1 --2- 172.21.15.114:0/1069325389 >> v2:172.21.15.149:6812/1185531135 conn(0x7f19a8038620 0x7f19a803aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:00.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.642+0000 7f19baffd640 10 monclient: get_auth_request con 0x7f19a8038620 auth_method 0 2024-09-16T05:25:00.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.642+0000 7f19baffd640 1 --2- 172.21.15.114:0/1069325389 >> v2:172.21.15.149:6812/1185531135 conn(0x7f19a8038620 0x7f19a803aae0 secure :-1 s=READY pgs=3610 cs=0 l=1 rev1=1 crypto rx=0x7f19ac003990 tx=0x7f19ac00a390 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:25:00.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.642+0000 7f19a37fe640 10 client.104324.objecter ms_handle_connect 0x7f19a8038620 2024-09-16T05:25:00.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.646+0000 7f19a37fe640 1 -- 172.21.15.114:0/1069325389 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f19b4028350 con 0x7f19bc13b610 2024-09-16T05:25:00.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.646+0000 7f19a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:25:00.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.646+0000 7f19a37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:25:00.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.874+0000 7f19c178c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:25:00.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.874+0000 7f19c178c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:25:00.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.874+0000 7f19c178c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:25:00.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.874+0000 7f19c178c640 1 -- 172.21.15.114:0/1069325389 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f19bc0a9010 con 0x7f19bc13b610 2024-09-16T05:25:00.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.874+0000 7f19a37fe640 1 -- 172.21.15.114:0/1069325389 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f19b4017430 con 0x7f19bc13b610 2024-09-16T05:25:00.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.874+0000 7f19a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:25:00.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.874+0000 7f19a37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:25:00.873 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":72,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:25:00.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 10 librados: watch_flush enter 2024-09-16T05:25:00.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 10 librados: watch_flush exit 2024-09-16T05:25:00.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 20 client.104324.objecter shutdown clearing up homeless session... 2024-09-16T05:25:00.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 10 client.104324.objecter successfully canceled tick 2024-09-16T05:25:00.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 1 -- 172.21.15.114:0/1069325389 >> v2:172.21.15.149:6812/1185531135 conn(0x7f19a8038620 msgr2=0x7f19a803aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:00.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 1 --2- 172.21.15.114:0/1069325389 >> v2:172.21.15.149:6812/1185531135 conn(0x7f19a8038620 0x7f19a803aae0 secure :-1 s=READY pgs=3610 cs=0 l=1 rev1=1 crypto rx=0x7f19ac003990 tx=0x7f19ac00a390 comp rx=0 tx=0).stop 2024-09-16T05:25:00.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 10 monclient: shutdown 2024-09-16T05:25:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:25:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 1 -- 172.21.15.114:0/1069325389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19bc13b610 msgr2=0x7f19bc13baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 1 --2- 172.21.15.114:0/1069325389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19bc13b610 0x7f19bc13baa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f19a8006190 tx=0x7f19b4007d80 comp rx=0 tx=0).stop 2024-09-16T05:25:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 1 -- 172.21.15.114:0/1069325389 shutdown_connections 2024-09-16T05:25:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 1 --2- 172.21.15.114:0/1069325389 >> v2:172.21.15.149:6812/1185531135 conn(0x7f19a8038620 0x7f19a803aae0 unknown :-1 s=CLOSED pgs=3610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 1 --2- 172.21.15.114:0/1069325389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19bc13b610 0x7f19bc13baa0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 1 --2- 172.21.15.114:0/1069325389 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19bc132b60 0x7f19bc13b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 1 --2- 172.21.15.114:0/1069325389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19bc132240 0x7f19bc132620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 1 -- 172.21.15.114:0/1069325389 >> 172.21.15.114:0/1069325389 conn(0x7f19bc0a9c70 msgr2=0x7f19bc0a7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:25:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 1 -- 172.21.15.114:0/1069325389 shutdown_connections 2024-09-16T05:25:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 1 -- 172.21.15.114:0/1069325389 wait complete. 2024-09-16T05:25:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:00.878+0000 7f19c178c640 1 librados: shutdown 2024-09-16T05:25:00.892 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:25:00.892 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T05:25:00.892 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:25:01.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 10 monclient: get_monmap_and_config 2024-09-16T05:25:01.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 10 monclient: build_initial_monmap 2024-09-16T05:25:01.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 10 monclient: monmap: 2024-09-16T05:25:01.007 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:25:01.007 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:01.008 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:25:01.013173+0000 2024-09-16T05:25:01.008 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:25:01.013173+0000 2024-09-16T05:25:01.008 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:25:01.008 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.008 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:25:01.008 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:25:01.008 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:25:01.008 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:25:01.008 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:25:01.008 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 1 Processor -- start 2024-09-16T05:25:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 1 -- start start 2024-09-16T05:25:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 10 monclient: init 2024-09-16T05:25:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 10 monclient: _reopen_session rank -1 2024-09-16T05:25:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-16T05:25:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a040a0ec0 0x7f1a0409dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 10 monclient(hunting): picked mon.noname-a con 0x7f1a040a0ec0 addr v2:172.21.15.114:3300/0 2024-09-16T05:25:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a0409e000 0x7f1a0409e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 10 monclient(hunting): picked mon.noname-b con 0x7f1a0409e000 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:25:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a04138630 addr v1:172.21.15.149:6789/0 2024-09-16T05:25:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a0409e950 con 0x7f1a04138630 2024-09-16T05:25:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f1a040a12a0 con 0x7f1a040a0ec0 2024-09-16T05:25:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1a040994b0 con 0x7f1a0409e000 2024-09-16T05:25:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 10 monclient(hunting): _renew_subs 2024-09-16T05:25:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0c376640 10 monclient(hunting): authenticate will time out at 1985.642272s 2024-09-16T05:25:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0a112640 1 -- >> v2:172.21.15.114:3300/0 conn(0x7f1a040a0ec0 msgr2=0x7f1a0409dac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:25:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a0a112640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a040a0ec0 0x7f1a0409dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:25:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a0409e000 0x7f1a0409e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a0409e000 0x7f1a0409e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51072/0 (socket says 172.21.15.114:51072) 2024-09-16T05:25:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 1 -- 172.21.15.114:0/2501072223 learned_addr learned my addr 172.21.15.114:0/2501072223 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:25:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 10 monclient(hunting): get_auth_request con 0x7f1a0409e000 auth_method 0 2024-09-16T05:25:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:25:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:25:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:25:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:25:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:25:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:25:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 10 monclient(hunting): handle_auth_done global_id 104327 payload 274 2024-09-16T05:25:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 1 -- 172.21.15.114:0/2501072223 >> v2:172.21.15.114:3300/0 conn(0x7f1a040a0ec0 msgr2=0x7f1a0409dac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:25:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 1 --2- 172.21.15.114:0/2501072223 >> v2:172.21.15.114:3300/0 conn(0x7f1a040a0ec0 0x7f1a0409dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 1 -- 172.21.15.114:0/2501072223 >> v1:172.21.15.149:6789/0 conn(0x7f1a04138630 legacy=0x7f1a0413aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:25:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 10 monclient: _finish_hunting 0 2024-09-16T05:25:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 1 monclient: found mon.noname-b 2024-09-16T05:25:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:25:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:25:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.010+0000 7f1a09911640 1 -- 172.21.15.114:0/2501072223 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a0409a760 con 0x7f1a0409e000 2024-09-16T05:25:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09911640 1 --2- 172.21.15.114:0/2501072223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a0409e000 0x7f1a0409e410 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f19f8006fd0 tx=0x7f19f801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98a1e106f4f40921 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:25:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09110640 1 -- 172.21.15.114:0/2501072223 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f80b0070 con 0x7f1a0409e000 2024-09-16T05:25:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09110640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09110640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:25:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09110640 10 monclient: dump: 2024-09-16T05:25:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:01.020 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09110640 10 monclient: _finish_auth 0 2024-09-16T05:25:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09110640 10 monclient: _check_auth_tickets 2024-09-16T05:25:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09110640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:25:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09110640 1 -- 172.21.15.114:0/2501072223 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19f80ab600 con 0x7f1a0409e000 2024-09-16T05:25:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 5 monclient: authenticate success, global_id 104327 2024-09-16T05:25:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:25:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09110640 10 monclient: handle_config config(0 keys) 2024-09-16T05:25:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 10 monclient: get_monmap_and_config success 2024-09-16T05:25:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09110640 1 -- 172.21.15.114:0/2501072223 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f80ab9d0 con 0x7f1a0409e000 2024-09-16T05:25:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09110640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 4 set_mon_vals no callback set 2024-09-16T05:25:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09110640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:25:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09110640 10 monclient: dump: 2024-09-16T05:25:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:01.022 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 10 monclient: shutdown 2024-09-16T05:25:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:25:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 -- 172.21.15.114:0/2501072223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a0409e000 msgr2=0x7f1a0409e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 --2- 172.21.15.114:0/2501072223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a0409e000 0x7f1a0409e410 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f19f8006fd0 tx=0x7f19f801c470 comp rx=0 tx=0).stop 2024-09-16T05:25:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 -- 172.21.15.114:0/2501072223 shutdown_connections 2024-09-16T05:25:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 --2- 172.21.15.114:0/2501072223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a0409e000 0x7f1a0409e410 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 --2- 172.21.15.114:0/2501072223 >> v2:172.21.15.114:3300/0 conn(0x7f1a040a0ec0 0x7f1a0409dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 -- 172.21.15.114:0/2501072223 >> 172.21.15.114:0/2501072223 conn(0x7f1a040a1fb0 msgr2=0x7f1a040a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:25:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 -- 172.21.15.114:0/2501072223 shutdown_connections 2024-09-16T05:25:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 -- 172.21.15.114:0/2501072223 wait complete. 2024-09-16T05:25:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 10 monclient: build_initial_monmap 2024-09-16T05:25:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 10 monclient: monmap: 2024-09-16T05:25:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:25:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:25:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:25:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:25:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:25:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:25:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:25:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:25:01.024 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 librados: starting msgr at 2024-09-16T05:25:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 librados: starting objecter 2024-09-16T05:25:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 Processor -- start 2024-09-16T05:25:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 -- start start 2024-09-16T05:25:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 librados: setting wanted keys 2024-09-16T05:25:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 librados: calling monclient init 2024-09-16T05:25:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 10 monclient: init 2024-09-16T05:25:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 10 monclient: _reopen_session rank -1 2024-09-16T05:25:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:25:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a0409e000 0x7f1a0409c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 10 monclient(hunting): picked mon.noname-a con 0x7f1a0409e000 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:25:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1a040a0ec0 0x7f1a04070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 10 monclient(hunting): picked mon.noname-b con 0x7f1a040a0ec0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:25:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a04138630 0x7f1a04071000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 10 monclient(hunting): picked mon.noname-c con 0x7f1a04138630 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:25:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0409c3b0 con 0x7f1a0409e000 2024-09-16T05:25:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a112640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a0409e000 msgr2=0x7f1a0409c9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:25:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0409c230 con 0x7f1a040a0ec0 2024-09-16T05:25:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1a0413e960 con 0x7f1a04138630 2024-09-16T05:25:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 10 monclient(hunting): _renew_subs 2024-09-16T05:25:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a112640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a0409e000 0x7f1a0409c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:25:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0c376640 10 monclient(hunting): authenticate will time out at 1985.647421s 2024-09-16T05:25:01.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a04138630 0x7f1a04071000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:01.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a04138630 0x7f1a04071000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51080/0 (socket says 172.21.15.114:51080) 2024-09-16T05:25:01.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 1 -- 172.21.15.114:0/2967290580 learned_addr learned my addr 172.21.15.114:0/2967290580 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:25:01.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 10 monclient(hunting): get_auth_request con 0x7f1a04138630 auth_method 0 2024-09-16T05:25:01.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:25:01.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:25:01.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:25:01.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:25:01.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:25:01.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:25:01.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09911640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1a040a0ec0 0x7f1a04070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:01.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09911640 10 monclient(hunting): get_auth_request con 0x7f1a040a0ec0 auth_method 0 2024-09-16T05:25:01.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09911640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:25:01.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09911640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:25:01.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a09911640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:25:01.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 10 monclient(hunting): handle_auth_done global_id 104330 payload 824 2024-09-16T05:25:01.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 1 -- 172.21.15.114:0/2967290580 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1a040a0ec0 msgr2=0x7f1a04070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:01.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1a040a0ec0 0x7f1a04070ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:01.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 1 -- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a0409e000 msgr2=0x7f1a0409c9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:25:01.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a0409e000 0x7f1a0409c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:01.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 10 monclient: _finish_hunting 0 2024-09-16T05:25:01.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 1 monclient: found mon.noname-c 2024-09-16T05:25:01.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:25:01.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:25:01.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.014+0000 7f1a0a913640 1 -- 172.21.15.114:0/2967290580 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a040729b0 con 0x7f1a04138630 2024-09-16T05:25:01.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0a913640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a04138630 0x7f1a04071000 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f19f400c3f0 tx=0x7f19f400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:25:01.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 client.?.objecter ms_handle_connect 0x7f1a04138630 2024-09-16T05:25:01.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:25:01.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 1 -- 172.21.15.114:0/2967290580 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f4011790 con 0x7f1a04138630 2024-09-16T05:25:01.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:01.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:25:01.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 monclient: dump: 2024-09-16T05:25:01.029 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:01.030 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:01.030 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.030 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.030 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:01.030 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.030 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:01.030 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:01.030 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:01.030 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 monclient: _finish_auth 0 2024-09-16T05:25:01.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:25:01.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:25:01.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0c376640 5 monclient: authenticate success, global_id 104330 2024-09-16T05:25:01.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 1 -- 172.21.15.114:0/2967290580 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19f4017070 con 0x7f1a04138630 2024-09-16T05:25:01.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:25:01.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0c376640 10 monclient: _renew_subs 2024-09-16T05:25:01.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0c376640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:25:01.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 1 -- 172.21.15.114:0/2967290580 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f4011b70 con 0x7f1a04138630 2024-09-16T05:25:01.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0c376640 1 -- 172.21.15.114:0/2967290580 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a04073380 con 0x7f1a04138630 2024-09-16T05:25:01.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0890f640 4 set_mon_vals no callback set 2024-09-16T05:25:01.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:01.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:25:01.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 monclient: dump: 2024-09-16T05:25:01.031 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:01.031 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:01.031 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.031 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.031 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:01.032 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.032 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:01.032 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:01.032 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:01.032 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0c376640 10 client.104330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:25:01.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0c376640 10 monclient: _renew_subs 2024-09-16T05:25:01.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0c376640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:25:01.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0c376640 1 -- 172.21.15.114:0/2967290580 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a04071f10 con 0x7f1a04138630 2024-09-16T05:25:01.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0c376640 1 librados: init done 2024-09-16T05:25:01.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0c376640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:25:01.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0c376640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a04071540 0x7f1a0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0c376640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:01.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0c376640 1 -- 172.21.15.114:0/2967290580 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1a04069390 con 0x7f1a04071540 2024-09-16T05:25:01.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 1 -- 172.21.15.114:0/2967290580 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f19f4011d10 con 0x7f1a04138630 2024-09-16T05:25:01.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0a112640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a04071540 0x7f1a0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:01.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0a112640 10 monclient: get_auth_request con 0x7f1a04071540 auth_method 0 2024-09-16T05:25:01.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0a112640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:25:01.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0a112640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:25:01.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0a112640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:25:01.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0a112640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:25:01.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 1 --2- 172.21.15.114:0/2967290580 >> v2:172.21.15.149:6812/1185531135 conn(0x7f19f00377c0 0x7f19f0039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0a112640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:25:01.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0a112640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:25:01.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 1 -- 172.21.15.114:0/2967290580 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(691..691 src has 1..691) ==== 5978+0+0 (secure 0 0 0) 0x7f19f4048f20 con 0x7f1a04138630 2024-09-16T05:25:01.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 client.104330.objecter ms_dispatch 0x7f1a04000ba0 osd_map(691..691 src has 1..691) 2024-09-16T05:25:01.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 3 client.104330.objecter handle_osd_map got epochs [691,691] > 0 2024-09-16T05:25:01.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 3 client.104330.objecter handle_osd_map decoding full epoch 691 2024-09-16T05:25:01.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0a112640 10 monclient(hunting): handle_auth_done global_id 104333 payload 824 2024-09-16T05:25:01.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0a112640 1 -- 172.21.15.114:0/2967290580 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1a0409d550 con 0x7f1a04071540 2024-09-16T05:25:01.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 20 client.104330.objecter dump_active .. 0 homeless 2024-09-16T05:25:01.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 client.104330.objecter ms_handle_connect 0x7f1a04071540 2024-09-16T05:25:01.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 client.104330.objecter resend_mon_ops 2024-09-16T05:25:01.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a0a112640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a04071540 0x7f1a0406d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00bcc0 tx=0x7f19fc009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:25:01.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a09911640 1 --2- 172.21.15.114:0/2967290580 >> v2:172.21.15.149:6812/1185531135 conn(0x7f19f00377c0 0x7f19f0039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:01.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a09911640 10 monclient: get_auth_request con 0x7f19f00377c0 auth_method 0 2024-09-16T05:25:01.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 1 -- 172.21.15.114:0/2967290580 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19fc014630 con 0x7f1a04071540 2024-09-16T05:25:01.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:01.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:25:01.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 monclient: dump: 2024-09-16T05:25:01.034 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:01.035 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:01.035 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.035 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.035 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:01.035 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.035 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:01.035 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:01.035 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:01.035 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 1 -- 172.21.15.114:0/2967290580 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f19fc014630 con 0x7f1a04071540 2024-09-16T05:25:01.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:25:01.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:25:01.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 1 -- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a04071540 msgr2=0x7f1a0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:01.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a04071540 0x7f1a0406d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00bcc0 tx=0x7f19fc009f30 comp rx=0 tx=0).stop 2024-09-16T05:25:01.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 1 -- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a04071540 msgr2=0x7f1a0406d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:25:01.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a04071540 0x7f1a0406d990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:01.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a02ffd640 10 client.104330.objecter ms_handle_connect 0x7f19f00377c0 2024-09-16T05:25:01.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.018+0000 7f1a09911640 1 --2- 172.21.15.114:0/2967290580 >> v2:172.21.15.149:6812/1185531135 conn(0x7f19f00377c0 0x7f19f0039c80 secure :-1 s=READY pgs=3611 cs=0 l=1 rev1=1 crypto rx=0x7f19f80ad000 tx=0x7f19f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:25:01.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:25:01.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a0406ded0 0x7f1a0406e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:01.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 1 -- 172.21.15.114:0/2967290580 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1a0409e410 con 0x7f1a0406ded0 2024-09-16T05:25:01.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0a112640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a0406ded0 0x7f1a0406e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:01.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0a112640 10 monclient: get_auth_request con 0x7f1a0406ded0 auth_method 0 2024-09-16T05:25:01.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0a112640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:25:01.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0a112640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:25:01.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0a112640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:25:01.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0a112640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:25:01.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0a112640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:25:01.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0a112640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:25:01.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0a112640 10 monclient(hunting): handle_auth_done global_id 104336 payload 824 2024-09-16T05:25:01.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0a112640 1 -- 172.21.15.114:0/2967290580 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1a040a42c0 con 0x7f1a0406ded0 2024-09-16T05:25:01.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a02ffd640 10 client.104330.objecter ms_handle_connect 0x7f1a0406ded0 2024-09-16T05:25:01.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a02ffd640 10 client.104330.objecter resend_mon_ops 2024-09-16T05:25:01.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0a112640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a0406ded0 0x7f1a0406e2e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009fd0 tx=0x7f19fc005d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:25:01.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a02ffd640 1 -- 172.21.15.114:0/2967290580 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19fc019040 con 0x7f1a0406ded0 2024-09-16T05:25:01.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:01.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:25:01.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a02ffd640 10 monclient: dump: 2024-09-16T05:25:01.039 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:01.040 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:01.040 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.040 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.040 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:01.040 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.040 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:01.040 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:01.040 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:01.040 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a02ffd640 1 -- 172.21.15.114:0/2967290580 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f19fc0133d0 con 0x7f1a0406ded0 2024-09-16T05:25:01.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a02ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:25:01.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a02ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:25:01.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a02ffd640 1 -- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a0406ded0 msgr2=0x7f1a0406e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:01.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a02ffd640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a0406ded0 0x7f1a0406e2e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009fd0 tx=0x7f19fc005d60 comp rx=0 tx=0).stop 2024-09-16T05:25:01.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a02ffd640 1 -- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a0406ded0 msgr2=0x7f1a0406e2e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:25:01.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a02ffd640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a0406ded0 0x7f1a0406e2e0 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009fd0 tx=0x7f19fc005d60 comp rx=0 tx=0).stop 2024-09-16T05:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 72, 2024-09-16T05:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 277190, 2024-09-16T05:25:01.042 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:25:01.042 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:25:01.042 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:25:01.047 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:25:01.047 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:25:01.047 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:25:01.047 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:25:01.047 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:25:01.047 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:25:01.047 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:25:01.047 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:25:01.047 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:25:01.047 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:25:01.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.047 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:25:01.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.047 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:25:01.048 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:25:01.048 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:25:01.048 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:25:01.048 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.048 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:25:01.048 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:25:01.049 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:25:01.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.049 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.049 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:25:01.049 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:25:01.049 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:25:01.049 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:25:01.049 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:25:01.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.049 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:25:01.049 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:25:01.049 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:25:01.049 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:25:01.050 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:25:01.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.050 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.050 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:25:01.050 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:25:01.050 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:25:01.050 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:25:01.050 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.050 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:25:01.050 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:25:01.050 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:25:01.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.050 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.050 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:25:01.051 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:25:01.051 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:25:01.051 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:25:01.051 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:25:01.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.051 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:25:01.051 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:25:01.051 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:25:01.051 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:25:01.051 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:25:01.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.051 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:25:01.052 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:25:01.052 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:25:01.052 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:25:01.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.052 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:25:01.052 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:25:01.052 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:25:01.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.052 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:25:01.052 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:25:01.052 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:25:01.052 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:25:01.052 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:25:01.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.053 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:25:01.053 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:25:01.053 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:25:01.053 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:25:01.053 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:25:01.053 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:25:01.053 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:25:01.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.053 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:25:01.053 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:25:01.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.054 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:25:01.054 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:25:01.054 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:25:01.054 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:25:01.054 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:25:01.054 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:25:01.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.054 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:25:01.054 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:25:01.054 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 6 2024-09-16T05:25:01.054 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:25:01.054 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:25:01.054 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:25:01.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.054 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:25:01.055 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:25:01.055 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 7 2024-09-16T05:25:01.055 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:25:01.055 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:25:01.055 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:25:01.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.055 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:25:01.055 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:25:01.055 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:25:01.055 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:25:01.055 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:25:01.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.055 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:25:01.055 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:25:01.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 10 librados: watch_flush enter 2024-09-16T05:25:01.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 10 librados: watch_flush exit 2024-09-16T05:25:01.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 20 client.104330.objecter shutdown clearing up homeless session... 2024-09-16T05:25:01.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 10 client.104330.objecter successfully canceled tick 2024-09-16T05:25:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 1 -- 172.21.15.114:0/2967290580 >> v2:172.21.15.149:6812/1185531135 conn(0x7f19f00377c0 msgr2=0x7f19f0039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 1 --2- 172.21.15.114:0/2967290580 >> v2:172.21.15.149:6812/1185531135 conn(0x7f19f00377c0 0x7f19f0039c80 secure :-1 s=READY pgs=3611 cs=0 l=1 rev1=1 crypto rx=0x7f19f80ad000 tx=0x7f19f8009040 comp rx=0 tx=0).stop 2024-09-16T05:25:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 10 monclient: shutdown 2024-09-16T05:25:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:25:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 1 -- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a04138630 msgr2=0x7f1a04071000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a04138630 0x7f1a04071000 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f19f400c3f0 tx=0x7f19f400c8b0 comp rx=0 tx=0).stop 2024-09-16T05:25:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0a913640 1 -- 172.21.15.114:0/2967290580 reap_dead start 2024-09-16T05:25:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 1 -- 172.21.15.114:0/2967290580 shutdown_connections 2024-09-16T05:25:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 1 --2- 172.21.15.114:0/2967290580 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a04138630 0x7f1a04071000 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 1 -- 172.21.15.114:0/2967290580 >> 172.21.15.114:0/2967290580 conn(0x7f1a040a1fb0 msgr2=0x7f1a04135830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:25:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 1 -- 172.21.15.114:0/2967290580 shutdown_connections 2024-09-16T05:25:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 1 -- 172.21.15.114:0/2967290580 wait complete. 2024-09-16T05:25:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.026+0000 7f1a0c376640 1 librados: shutdown 2024-09-16T05:25:01.059 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:25:01.059 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T05:25:01.059 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:25:01.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 10 monclient: get_monmap_and_config 2024-09-16T05:25:01.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 10 monclient: build_initial_monmap 2024-09-16T05:25:01.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 10 monclient: monmap: 2024-09-16T05:25:01.171 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:25:01.171 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:01.172 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:25:01.177118+0000 2024-09-16T05:25:01.172 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:25:01.177118+0000 2024-09-16T05:25:01.172 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:25:01.172 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.172 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:25:01.172 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:25:01.172 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:25:01.172 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:25:01.172 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:25:01.172 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 1 Processor -- start 2024-09-16T05:25:01.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 1 -- start start 2024-09-16T05:25:01.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 10 monclient: init 2024-09-16T05:25:01.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 10 monclient: _reopen_session rank -1 2024-09-16T05:25:01.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-16T05:25:01.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4cfc09b6c0 0x7f4cfc0a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 10 monclient(hunting): picked mon.noname-c con 0x7f4cfc09b6c0 addr v2:172.21.15.149:3300/0 2024-09-16T05:25:01.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4cfc0a1400 addr v1:172.21.15.114:6789/0 2024-09-16T05:25:01.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc0a1810 0x7f4cfc13cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 10 monclient(hunting): picked mon.noname-b con 0x7f4cfc0a1810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:25:01.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cfc09d4e0 con 0x7f4cfc0a1400 2024-09-16T05:25:01.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:01.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f4cfc09e390 con 0x7f4cfc09b6c0 2024-09-16T05:25:01.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:01.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4cfc09ece0 con 0x7f4cfc0a1810 2024-09-16T05:25:01.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 10 monclient(hunting): _renew_subs 2024-09-16T05:25:01.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4d01c25640 10 monclient(hunting): authenticate will time out at 1985.806906s 2024-09-16T05:25:01.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cf3fff640 1 -- >> v1:172.21.15.114:6789/0 conn(0x7f4cfc0a1400 legacy=0x7f4cfc139640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.114:6789/0 2024-09-16T05:25:01.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc0a1810 0x7f4cfc13cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:01.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc0a1810 0x7f4cfc13cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51116/0 (socket says 172.21.15.114:51116) 2024-09-16T05:25:01.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 1 -- 172.21.15.114:0/3657541817 learned_addr learned my addr 172.21.15.114:0/3657541817 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:25:01.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 10 monclient(hunting): get_auth_request con 0x7f4cfc0a1810 auth_method 0 2024-09-16T05:25:01.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:25:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:25:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:25:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:25:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:25:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:25:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfb7fe640 1 --2- 172.21.15.114:0/3657541817 >> v2:172.21.15.149:3300/0 conn(0x7f4cfc09b6c0 0x7f4cfc0a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfb7fe640 10 monclient(hunting): get_auth_request con 0x7f4cfc09b6c0 auth_method 0 2024-09-16T05:25:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:25:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:25:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:25:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 10 monclient(hunting): handle_auth_done global_id 104339 payload 274 2024-09-16T05:25:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 1 -- 172.21.15.114:0/3657541817 >> v2:172.21.15.149:3300/0 conn(0x7f4cfc09b6c0 msgr2=0x7f4cfc0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:25:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 1 --2- 172.21.15.114:0/3657541817 >> v2:172.21.15.149:3300/0 conn(0x7f4cfc09b6c0 0x7f4cfc0a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 1 -- 172.21.15.114:0/3657541817 >> v1:172.21.15.114:6789/0 conn(0x7f4cfc0a1400 legacy=0x7f4cfc139640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:25:01.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 10 monclient: _finish_hunting 0 2024-09-16T05:25:01.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 1 monclient: found mon.noname-b 2024-09-16T05:25:01.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:25:01.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:25:01.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.174+0000 7f4cfbfff640 1 -- 172.21.15.114:0/3657541817 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cfc09f430 con 0x7f4cfc0a1810 2024-09-16T05:25:01.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfbfff640 1 --2- 172.21.15.114:0/3657541817 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc0a1810 0x7f4cfc13cd70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc0a2f70 tx=0x7f4cf401c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4db137c624f1e31 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:25:01.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfaffd640 1 -- 172.21.15.114:0/3657541817 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf40a5600 con 0x7f4cfc0a1810 2024-09-16T05:25:01.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:01.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:25:01.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfaffd640 10 monclient: dump: 2024-09-16T05:25:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:01.185 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfaffd640 10 monclient: _finish_auth 0 2024-09-16T05:25:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfaffd640 10 monclient: _check_auth_tickets 2024-09-16T05:25:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:25:01.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfaffd640 1 -- 172.21.15.114:0/3657541817 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cf40a57a0 con 0x7f4cfc0a1810 2024-09-16T05:25:01.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 5 monclient: authenticate success, global_id 104339 2024-09-16T05:25:01.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:25:01.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:25:01.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 10 monclient: get_monmap_and_config success 2024-09-16T05:25:01.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfaffd640 1 -- 172.21.15.114:0/3657541817 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf40a5b70 con 0x7f4cfc0a1810 2024-09-16T05:25:01.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:01.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 4 set_mon_vals no callback set 2024-09-16T05:25:01.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:25:01.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfaffd640 10 monclient: dump: 2024-09-16T05:25:01.186 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:01.186 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:01.186 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:01.187 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 10 monclient: shutdown 2024-09-16T05:25:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:25:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 -- 172.21.15.114:0/3657541817 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc0a1810 msgr2=0x7f4cfc13cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 --2- 172.21.15.114:0/3657541817 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc0a1810 0x7f4cfc13cd70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc0a2f70 tx=0x7f4cf401c890 comp rx=0 tx=0).stop 2024-09-16T05:25:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 -- 172.21.15.114:0/3657541817 shutdown_connections 2024-09-16T05:25:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 --2- 172.21.15.114:0/3657541817 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc0a1810 0x7f4cfc13cd70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 --2- 172.21.15.114:0/3657541817 >> v2:172.21.15.149:3300/0 conn(0x7f4cfc09b6c0 0x7f4cfc0a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 -- 172.21.15.114:0/3657541817 >> 172.21.15.114:0/3657541817 conn(0x7f4cfc0a1fb0 msgr2=0x7f4cfc0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:25:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 -- 172.21.15.114:0/3657541817 shutdown_connections 2024-09-16T05:25:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 -- 172.21.15.114:0/3657541817 wait complete. 2024-09-16T05:25:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 10 monclient: build_initial_monmap 2024-09-16T05:25:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 10 monclient: monmap: 2024-09-16T05:25:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:25:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:25:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:25:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:25:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:25:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:25:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:25:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:25:01.189 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 librados: starting msgr at 2024-09-16T05:25:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 librados: starting objecter 2024-09-16T05:25:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 Processor -- start 2024-09-16T05:25:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 -- start start 2024-09-16T05:25:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 librados: setting wanted keys 2024-09-16T05:25:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 librados: calling monclient init 2024-09-16T05:25:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 10 monclient: init 2024-09-16T05:25:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 10 monclient: _reopen_session rank -1 2024-09-16T05:25:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:25:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cfc09b6c0 0x7f4cfc13c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 10 monclient(hunting): picked mon.noname-a con 0x7f4cfc09b6c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:25:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc0a1400 0x7f4cfc091d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 10 monclient(hunting): picked mon.noname-c con 0x7f4cfc0a1400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:25:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc0a1810 0x7f4cfc092280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 10 monclient(hunting): picked mon.noname-b con 0x7f4cfc0a1810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:25:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc09cad0 con 0x7f4cfc09b6c0 2024-09-16T05:25:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc09dd40 con 0x7f4cfc0a1810 2024-09-16T05:25:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4cfc13fad0 con 0x7f4cfc0a1400 2024-09-16T05:25:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 10 monclient(hunting): _renew_subs 2024-09-16T05:25:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4d01c25640 10 monclient(hunting): authenticate will time out at 1985.811911s 2024-09-16T05:25:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfb7fe640 1 -- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cfc09b6c0 msgr2=0x7f4cfc13c9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:3300/0 2024-09-16T05:25:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.178+0000 7f4cfb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cfc09b6c0 0x7f4cfc13c9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:25:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc0a1400 0x7f4cfc091d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc0a1400 0x7f4cfc091d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51118/0 (socket says 172.21.15.114:51118) 2024-09-16T05:25:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 1 -- 172.21.15.114:0/3936778047 learned_addr learned my addr 172.21.15.114:0/3936778047 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:25:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 10 monclient(hunting): get_auth_request con 0x7f4cfc0a1400 auth_method 0 2024-09-16T05:25:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:25:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:25:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:25:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:25:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:25:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:25:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfbfff640 1 --2- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc0a1810 0x7f4cfc092280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfbfff640 10 monclient(hunting): get_auth_request con 0x7f4cfc0a1810 auth_method 0 2024-09-16T05:25:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:25:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:25:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:25:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 10 monclient(hunting): handle_auth_done global_id 104342 payload 824 2024-09-16T05:25:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 1 -- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc0a1810 msgr2=0x7f4cfc092280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 1 --2- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc0a1810 0x7f4cfc092280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 1 -- 172.21.15.114:0/3936778047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cfc09b6c0 msgr2=0x7f4cfc13c9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:25:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 1 --2- 172.21.15.114:0/3936778047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cfc09b6c0 0x7f4cfc13c9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:25:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 10 monclient: _finish_hunting 0 2024-09-16T05:25:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 1 monclient: found mon.noname-c 2024-09-16T05:25:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:25:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:25:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 1 -- 172.21.15.114:0/3936778047 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cfc093940 con 0x7f4cfc0a1400 2024-09-16T05:25:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf3fff640 1 --2- 172.21.15.114:0/3936778047 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc0a1400 0x7f4cfc091d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4ce400b550 tx=0x7f4ce400ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:25:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 client.?.objecter ms_handle_connect 0x7f4cfc0a1400 2024-09-16T05:25:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:25:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 1 -- 172.21.15.114:0/3936778047 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4003dd0 con 0x7f4cfc0a1400 2024-09-16T05:25:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:25:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 monclient: dump: 2024-09-16T05:25:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.194 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.194 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:01.194 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.194 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:01.194 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:01.194 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:01.194 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 monclient: _finish_auth 0 2024-09-16T05:25:01.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:25:01.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:25:01.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 1 -- 172.21.15.114:0/3936778047 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ce4015070 con 0x7f4cfc0a1400 2024-09-16T05:25:01.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4d01c25640 5 monclient: authenticate success, global_id 104342 2024-09-16T05:25:01.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4d01c25640 10 monclient: _renew_subs 2024-09-16T05:25:01.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4d01c25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:25:01.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4d01c25640 1 -- 172.21.15.114:0/3936778047 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cfc09c6a0 con 0x7f4cfc0a1400 2024-09-16T05:25:01.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:25:01.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 1 -- 172.21.15.114:0/3936778047 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce40052b0 con 0x7f4cfc0a1400 2024-09-16T05:25:01.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:01.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:25:01.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 monclient: dump: 2024-09-16T05:25:01.195 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:01.195 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:01.195 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.195 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.195 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:01.195 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.195 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:01.195 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:01.196 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:01.196 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfa7fc640 4 set_mon_vals no callback set 2024-09-16T05:25:01.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4d01c25640 10 client.104342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:25:01.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4d01c25640 10 monclient: _renew_subs 2024-09-16T05:25:01.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4d01c25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:25:01.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4d01c25640 1 -- 172.21.15.114:0/3936778047 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cfc095c30 con 0x7f4cfc0a1400 2024-09-16T05:25:01.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4d01c25640 1 librados: init done 2024-09-16T05:25:01.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 1 -- 172.21.15.114:0/3936778047 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4ce4005450 con 0x7f4cfc0a1400 2024-09-16T05:25:01.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4d01c25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:25:01.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4d01c25640 1 --2- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc0a0cd0 0x7f4cfc0927c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4d01c25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:01.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4d01c25640 1 -- 172.21.15.114:0/3936778047 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc069390 con 0x7f4cfc0a0cd0 2024-09-16T05:25:01.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 1 --2- 172.21.15.114:0/3936778047 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4cdc0377c0 0x7f4cdc039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 1 -- 172.21.15.114:0/3936778047 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(691..691 src has 1..691) ==== 5978+0+0 (secure 0 0 0) 0x7f4ce4048e70 con 0x7f4cfc0a1400 2024-09-16T05:25:01.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 client.104342.objecter ms_dispatch 0x7f4cfc000ba0 osd_map(691..691 src has 1..691) 2024-09-16T05:25:01.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 3 client.104342.objecter handle_osd_map got epochs [691,691] > 0 2024-09-16T05:25:01.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 3 client.104342.objecter handle_osd_map decoding full epoch 691 2024-09-16T05:25:01.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 20 client.104342.objecter dump_active .. 0 homeless 2024-09-16T05:25:01.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfb7fe640 1 --2- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc0a0cd0 0x7f4cfc0927c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:01.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfb7fe640 10 monclient: get_auth_request con 0x7f4cfc0a0cd0 auth_method 0 2024-09-16T05:25:01.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:25:01.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:25:01.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:25:01.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfbfff640 1 --2- 172.21.15.114:0/3936778047 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4cdc0377c0 0x7f4cdc039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:01.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfbfff640 10 monclient: get_auth_request con 0x7f4cdc0377c0 auth_method 0 2024-09-16T05:25:01.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:25:01.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:25:01.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:25:01.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfb7fe640 10 monclient(hunting): handle_auth_done global_id 124147 payload 824 2024-09-16T05:25:01.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfb7fe640 1 -- 172.21.15.114:0/3936778047 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4cfc13cf00 con 0x7f4cfc0a0cd0 2024-09-16T05:25:01.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 client.104342.objecter ms_handle_connect 0x7f4cdc0377c0 2024-09-16T05:25:01.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfbfff640 1 --2- 172.21.15.114:0/3936778047 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4cdc0377c0 0x7f4cdc039c80 secure :-1 s=READY pgs=3612 cs=0 l=1 rev1=1 crypto rx=0x7f4cf40ae000 tx=0x7f4cf40a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:25:01.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 client.104342.objecter ms_handle_connect 0x7f4cfc0a0cd0 2024-09-16T05:25:01.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 client.104342.objecter resend_mon_ops 2024-09-16T05:25:01.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cfb7fe640 1 --2- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc0a0cd0 0x7f4cfc0927c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00d4d0 tx=0x7f4cec00d990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:25:01.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 1 -- 172.21.15.114:0/3936778047 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cec004020 con 0x7f4cfc0a0cd0 2024-09-16T05:25:01.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:01.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:25:01.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.182+0000 7f4cf8ff9640 10 monclient: dump: 2024-09-16T05:25:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:01.199 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.186+0000 7f4cf8ff9640 1 -- 172.21.15.114:0/3936778047 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4cec004020 con 0x7f4cfc0a0cd0 2024-09-16T05:25:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.186+0000 7f4cf8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:25:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.186+0000 7f4cf8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:25:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.186+0000 7f4cf8ff9640 1 -- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc0a0cd0 msgr2=0x7f4cfc0927c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:01.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.186+0000 7f4cf8ff9640 1 --2- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc0a0cd0 0x7f4cfc0927c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00d4d0 tx=0x7f4cec00d990 comp rx=0 tx=0).stop 2024-09-16T05:25:01.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.186+0000 7f4cf8ff9640 1 -- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc0a0cd0 msgr2=0x7f4cfc0927c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:25:01.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.186+0000 7f4cf8ff9640 1 --2- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc0a0cd0 0x7f4cfc0927c0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00d4d0 tx=0x7f4cec00d990 comp rx=0 tx=0).stop 2024-09-16T05:25:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4d01c25640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:25:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4d01c25640 1 --2- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc08d000 0x7f4cfc08d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:25:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4d01c25640 10 monclient(hunting): start opening mon connection 2024-09-16T05:25:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4d01c25640 1 -- 172.21.15.114:0/3936778047 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc072d80 con 0x7f4cfc08d000 2024-09-16T05:25:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cfb7fe640 1 --2- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc08d000 0x7f4cfc08d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:25:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cfb7fe640 10 monclient: get_auth_request con 0x7f4cfc08d000 auth_method 0 2024-09-16T05:25:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:25:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:25:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:25:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:25:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:25:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:25:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cfb7fe640 10 monclient(hunting): handle_auth_done global_id 124150 payload 824 2024-09-16T05:25:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cfb7fe640 1 -- 172.21.15.114:0/3936778047 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4cfc0686f0 con 0x7f4cfc08d000 2024-09-16T05:25:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cfb7fe640 1 --2- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc08d000 0x7f4cfc08d410 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00d4d0 tx=0x7f4cec004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:25:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cf8ff9640 10 client.104342.objecter ms_handle_connect 0x7f4cfc08d000 2024-09-16T05:25:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cf8ff9640 10 client.104342.objecter resend_mon_ops 2024-09-16T05:25:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cf8ff9640 1 -- 172.21.15.114:0/3936778047 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cec0038b0 con 0x7f4cfc08d000 2024-09-16T05:25:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cf8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:25:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cf8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:25:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.190+0000 7f4cf8ff9640 10 monclient: dump: 2024-09-16T05:25:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:25:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:25:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:25:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:25:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:25:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:25:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:25:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:25:01.204 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:25:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4cf8ff9640 1 -- 172.21.15.114:0/3936778047 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f4cec0038b0 con 0x7f4cfc08d000 2024-09-16T05:25:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4cf8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:25:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4cf8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:25:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4cf8ff9640 1 -- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc08d000 msgr2=0x7f4cfc08d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4cf8ff9640 1 --2- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc08d000 0x7f4cfc08d410 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00d4d0 tx=0x7f4cec004740 comp rx=0 tx=0).stop 2024-09-16T05:25:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4cf8ff9640 1 -- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc08d000 msgr2=0x7f4cfc08d410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:25:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4cf8ff9640 1 --2- 172.21.15.114:0/3936778047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cfc08d000 0x7f4cfc08d410 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00d4d0 tx=0x7f4cec004740 comp rx=0 tx=0).stop 2024-09-16T05:25:01.205 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:25:01.205 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:25:01.205 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:25:01.205 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:25:01.205 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 72, 2024-09-16T05:25:01.205 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 23215, 2024-09-16T05:25:01.205 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:25:01.205 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:25:01.205 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:25:01.205 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:25:01.205 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 0, 2024-09-16T05:25:01.205 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:25:01.205 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:25:01.206 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:25:01.206 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:25:01.206 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:25:01.206 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:25:01.206 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:25:01.206 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:25:01.206 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:25:01.206 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:25:01.206 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:25:01.206 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:25:01.206 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:25:01.206 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:25:01.206 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:25:01.207 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:25:01.207 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:25:01.207 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:25:01.207 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:25:01.207 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:25:01.207 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:25:01.207 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:25:01.207 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:25:01.207 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:25:01.207 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:25:01.207 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:25:01.207 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:25:01.207 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:25:01.207 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:25:01.208 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.208 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:25:01.208 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:25:01.208 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:25:01.208 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:25:01.208 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:25:01.208 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:25:01.208 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:25:01.208 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:25:01.208 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:25:01.208 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:25:01.208 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:25:01.209 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:25:01.209 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:25:01.209 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:25:01.209 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:25:01.209 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:25:01.209 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:25:01.209 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:25:01.209 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:25:01.209 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:25:01.209 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:25:01.209 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:25:01.209 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:25:01.209 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:25:01.209 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:25:01.210 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:25:01.210 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:25:01.210 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:25:01.210 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:25:01.210 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:25:01.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.210 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:25:01.210 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.210 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:25:01.210 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:25:01.210 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:25:01.210 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:25:01.210 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.211 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:25:01.211 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:25:01.211 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:25:01.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.211 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.211 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:25:01.211 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:25:01.211 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:25:01.211 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:25:01.211 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:25:01.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.211 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:25:01.211 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:25:01.211 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:25:01.212 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:25:01.212 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:25:01.212 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.212 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.212 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:25:01.212 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:25:01.212 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:25:01.212 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:25:01.212 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.212 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:25:01.212 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:25:01.212 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:25:01.212 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.213 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.213 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:25:01.213 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:25:01.213 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:25:01.213 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:25:01.213 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:25:01.213 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.213 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:25:01.213 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:25:01.213 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:25:01.213 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:25:01.213 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:25:01.213 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.213 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.214 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:25:01.214 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:25:01.214 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:25:01.214 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:25:01.214 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.214 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:25:01.214 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:25:01.214 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:25:01.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.214 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.214 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:25:01.214 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:25:01.214 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:25:01.214 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:25:01.215 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:25:01.215 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.215 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:25:01.215 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:25:01.215 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:25:01.215 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:25:01.215 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:25:01.215 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:25:01.215 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:25:01.215 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.215 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:25:01.215 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:25:01.215 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.216 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:25:01.216 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:25:01.216 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:25:01.216 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:25:01.216 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:25:01.216 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:25:01.216 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:25:01.216 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:25:01.216 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:25:01.216 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:25:01.216 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:25:01.216 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:25:01.216 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:25:01.216 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:25:01.217 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:25:01.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4d01c25640 10 librados: watch_flush enter 2024-09-16T05:25:01.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4d01c25640 10 librados: watch_flush exit 2024-09-16T05:25:01.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4d01c25640 20 client.104342.objecter shutdown clearing up homeless session... 2024-09-16T05:25:01.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4d01c25640 10 client.104342.objecter successfully canceled tick 2024-09-16T05:25:01.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4d01c25640 1 -- 172.21.15.114:0/3936778047 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4cdc0377c0 msgr2=0x7f4cdc039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:01.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4d01c25640 1 --2- 172.21.15.114:0/3936778047 >> v2:172.21.15.149:6812/1185531135 conn(0x7f4cdc0377c0 0x7f4cdc039c80 secure :-1 s=READY pgs=3612 cs=0 l=1 rev1=1 crypto rx=0x7f4cf40ae000 tx=0x7f4cf40a7070 comp rx=0 tx=0).stop 2024-09-16T05:25:01.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4d01c25640 10 monclient: shutdown 2024-09-16T05:25:01.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4d01c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:25:01.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4d01c25640 1 -- 172.21.15.114:0/3936778047 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc0a1400 msgr2=0x7f4cfc091d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:25:01.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4d01c25640 1 --2- 172.21.15.114:0/3936778047 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cfc0a1400 0x7f4cfc091d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4ce400b550 tx=0x7f4ce400ba10 comp rx=0 tx=0).stop 2024-09-16T05:25:01.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4cfbfff640 1 -- 172.21.15.114:0/3936778047 reap_dead start 2024-09-16T05:25:01.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4cfbfff640 1 -- 172.21.15.114:0/3936778047 reap_dead start 2024-09-16T05:25:01.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4d01c25640 1 -- 172.21.15.114:0/3936778047 shutdown_connections 2024-09-16T05:25:01.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4d01c25640 1 -- 172.21.15.114:0/3936778047 >> 172.21.15.114:0/3936778047 conn(0x7f4cfc0a1fb0 msgr2=0x7f4cfc0a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:25:01.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4d01c25640 1 -- 172.21.15.114:0/3936778047 shutdown_connections 2024-09-16T05:25:01.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4d01c25640 1 -- 172.21.15.114:0/3936778047 wait complete. 2024-09-16T05:25:01.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:25:01.194+0000 7f4d01c25640 1 librados: shutdown 2024-09-16T05:25:01.220 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:25:01.220 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:25:04.473 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7934 ms) 2024-09-16T05:25:04.473 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-09-16T05:25:04.473 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13028 ms) 2024-09-16T05:25:04.473 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-09-16T05:25:04.473 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8076 ms) 2024-09-16T05:25:04.473 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-09-16T05:25:04.474 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (12957 ms) 2024-09-16T05:25:04.474 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-09-16T05:25:04.474 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (22570 ms) 2024-09-16T05:25:04.474 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-09-16T05:25:04.474 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (22180 ms) 2024-09-16T05:25:04.474 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-09-16T05:25:04.474 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-16T05:25:04.474 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: ok, hit_set contains 333:602f83fe:::foo:head 2024-09-16T05:25:04.474 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8284 ms) 2024-09-16T05:25:04.474 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-09-16T05:25:04.474 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726464206,0 2024-09-16T05:25:04.474 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: first is 1726464206 2024-09-16T05:25:04.474 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726464206,0 2024-09-16T05:25:04.475 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726464206,0 2024-09-16T05:25:04.475 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726464206,1726464208,1726464209,0 2024-09-16T05:25:04.475 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726464206,1726464208,1726464209,0 2024-09-16T05:25:04.475 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726464206,1726464208,1726464209,0 2024-09-16T05:25:04.475 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726464206,1726464208,1726464209,1726464211,1726464212,0 2024-09-16T05:25:04.475 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726464206,1726464208,1726464209,1726464211,1726464212,0 2024-09-16T05:25:04.475 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726464206,1726464208,1726464209,1726464211,1726464212,0 2024-09-16T05:25:04.475 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726464206,1726464208,1726464209,1726464211,1726464212,1726464214,1726464215,0 2024-09-16T05:25:04.475 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726464206,1726464208,1726464209,1726464211,1726464212,1726464214,1726464215,0 2024-09-16T05:25:04.475 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726464206,1726464208,1726464209,1726464211,1726464212,1726464214,1726464215,0 2024-09-16T05:25:04.475 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: got ls 1726464209,1726464211,1726464212,1726464214,1726464215,1726464217,1726464218,0 2024-09-16T05:25:04.475 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: first now 1726464209, trimmed 2024-09-16T05:25:04.475 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20971 ms) 2024-09-16T05:25:04.475 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-09-16T05:25:04.476 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: foo0 2024-09-16T05:25:04.476 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-16T05:25:04.476 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13991 ms) 2024-09-16T05:25:04.476 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-09-16T05:25:04.476 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18379 ms) 2024-09-16T05:25:04.476 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-09-16T05:25:04.476 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22003 ms) 2024-09-16T05:25:04.476 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-09-16T05:25:04.476 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (4637 ms) 2024-09-16T05:25:04.476 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-09-16T05:25:04.476 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3040 ms) 2024-09-16T05:25:04.476 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-09-16T05:25:04.476 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3005 ms) 2024-09-16T05:25:04.476 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-09-16T05:25:04.477 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (17199 ms) 2024-09-16T05:25:04.477 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (284349 ms total) 2024-09-16T05:25:04.477 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: 2024-09-16T05:25:04.477 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-09-16T05:25:04.477 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (975086 ms total) 2024-09-16T05:25:04.477 INFO:tasks.workunit.client.0.smithi149.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-09-16T05:25:04.480 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.480 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29472 2024-09-16T05:25:04.480 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29472 2024-09-16T05:25:04.480 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.480 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29667 2024-09-16T05:25:04.480 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29667 2024-09-16T05:25:04.480 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29981 2024-09-16T05:25:04.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29981 2024-09-16T05:25:04.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29785 2024-09-16T05:25:04.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29785 2024-09-16T05:25:04.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29728 2024-09-16T05:25:04.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29728 2024-09-16T05:25:04.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29531 2024-09-16T05:25:04.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29531 2024-09-16T05:25:04.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29766 2024-09-16T05:25:04.481 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29766 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30115 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 30115 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30186 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 30186 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29544 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29544 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30037 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 30037 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29939 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29939 2024-09-16T05:25:04.482 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.483 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29456 2024-09-16T05:25:04.483 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29456 2024-09-16T05:25:04.483 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.483 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29488 2024-09-16T05:25:04.483 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29488 2024-09-16T05:25:04.483 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.483 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29953 2024-09-16T05:25:04.483 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29953 2024-09-16T05:25:04.483 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.483 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29448 2024-09-16T05:25:04.483 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29448 2024-09-16T05:25:04.483 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.483 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29464 2024-09-16T05:25:04.483 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29464 2024-09-16T05:25:04.484 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.484 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30240 2024-09-16T05:25:04.484 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 30240 2024-09-16T05:25:04.484 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.484 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29683 2024-09-16T05:25:04.484 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29683 2024-09-16T05:25:04.484 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.484 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30256 2024-09-16T05:25:04.484 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 30256 2024-09-16T05:25:04.484 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.484 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29739 2024-09-16T05:25:04.484 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29739 2024-09-16T05:25:04.484 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.484 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30096 2024-09-16T05:25:04.485 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 30096 2024-09-16T05:25:04.485 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.485 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30359 2024-09-16T05:25:04.485 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 30359 2024-09-16T05:25:04.485 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.485 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29706 2024-09-16T05:25:04.485 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29706 2024-09-16T05:25:04.485 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.485 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29973 2024-09-16T05:25:04.485 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29973 2024-09-16T05:25:04.488 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30228 2024-09-16T05:25:04.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 30228 2024-09-16T05:25:04.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30044 2024-09-16T05:25:04.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 30044 2024-09-16T05:25:04.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30248 2024-09-16T05:25:04.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 30248 2024-09-16T05:25:04.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29932 2024-09-16T05:25:04.489 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29932 2024-09-16T05:25:04.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29628 2024-09-16T05:25:04.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29628 2024-09-16T05:25:04.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30265 2024-09-16T05:25:04.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 30265 2024-09-16T05:25:04.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29813 2024-09-16T05:25:04.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29813 2024-09-16T05:25:04.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29440 2024-09-16T05:25:04.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29440 2024-09-16T05:25:04.490 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.491 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29480 2024-09-16T05:25:04.491 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29480 2024-09-16T05:25:04.491 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.491 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29774 2024-09-16T05:25:04.491 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29774 2024-09-16T05:25:04.491 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.491 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29496 2024-09-16T05:25:04.491 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29496 2024-09-16T05:25:04.491 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.491 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29901 2024-09-16T05:25:04.491 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29901 2024-09-16T05:25:04.491 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.491 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=29916 2024-09-16T05:25:04.492 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 29916 2024-09-16T05:25:04.492 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.492 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30216 2024-09-16T05:25:04.492 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 30216 2024-09-16T05:25:04.492 INFO:tasks.workunit.client.0.smithi149.stderr:+ for t in "${!pids[@]}" 2024-09-16T05:25:04.492 INFO:tasks.workunit.client.0.smithi149.stderr:+ pid=30378 2024-09-16T05:25:04.492 INFO:tasks.workunit.client.0.smithi149.stderr:+ wait 30378 2024-09-16T05:25:04.492 INFO:tasks.workunit.client.0.smithi149.stderr:+ exit 0 2024-09-16T05:25:04.492 INFO:tasks.workunit.client.0.smithi149.stderr:+ cleanup 2024-09-16T05:25:04.492 INFO:tasks.workunit.client.0.smithi149.stderr:+ pkill -P 29434 2024-09-16T05:25:04.555 INFO:tasks.workunit.client.0.smithi149.stderr:+ true 2024-09-16T05:25:04.555 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T05:25:04.555 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T05:25:04.613 INFO:tasks.workunit:Stopping ['rados/test.sh unit_test_scan'] on client.0... 2024-09-16T05:25:04.614 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T05:25:04.971 DEBUG:teuthology.parallel:result is None 2024-09-16T05:25:04.972 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T05:25:04.984 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T05:25:04.984 DEBUG:teuthology.orchestra.run.smithi149:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T05:25:05.040 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T05:25:05.040 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-16T05:25:05.050 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-16T05:26:31.221 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T05:26:31.222 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T05:26:31.222 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T05:26:31.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T05:26:31.226 INFO:tasks.ceph.mon.a:Started 2024-09-16T05:26:31.226 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:26:31.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:26:31.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.355+0000 7fa36010e640 10 monclient: get_monmap_and_config 2024-09-16T05:26:31.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.355+0000 7fa36010e640 10 monclient: build_initial_monmap 2024-09-16T05:26:31.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.355+0000 7fa36010e640 10 monclient: monmap: 2024-09-16T05:26:31.354 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:31.354 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:31.354 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:26:31.359508+0000 2024-09-16T05:26:31.354 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:26:31.359508+0000 2024-09-16T05:26:31.354 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:31.354 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:31.355 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:26:31.355 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:26:31.355 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:26:31.355 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:26:31.355 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:26:31.355 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:31.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.355+0000 7fa36010e640 1 Processor -- start 2024-09-16T05:26:31.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.355+0000 7fa36010e640 1 -- start start 2024-09-16T05:26:31.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.355+0000 7fa36010e640 10 monclient: init 2024-09-16T05:26:31.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:31.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-16T05:26:31.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3581376e0 0x7fa358137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:31.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 10 monclient(hunting): picked mon.noname-b con 0x7fa3581376e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:31.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3581370a0 addr v1:172.21.15.114:6789/0 2024-09-16T05:26:31.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa3580a5d00 0x7fa3580a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:31.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 10 monclient(hunting): picked mon.noname-a con 0x7fa3580a5d00 addr v2:172.21.15.114:3300/0 2024-09-16T05:26:31.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa35809bd20 con 0x7fa3581370a0 2024-09-16T05:26:31.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:31.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa358138000 con 0x7fa3580a5d00 2024-09-16T05:26:31.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:31.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa35809c5c0 con 0x7fa3581376e0 2024-09-16T05:26:31.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:31.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 10 monclient(hunting): authenticate will time out at 2075.988514s 2024-09-16T05:26:31.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3581376e0 0x7fa358137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:31.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3581376e0 0x7fa358137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:54854/0 (socket says 172.21.15.114:54854) 2024-09-16T05:26:31.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35e6ab640 1 --2- 172.21.15.114:0/1233663561 >> v2:172.21.15.114:3300/0 conn(0x7fa3580a5d00 0x7fa3580a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:31.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 1 -- 172.21.15.114:0/1233663561 learned_addr learned my addr 172.21.15.114:0/1233663561 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:31.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 10 monclient(hunting): get_auth_request con 0x7fa3581376e0 auth_method 0 2024-09-16T05:26:31.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:31.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35e6ab640 10 monclient(hunting): get_auth_request con 0x7fa3580a5d00 auth_method 0 2024-09-16T05:26:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35e6ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35e6ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35e6ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35e6ab640 1 -- 172.21.15.114:0/1233663561 >> v2:172.21.15.114:3300/0 conn(0x7fa3580a5d00 msgr2=0x7fa3580a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:26:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35e6ab640 1 -- 172.21.15.114:0/1233663561 >> v2:172.21.15.114:3300/0 conn(0x7fa3580a5d00 msgr2=0x7fa3580a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35e6ab640 1 --2- 172.21.15.114:0/1233663561 >> v2:172.21.15.114:3300/0 conn(0x7fa3580a5d00 0x7fa3580a4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35e6ab640 1 --2- 172.21.15.114:0/1233663561 >> v2:172.21.15.114:3300/0 conn(0x7fa3580a5d00 0x7fa3580a4980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:26:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 10 monclient(hunting): handle_auth_done global_id 104345 payload 274 2024-09-16T05:26:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 1 -- 172.21.15.114:0/1233663561 >> v2:172.21.15.114:3300/0 conn(0x7fa3580a5d00 msgr2=0x7fa3580a4980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:26:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 1 --2- 172.21.15.114:0/1233663561 >> v2:172.21.15.114:3300/0 conn(0x7fa3580a5d00 0x7fa3580a4980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 1 -- 172.21.15.114:0/1233663561 >> v1:172.21.15.114:6789/0 conn(0x7fa3581370a0 legacy=0x7fa3580a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 10 monclient: _finish_hunting 0 2024-09-16T05:26:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 1 monclient: found mon.noname-b 2024-09-16T05:26:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:26:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 1 -- 172.21.15.114:0/1233663561 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa35809c000 con 0x7fa3581376e0 2024-09-16T05:26:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35deaa640 1 --2- 172.21.15.114:0/1233663561 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3581376e0 0x7fa358137ac0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa354009870 tx=0x7fa35401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e2b5966274914cb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35cea8640 1 -- 172.21.15.114:0/1233663561 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3540b0070 con 0x7fa3581376e0 2024-09-16T05:26:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35cea8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35cea8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:26:31.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35cea8640 10 monclient: dump: 2024-09-16T05:26:31.362 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:31.362 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:31.362 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:31.362 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:31.362 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:31.362 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:31.362 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:31.362 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:31.362 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:31.362 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:31.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35cea8640 10 monclient: _finish_auth 0 2024-09-16T05:26:31.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35cea8640 10 monclient: _check_auth_tickets 2024-09-16T05:26:31.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35cea8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35cea8640 1 -- 172.21.15.114:0/1233663561 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3540ab600 con 0x7fa3581376e0 2024-09-16T05:26:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 5 monclient: authenticate success, global_id 104345 2024-09-16T05:26:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:26:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35cea8640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 10 monclient: get_monmap_and_config success 2024-09-16T05:26:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 4 set_mon_vals no callback set 2024-09-16T05:26:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35cea8640 1 -- 172.21.15.114:0/1233663561 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3540ab9d0 con 0x7fa3581376e0 2024-09-16T05:26:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35cea8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35cea8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa35cea8640 10 monclient: dump: 2024-09-16T05:26:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:31.364 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:31.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 10 monclient: shutdown 2024-09-16T05:26:31.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:31.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 1 -- 172.21.15.114:0/1233663561 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3581376e0 msgr2=0x7fa358137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:31.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 1 --2- 172.21.15.114:0/1233663561 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3581376e0 0x7fa358137ac0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa354009870 tx=0x7fa35401c420 comp rx=0 tx=0).stop 2024-09-16T05:26:31.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 1 -- 172.21.15.114:0/1233663561 shutdown_connections 2024-09-16T05:26:31.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 1 --2- 172.21.15.114:0/1233663561 >> v2:172.21.15.114:3300/0 conn(0x7fa3580a5d00 0x7fa3580a4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:31.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 1 --2- 172.21.15.114:0/1233663561 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3581376e0 0x7fa358137ac0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.359+0000 7fa36010e640 1 -- 172.21.15.114:0/1233663561 >> 172.21.15.114:0/1233663561 conn(0x7fa3580a9a40 msgr2=0x7fa3580a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 -- 172.21.15.114:0/1233663561 shutdown_connections 2024-09-16T05:26:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 -- 172.21.15.114:0/1233663561 wait complete. 2024-09-16T05:26:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient: build_initial_monmap 2024-09-16T05:26:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient: monmap: 2024-09-16T05:26:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:26:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:26:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:26:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:26:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:26:31.371 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:26:31.371 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:31.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 librados: starting msgr at 2024-09-16T05:26:31.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 librados: starting objecter 2024-09-16T05:26:31.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 Processor -- start 2024-09-16T05:26:31.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 -- start start 2024-09-16T05:26:31.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 librados: setting wanted keys 2024-09-16T05:26:31.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 librados: calling monclient init 2024-09-16T05:26:31.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient: init 2024-09-16T05:26:31.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:31.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:26:31.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3580a5d00 0x7fa3581cf200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:31.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient(hunting): picked mon.noname-a con 0x7fa3580a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:26:31.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3581370a0 0x7fa35813e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:31.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient(hunting): picked mon.noname-c con 0x7fa3581370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:31.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa3581376e0 0x7fa35813eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:31.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient(hunting): picked mon.noname-b con 0x7fa3581376e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:26:31.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:31.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa358141d80 con 0x7fa3580a5d00 2024-09-16T05:26:31.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:31.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fa358141c00 con 0x7fa3581376e0 2024-09-16T05:26:31.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:31.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa358141f00 con 0x7fa3581370a0 2024-09-16T05:26:31.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:31.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient(hunting): authenticate will time out at 2075.993574s 2024-09-16T05:26:31.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35deaa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3580a5d00 0x7fa3581cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:31.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3581370a0 0x7fa35813e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:31.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35deaa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3580a5d00 0x7fa3581cf200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54020/0 (socket says 172.21.15.114:54020) 2024-09-16T05:26:31.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35deaa640 1 -- 172.21.15.114:0/2481626080 learned_addr learned my addr 172.21.15.114:0/2481626080 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:31.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35deaa640 10 monclient(hunting): get_auth_request con 0x7fa3580a5d00 auth_method 0 2024-09-16T05:26:31.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35deaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:31.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35deaa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:31.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35deaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:31.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 10 monclient(hunting): get_auth_request con 0x7fa3581370a0 auth_method 0 2024-09-16T05:26:31.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:31.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:31.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:31.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:31.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:31.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35deaa640 1 -- 172.21.15.114:0/2481626080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3580a5d00 msgr2=0x7fa3581cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:26:31.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35deaa640 1 -- 172.21.15.114:0/2481626080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3580a5d00 msgr2=0x7fa3581cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:26:31.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35deaa640 1 --2- 172.21.15.114:0/2481626080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3580a5d00 0x7fa3581cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:31.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:31.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35deaa640 1 --2- 172.21.15.114:0/2481626080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3580a5d00 0x7fa3581cf200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:26:31.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35e6ab640 1 --2- 172.21.15.114:0/2481626080 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa3581376e0 0x7fa35813eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:31.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 10 monclient(hunting): handle_auth_done global_id 104348 payload 824 2024-09-16T05:26:31.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 1 -- 172.21.15.114:0/2481626080 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa3581376e0 msgr2=0x7fa35813eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:31.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 1 --2- 172.21.15.114:0/2481626080 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa3581376e0 0x7fa35813eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:31.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 1 -- 172.21.15.114:0/2481626080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3580a5d00 msgr2=0x7fa3581cf200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:26:31.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 1 --2- 172.21.15.114:0/2481626080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3580a5d00 0x7fa3581cf200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:31.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 10 monclient: _finish_hunting 0 2024-09-16T05:26:31.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 1 monclient: found mon.noname-c 2024-09-16T05:26:31.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:31.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:26:31.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 1 -- 172.21.15.114:0/2481626080 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa35813cde0 con 0x7fa3581370a0 2024-09-16T05:26:31.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35d6a9640 1 --2- 172.21.15.114:0/2481626080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3581370a0 0x7fa35813e5f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa34800c3c0 tx=0x7fa34800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:31.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 10 client.?.objecter ms_handle_connect 0x7fa3581370a0 2024-09-16T05:26:31.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T05:26:31.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 1 -- 172.21.15.114:0/2481626080 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3480052d0 con 0x7fa3581370a0 2024-09-16T05:26:31.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:31.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:26:31.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 10 monclient: dump: 2024-09-16T05:26:31.376 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:31.376 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:31.376 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:31.376 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:31.376 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:31.376 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:31.376 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:31.376 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:31.376 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:31.377 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:31.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:26:31.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:26:31.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:31.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 5 monclient: authenticate success, global_id 104348 2024-09-16T05:26:31.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient: _renew_subs 2024-09-16T05:26:31.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 1 -- 172.21.15.114:0/2481626080 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa348005470 con 0x7fa3581370a0 2024-09-16T05:26:31.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:31.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 -- 172.21.15.114:0/2481626080 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3581cf740 con 0x7fa3581370a0 2024-09-16T05:26:31.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:31.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 1 -- 172.21.15.114:0/2481626080 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa348007260 con 0x7fa3581370a0 2024-09-16T05:26:31.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:31.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:31.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 10 monclient: dump: 2024-09-16T05:26:31.378 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:31.378 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:31.378 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:31.378 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:31.378 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:31.378 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:31.378 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:31.378 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:31.378 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:31.378 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:31.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34ffff640 4 set_mon_vals no callback set 2024-09-16T05:26:31.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 client.104348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:26:31.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient: _renew_subs 2024-09-16T05:26:31.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 -- 172.21.15.114:0/2481626080 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3580a9050 con 0x7fa3581370a0 2024-09-16T05:26:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 librados: init done 2024-09-16T05:26:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 1 -- 172.21.15.114:0/2481626080 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa348007400 con 0x7fa3581370a0 2024-09-16T05:26:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa36010e640 1 -- 172.21.15.114:0/2481626080 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa358073d20 con 0x7fa3581370a0 2024-09-16T05:26:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 1 --2- 172.21.15.114:0/2481626080 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa33c037810 0x7fa33c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 1 -- 172.21.15.114:0/2481626080 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7fa348047ed0 con 0x7fa3581370a0 2024-09-16T05:26:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 10 client.104348.objecter ms_dispatch 0x7fa358000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:26:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 3 client.104348.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:26:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 3 client.104348.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:26:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa34e7fc640 20 client.104348.objecter dump_active .. 0 homeless 2024-09-16T05:26:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35deaa640 1 --2- 172.21.15.114:0/2481626080 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa33c037810 0x7fa33c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:31.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.363+0000 7fa35deaa640 10 monclient: get_auth_request con 0x7fa33c037810 auth_method 0 2024-09-16T05:26:31.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.367+0000 7fa35deaa640 1 --2- 172.21.15.114:0/2481626080 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa33c037810 0x7fa33c039cd0 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 crypto rx=0x7fa35401a050 tx=0x7fa3540a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:31.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.367+0000 7fa34e7fc640 10 client.104348.objecter ms_handle_connect 0x7fa33c037810 2024-09-16T05:26:31.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.371+0000 7fa34e7fc640 1 -- 172.21.15.114:0/2481626080 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fa348012a50 con 0x7fa3581370a0 2024-09-16T05:26:31.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.371+0000 7fa34e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:31.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.371+0000 7fa34e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:26:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.607+0000 7fa36010e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:26:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.607+0000 7fa36010e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:26:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.607+0000 7fa36010e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.607+0000 7fa36010e640 1 -- 172.21.15.114:0/2481626080 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa358137ac0 con 0x7fa3581370a0 2024-09-16T05:26:31.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.607+0000 7fa34e7fc640 1 -- 172.21.15.114:0/2481626080 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fa348012d10 con 0x7fa3581370a0 2024-09-16T05:26:31.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa34e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:26:31.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa34e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:26:31.607 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":72,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":91,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:26:31.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 10 librados: watch_flush enter 2024-09-16T05:26:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 10 librados: watch_flush exit 2024-09-16T05:26:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 20 client.104348.objecter shutdown clearing up homeless session... 2024-09-16T05:26:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 10 client.104348.objecter successfully canceled tick 2024-09-16T05:26:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 1 -- 172.21.15.114:0/2481626080 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa33c037810 msgr2=0x7fa33c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 1 --2- 172.21.15.114:0/2481626080 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa33c037810 0x7fa33c039cd0 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 crypto rx=0x7fa35401a050 tx=0x7fa3540a6000 comp rx=0 tx=0).stop 2024-09-16T05:26:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 10 monclient: shutdown 2024-09-16T05:26:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 1 -- 172.21.15.114:0/2481626080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3581370a0 msgr2=0x7fa35813e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 1 --2- 172.21.15.114:0/2481626080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3581370a0 0x7fa35813e5f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa34800c3c0 tx=0x7fa34800c880 comp rx=0 tx=0).stop 2024-09-16T05:26:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 1 -- 172.21.15.114:0/2481626080 shutdown_connections 2024-09-16T05:26:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 1 --2- 172.21.15.114:0/2481626080 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa33c037810 0x7fa33c039cd0 unknown :-1 s=CLOSED pgs=3617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 1 --2- 172.21.15.114:0/2481626080 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa3581376e0 0x7fa35813eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 1 --2- 172.21.15.114:0/2481626080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3581370a0 0x7fa35813e5f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 1 --2- 172.21.15.114:0/2481626080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3580a5d00 0x7fa3581cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 1 -- 172.21.15.114:0/2481626080 >> 172.21.15.114:0/2481626080 conn(0x7fa3580a9a40 msgr2=0x7fa358136220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 1 -- 172.21.15.114:0/2481626080 shutdown_connections 2024-09-16T05:26:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 1 -- 172.21.15.114:0/2481626080 wait complete. 2024-09-16T05:26:31.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:31.611+0000 7fa36010e640 1 librados: shutdown 2024-09-16T05:26:34.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:26:34.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 10 monclient: get_monmap_and_config 2024-09-16T05:26:34.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 10 monclient: build_initial_monmap 2024-09-16T05:26:34.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 10 monclient: monmap: 2024-09-16T05:26:34.753 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:34.753 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:34.753 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:26:34.758324+0000 2024-09-16T05:26:34.753 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:26:34.758324+0000 2024-09-16T05:26:34.753 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:34.753 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:34.753 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:26:34.754 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:26:34.754 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:26:34.754 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:26:34.754 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:26:34.754 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:34.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 1 Processor -- start 2024-09-16T05:26:34.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 1 -- start start 2024-09-16T05:26:34.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 10 monclient: init 2024-09-16T05:26:34.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:34.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-16T05:26:34.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93b8137730 addr v1:172.21.15.114:6789/0 2024-09-16T05:26:34.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93b80a5d00 0x7f93b813c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:34.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 10 monclient(hunting): picked mon.noname-b con 0x7f93b80a5d00 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:34.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f93b813ca00 0x7f93b813edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:34.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 10 monclient(hunting): picked mon.noname-a con 0x7f93b813ca00 addr v2:172.21.15.114:3300/0 2024-09-16T05:26:34.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93b80a55e0 con 0x7f93b8137730 2024-09-16T05:26:34.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:34.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f93b809c450 con 0x7f93b813ca00 2024-09-16T05:26:34.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:34.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f93b80a3710 con 0x7f93b80a5d00 2024-09-16T05:26:34.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:34.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.755+0000 7f93bdede640 10 monclient(hunting): authenticate will time out at 2079.387908s 2024-09-16T05:26:34.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93b80a5d00 0x7f93b813c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:34.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b7fff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f93b813ca00 0x7f93b813edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:34.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b77fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f93b8137730 0x7f93b8137b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:47606/0 (socket says 172.21.15.114:47606) 2024-09-16T05:26:34.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b77fe640 1 -- 172.21.15.114:0/960987127 learned_addr learned my addr 172.21.15.114:0/960987127 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:26:34.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 10 monclient(hunting): get_auth_request con 0x7f93b80a5d00 auth_method 0 2024-09-16T05:26:34.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:34.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:34.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:34.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b7fff640 10 monclient(hunting): get_auth_request con 0x7f93b813ca00 auth_method 0 2024-09-16T05:26:34.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:34.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:34.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:34.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:34.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:34.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b7fff640 1 -- 172.21.15.114:0/960987127 >> v2:172.21.15.114:3300/0 conn(0x7f93b813ca00 msgr2=0x7f93b813edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:26:34.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:34.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b7fff640 1 -- 172.21.15.114:0/960987127 >> v2:172.21.15.114:3300/0 conn(0x7f93b813ca00 msgr2=0x7f93b813edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:34.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b7fff640 1 --2- 172.21.15.114:0/960987127 >> v2:172.21.15.114:3300/0 conn(0x7f93b813ca00 0x7f93b813edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:34.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b7fff640 1 --2- 172.21.15.114:0/960987127 >> v2:172.21.15.114:3300/0 conn(0x7f93b813ca00 0x7f93b813edf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:26:34.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 10 monclient(hunting): handle_auth_done global_id 104351 payload 274 2024-09-16T05:26:34.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 1 -- 172.21.15.114:0/960987127 >> v2:172.21.15.114:3300/0 conn(0x7f93b813ca00 msgr2=0x7f93b813edf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:26:34.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 1 --2- 172.21.15.114:0/960987127 >> v2:172.21.15.114:3300/0 conn(0x7f93b813ca00 0x7f93b813edf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:34.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 1 -- 172.21.15.114:0/960987127 >> v1:172.21.15.114:6789/0 conn(0x7f93b8137730 legacy=0x7f93b8137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:34.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:26:34.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 1 monclient: found mon.noname-b 2024-09-16T05:26:34.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:34.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:26:34.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 1 -- 172.21.15.114:0/960987127 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b809be00 con 0x7f93b80a5d00 2024-09-16T05:26:34.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b6ffd640 1 --2- 172.21.15.114:0/960987127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93b80a5d00 0x7f93b813c4c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009d90 tx=0x7f93b001c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee31bf624f548070 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:34.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b67fc640 1 -- 172.21.15.114:0/960987127 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b00b2070 con 0x7f93b80a5d00 2024-09-16T05:26:34.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:34.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:26:34.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b67fc640 10 monclient: dump: 2024-09-16T05:26:34.765 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:34.765 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:34.765 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:34.766 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:34.766 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:34.766 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:34.766 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:34.766 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:34.766 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:34.766 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:34.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b67fc640 10 monclient: _finish_auth 0 2024-09-16T05:26:34.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b67fc640 10 monclient: _check_auth_tickets 2024-09-16T05:26:34.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:34.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b67fc640 1 -- 172.21.15.114:0/960987127 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93b00a5800 con 0x7f93b80a5d00 2024-09-16T05:26:34.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93bdede640 5 monclient: authenticate success, global_id 104351 2024-09-16T05:26:34.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93bdede640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:26:34.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:34.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b67fc640 1 -- 172.21.15.114:0/960987127 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b00a5bd0 con 0x7f93b80a5d00 2024-09-16T05:26:34.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93bdede640 10 monclient: get_monmap_and_config success 2024-09-16T05:26:34.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93bdede640 4 set_mon_vals no callback set 2024-09-16T05:26:34.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:34.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:34.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93b67fc640 10 monclient: dump: 2024-09-16T05:26:34.767 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:34.767 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:34.767 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:34.767 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:34.767 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:34.767 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:34.767 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:34.767 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:34.768 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:34.768 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:34.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93bdede640 10 monclient: shutdown 2024-09-16T05:26:34.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93bdede640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:34.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93bdede640 1 -- 172.21.15.114:0/960987127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93b80a5d00 msgr2=0x7f93b813c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:34.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93bdede640 1 --2- 172.21.15.114:0/960987127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93b80a5d00 0x7f93b813c4c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009d90 tx=0x7f93b001c830 comp rx=0 tx=0).stop 2024-09-16T05:26:34.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93bdede640 1 -- 172.21.15.114:0/960987127 shutdown_connections 2024-09-16T05:26:34.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93bdede640 1 --2- 172.21.15.114:0/960987127 >> v2:172.21.15.114:3300/0 conn(0x7f93b813ca00 0x7f93b813edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:34.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93bdede640 1 --2- 172.21.15.114:0/960987127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93b80a5d00 0x7f93b813c4c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:34.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93bdede640 1 -- 172.21.15.114:0/960987127 >> 172.21.15.114:0/960987127 conn(0x7f93b80a9c70 msgr2=0x7f93b80aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:34.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93bdede640 1 -- 172.21.15.114:0/960987127 shutdown_connections 2024-09-16T05:26:34.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.759+0000 7f93bdede640 1 -- 172.21.15.114:0/960987127 wait complete. 2024-09-16T05:26:34.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient: build_initial_monmap 2024-09-16T05:26:34.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient: monmap: 2024-09-16T05:26:34.769 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:34.769 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:26:34.769 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:26:34.769 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:26:34.769 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:34.769 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:34.769 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:26:34.769 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:26:34.769 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:26:34.769 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:34.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 librados: starting msgr at 2024-09-16T05:26:34.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 librados: starting objecter 2024-09-16T05:26:34.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 Processor -- start 2024-09-16T05:26:34.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 -- start start 2024-09-16T05:26:34.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 librados: setting wanted keys 2024-09-16T05:26:34.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 librados: calling monclient init 2024-09-16T05:26:34.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient: init 2024-09-16T05:26:34.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:34.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:26:34.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f93b80a5d00 0x7f93b81cf310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:34.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient(hunting): picked mon.noname-b con 0x7f93b80a5d00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:26:34.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93b8137730 0x7f93b80a1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:34.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient(hunting): picked mon.noname-c con 0x7f93b8137730 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:34.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93b813ca00 0x7f93b80a24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:34.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient(hunting): picked mon.noname-a con 0x7f93b813ca00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:26:34.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:34.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f93b8141cd0 con 0x7f93b813ca00 2024-09-16T05:26:34.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:34.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f93b8141b50 con 0x7f93b80a5d00 2024-09-16T05:26:34.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:34.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f93b8141e50 con 0x7f93b8137730 2024-09-16T05:26:34.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:34.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient(hunting): authenticate will time out at 2079.392971s 2024-09-16T05:26:34.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93b8137730 0x7f93b80a1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:34.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93b813ca00 0x7f93b80a24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:34.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93b8137730 0x7f93b80a1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:54900/0 (socket says 172.21.15.114:54900) 2024-09-16T05:26:34.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 1 -- 172.21.15.114:0/2808221810 learned_addr learned my addr 172.21.15.114:0/2808221810 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:34.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 10 monclient(hunting): get_auth_request con 0x7f93b8137730 auth_method 0 2024-09-16T05:26:34.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:34.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:34.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:34.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b7fff640 10 monclient(hunting): get_auth_request con 0x7f93b813ca00 auth_method 0 2024-09-16T05:26:34.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:34.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:34.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:34.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:34.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:34.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b7fff640 1 -- 172.21.15.114:0/2808221810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93b813ca00 msgr2=0x7f93b80a24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T05:26:34.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b7fff640 1 -- 172.21.15.114:0/2808221810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93b813ca00 msgr2=0x7f93b80a24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:26:34.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b7fff640 1 --2- 172.21.15.114:0/2808221810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93b813ca00 0x7f93b80a24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:34.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:34.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b7fff640 1 --2- 172.21.15.114:0/2808221810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93b813ca00 0x7f93b80a24b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:26:34.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b77fe640 1 --2- 172.21.15.114:0/2808221810 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f93b80a5d00 0x7f93b81cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:34.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b77fe640 10 monclient(hunting): get_auth_request con 0x7f93b80a5d00 auth_method 0 2024-09-16T05:26:34.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:34.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:34.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:34.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 10 monclient(hunting): handle_auth_done global_id 104354 payload 824 2024-09-16T05:26:34.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 1 -- 172.21.15.114:0/2808221810 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f93b80a5d00 msgr2=0x7f93b81cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:34.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 1 --2- 172.21.15.114:0/2808221810 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f93b80a5d00 0x7f93b81cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:34.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 1 -- 172.21.15.114:0/2808221810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93b813ca00 msgr2=0x7f93b80a24b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:26:34.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 1 --2- 172.21.15.114:0/2808221810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93b813ca00 0x7f93b80a24b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:34.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:26:34.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 1 monclient: found mon.noname-c 2024-09-16T05:26:34.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:34.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:26:34.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 1 -- 172.21.15.114:0/2808221810 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b80a0780 con 0x7f93b8137730 2024-09-16T05:26:34.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b6ffd640 1 --2- 172.21.15.114:0/2808221810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93b8137730 0x7f93b80a1f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f93b00b0040 tx=0x7f93b00195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:34.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 10 client.?.objecter ms_handle_connect 0x7f93b8137730 2024-09-16T05:26:34.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:26:34.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 1 -- 172.21.15.114:0/2808221810 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b00b2050 con 0x7f93b8137730 2024-09-16T05:26:34.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:34.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:26:34.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 10 monclient: dump: 2024-09-16T05:26:34.774 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:34.774 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:34.774 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:34.774 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:34.774 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:34.774 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:34.774 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:34.774 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:34.774 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:34.775 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:34.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 10 monclient: _finish_auth 0 2024-09-16T05:26:34.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 10 monclient: _check_auth_tickets 2024-09-16T05:26:34.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:34.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 1 -- 172.21.15.114:0/2808221810 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93b00be070 con 0x7f93b8137730 2024-09-16T05:26:34.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 5 monclient: authenticate success, global_id 104354 2024-09-16T05:26:34.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient: _renew_subs 2024-09-16T05:26:34.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:34.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 -- 172.21.15.114:0/2808221810 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b80a04a0 con 0x7f93b8137730 2024-09-16T05:26:34.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:34.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 1 -- 172.21.15.114:0/2808221810 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b00ada50 con 0x7f93b8137730 2024-09-16T05:26:34.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:34.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:34.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 10 monclient: dump: 2024-09-16T05:26:34.775 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:34.776 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:34.776 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:34.776 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:34.776 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:34.776 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:34.776 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:34.776 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:34.776 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:34.776 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:34.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b5ffb640 4 set_mon_vals no callback set 2024-09-16T05:26:34.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 client.104354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:26:34.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient: _renew_subs 2024-09-16T05:26:34.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 -- 172.21.15.114:0/2808221810 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b809f050 con 0x7f93b8137730 2024-09-16T05:26:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 librados: init done 2024-09-16T05:26:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 1 -- 172.21.15.114:0/2808221810 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f93b00adbf0 con 0x7f93b8137730 2024-09-16T05:26:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93bdede640 1 -- 172.21.15.114:0/2808221810 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b8138c90 con 0x7f93b8137730 2024-09-16T05:26:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 1 --2- 172.21.15.114:0/2808221810 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9398037810 0x7f9398039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 1 -- 172.21.15.114:0/2808221810 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7f93b00e3dd0 con 0x7f93b8137730 2024-09-16T05:26:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 10 client.104354.objecter ms_dispatch 0x7f93b8000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:26:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 3 client.104354.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:26:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 3 client.104354.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:26:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93affff640 20 client.104354.objecter dump_active .. 0 homeless 2024-09-16T05:26:34.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b77fe640 1 --2- 172.21.15.114:0/2808221810 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9398037810 0x7f9398039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:34.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.763+0000 7f93b77fe640 10 monclient: get_auth_request con 0x7f9398037810 auth_method 0 2024-09-16T05:26:34.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.767+0000 7f93affff640 10 client.104354.objecter ms_handle_connect 0x7f9398037810 2024-09-16T05:26:34.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.767+0000 7f93b77fe640 1 --2- 172.21.15.114:0/2808221810 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9398037810 0x7f9398039cd0 secure :-1 s=READY pgs=3618 cs=0 l=1 rev1=1 crypto rx=0x7f93a8000f70 tx=0x7f93a8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:34.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.767+0000 7f93affff640 1 -- 172.21.15.114:0/2808221810 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f93b00e7030 con 0x7f93b8137730 2024-09-16T05:26:34.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.767+0000 7f93affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:34.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.767+0000 7f93affff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:26:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.991+0000 7f93bdede640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:26:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.991+0000 7f93bdede640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:26:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.991+0000 7f93bdede640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.991+0000 7f93bdede640 1 -- 172.21.15.114:0/2808221810 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f93b8068650 con 0x7f93b8137730 2024-09-16T05:26:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.991+0000 7f93affff640 1 -- 172.21.15.114:0/2808221810 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f93b0016540 con 0x7f93b8137730 2024-09-16T05:26:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.991+0000 7f93affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:26:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.991+0000 7f93affff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:26:34.989 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":72,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:26:34.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.991+0000 7f93bdede640 10 librados: watch_flush enter 2024-09-16T05:26:34.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.991+0000 7f93bdede640 10 librados: watch_flush exit 2024-09-16T05:26:34.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.991+0000 7f93bdede640 20 client.104354.objecter shutdown clearing up homeless session... 2024-09-16T05:26:34.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.991+0000 7f93bdede640 10 client.104354.objecter successfully canceled tick 2024-09-16T05:26:34.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.995+0000 7f93bdede640 1 -- 172.21.15.114:0/2808221810 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9398037810 msgr2=0x7f9398039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:34.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.995+0000 7f93bdede640 1 --2- 172.21.15.114:0/2808221810 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9398037810 0x7f9398039cd0 secure :-1 s=READY pgs=3618 cs=0 l=1 rev1=1 crypto rx=0x7f93a8000f70 tx=0x7f93a8009040 comp rx=0 tx=0).stop 2024-09-16T05:26:34.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.995+0000 7f93bdede640 10 monclient: shutdown 2024-09-16T05:26:34.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.995+0000 7f93bdede640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:34.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.995+0000 7f93bdede640 1 -- 172.21.15.114:0/2808221810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93b8137730 msgr2=0x7f93b80a1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:34.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.995+0000 7f93bdede640 1 --2- 172.21.15.114:0/2808221810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93b8137730 0x7f93b80a1f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f93b00b0040 tx=0x7f93b00195d0 comp rx=0 tx=0).stop 2024-09-16T05:26:34.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.995+0000 7f93bdede640 1 -- 172.21.15.114:0/2808221810 shutdown_connections 2024-09-16T05:26:34.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.995+0000 7f93bdede640 1 --2- 172.21.15.114:0/2808221810 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9398037810 0x7f9398039cd0 unknown :-1 s=CLOSED pgs=3618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:34.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.995+0000 7f93bdede640 1 --2- 172.21.15.114:0/2808221810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93b813ca00 0x7f93b80a24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:34.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.995+0000 7f93bdede640 1 --2- 172.21.15.114:0/2808221810 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93b8137730 0x7f93b80a1f70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:34.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.995+0000 7f93bdede640 1 --2- 172.21.15.114:0/2808221810 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f93b80a5d00 0x7f93b81cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:34.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.995+0000 7f93bdede640 1 -- 172.21.15.114:0/2808221810 >> 172.21.15.114:0/2808221810 conn(0x7f93b80a9c70 msgr2=0x7f93b8136220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:34.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.995+0000 7f93bdede640 1 -- 172.21.15.114:0/2808221810 shutdown_connections 2024-09-16T05:26:34.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.995+0000 7f93bdede640 1 -- 172.21.15.114:0/2808221810 wait complete. 2024-09-16T05:26:34.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:34.995+0000 7f93bdede640 1 librados: shutdown 2024-09-16T05:26:38.008 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:26:38.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 10 monclient: get_monmap_and_config 2024-09-16T05:26:38.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 10 monclient: build_initial_monmap 2024-09-16T05:26:38.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 10 monclient: monmap: 2024-09-16T05:26:38.136 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:38.136 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:38.136 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:26:38.141640+0000 2024-09-16T05:26:38.136 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:26:38.141640+0000 2024-09-16T05:26:38.137 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:38.137 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:38.137 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:26:38.137 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:26:38.137 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:26:38.137 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:26:38.137 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:26:38.137 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:38.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 1 Processor -- start 2024-09-16T05:26:38.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 1 -- start start 2024-09-16T05:26:38.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 10 monclient: init 2024-09-16T05:26:38.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:38.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-16T05:26:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7320098500 0x7f73200988e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 10 monclient(hunting): picked mon.noname-a con 0x7f7320098500 addr v2:172.21.15.114:3300/0 2024-09-16T05:26:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f732009c9b0 addr v1:172.21.15.114:6789/0 2024-09-16T05:26:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f732009d4d0 0x7f732013b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 10 monclient(hunting): picked mon.noname-c con 0x7f732009d4d0 addr v2:172.21.15.149:3300/0 2024-09-16T05:26:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f732009db80 con 0x7f732009c9b0 2024-09-16T05:26:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f732009e070 con 0x7f7320098500 2024-09-16T05:26:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f732009f550 con 0x7f732009d4d0 2024-09-16T05:26:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7326e5e640 10 monclient(hunting): authenticate will time out at 2082.770934s 2024-09-16T05:26:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7324bfa640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7320098500 0x7f73200988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7324bfa640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7320098500 0x7f73200988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37008/0 (socket says 172.21.15.114:37008) 2024-09-16T05:26:38.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7324bfa640 1 -- 172.21.15.114:0/1390918176 learned_addr learned my addr 172.21.15.114:0/1390918176 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:38.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7324bfa640 10 monclient(hunting): get_auth_request con 0x7f7320098500 auth_method 0 2024-09-16T05:26:38.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7324bfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:38.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7324bfa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:38.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7324bfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:38.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7324bfa640 1 -- 172.21.15.114:0/1390918176 >> v2:172.21.15.114:3300/0 conn(0x7f7320098500 msgr2=0x7f73200988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:26:38.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7324bfa640 1 -- 172.21.15.114:0/1390918176 >> v2:172.21.15.114:3300/0 conn(0x7f7320098500 msgr2=0x7f73200988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:38.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7324bfa640 1 --2- 172.21.15.114:0/1390918176 >> v2:172.21.15.114:3300/0 conn(0x7f7320098500 0x7f73200988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:38.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f7324bfa640 1 --2- 172.21.15.114:0/1390918176 >> v2:172.21.15.114:3300/0 conn(0x7f7320098500 0x7f73200988e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:26:38.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 1 --2- 172.21.15.114:0/1390918176 >> v2:172.21.15.149:3300/0 conn(0x7f732009d4d0 0x7f732013b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:38.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 10 monclient(hunting): get_auth_request con 0x7f732009d4d0 auth_method 0 2024-09-16T05:26:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 10 monclient(hunting): handle_auth_done global_id 124156 payload 274 2024-09-16T05:26:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 1 -- 172.21.15.114:0/1390918176 >> v2:172.21.15.114:3300/0 conn(0x7f7320098500 msgr2=0x7f73200988e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:26:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 1 --2- 172.21.15.114:0/1390918176 >> v2:172.21.15.114:3300/0 conn(0x7f7320098500 0x7f73200988e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 1 -- 172.21.15.114:0/1390918176 >> v1:172.21.15.114:6789/0 conn(0x7f732009c9b0 legacy=0x7f732009cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 10 monclient: _finish_hunting 0 2024-09-16T05:26:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 1 monclient: found mon.noname-c 2024-09-16T05:26:38.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.149:3300/0 2024-09-16T05:26:38.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.139+0000 7f73253fb640 1 -- 172.21.15.114:0/1390918176 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f732009e2d0 con 0x7f732009d4d0 2024-09-16T05:26:38.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f73253fb640 1 --2- 172.21.15.114:0/1390918176 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f732009d4d0 0x7f732013b8d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f732009ef70 tx=0x7f731801c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=803e2a4ee1a0b547 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:38.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f731f7fe640 1 -- 172.21.15.114:0/1390918176 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73180b2070 con 0x7f732009d4d0 2024-09-16T05:26:38.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f731f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:38.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f731f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:26:38.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f731f7fe640 10 monclient: dump: 2024-09-16T05:26:38.149 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:38.149 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:38.149 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:38.149 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:38.149 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:38.149 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:38.150 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:38.150 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:38.150 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:38.150 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:38.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f731f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:26:38.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f731f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:26:38.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f731f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:38.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f731f7fe640 1 -- 172.21.15.114:0/1390918176 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73180a5820 con 0x7f732009d4d0 2024-09-16T05:26:38.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f731f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:38.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f731f7fe640 1 -- 172.21.15.114:0/1390918176 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73180a5bf0 con 0x7f732009d4d0 2024-09-16T05:26:38.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f7326e5e640 5 monclient: authenticate success, global_id 124156 2024-09-16T05:26:38.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f7326e5e640 10 monclient: get_monmap_and_config success 2024-09-16T05:26:38.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f7326e5e640 4 set_mon_vals no callback set 2024-09-16T05:26:38.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f731f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:38.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f731f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:26:38.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f731f7fe640 10 monclient: dump: 2024-09-16T05:26:38.151 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:38.151 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:38.151 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:38.151 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:38.151 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:38.151 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:38.151 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:38.151 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:38.151 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:38.152 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f7326e5e640 10 monclient: shutdown 2024-09-16T05:26:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f7326e5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f7326e5e640 1 -- 172.21.15.114:0/1390918176 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f732009d4d0 msgr2=0x7f732013b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f7326e5e640 1 --2- 172.21.15.114:0/1390918176 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f732009d4d0 0x7f732013b8d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f732009ef70 tx=0x7f731801c7c0 comp rx=0 tx=0).stop 2024-09-16T05:26:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f7326e5e640 1 -- 172.21.15.114:0/1390918176 shutdown_connections 2024-09-16T05:26:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f7326e5e640 1 --2- 172.21.15.114:0/1390918176 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f732009d4d0 0x7f732013b8d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f7326e5e640 1 --2- 172.21.15.114:0/1390918176 >> v2:172.21.15.114:3300/0 conn(0x7f7320098500 0x7f73200988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:38.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f7326e5e640 1 -- 172.21.15.114:0/1390918176 >> 172.21.15.114:0/1390918176 conn(0x7f73200a1fb0 msgr2=0x7f73200a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f7326e5e640 1 -- 172.21.15.114:0/1390918176 shutdown_connections 2024-09-16T05:26:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.143+0000 7f7326e5e640 1 -- 172.21.15.114:0/1390918176 wait complete. 2024-09-16T05:26:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient: build_initial_monmap 2024-09-16T05:26:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient: monmap: 2024-09-16T05:26:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:26:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:26:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:26:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:26:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:26:38.153 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:26:38.153 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:38.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 librados: starting msgr at 2024-09-16T05:26:38.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 librados: starting objecter 2024-09-16T05:26:38.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 Processor -- start 2024-09-16T05:26:38.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 -- start start 2024-09-16T05:26:38.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 librados: setting wanted keys 2024-09-16T05:26:38.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 librados: calling monclient init 2024-09-16T05:26:38.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient: init 2024-09-16T05:26:38.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:38.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:26:38.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7320098500 0x7f73201cf070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient(hunting): picked mon.noname-a con 0x7f7320098500 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:26:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f732009c9b0 0x7f73201cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient(hunting): picked mon.noname-b con 0x7f732009c9b0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:26:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f732009d4d0 0x7f73201cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient(hunting): picked mon.noname-c con 0x7f732009d4d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7320098fa0 con 0x7f7320098500 2024-09-16T05:26:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f7320098e20 con 0x7f732009c9b0 2024-09-16T05:26:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7320099e00 con 0x7f732009d4d0 2024-09-16T05:26:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient(hunting): authenticate will time out at 2082.777152s 2024-09-16T05:26:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f732009d4d0 0x7f73201cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7324bfa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7320098500 0x7f73201cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f732009d4d0 0x7f73201cfbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:55996/0 (socket says 172.21.15.114:55996) 2024-09-16T05:26:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7324bfa640 10 monclient(hunting): get_auth_request con 0x7f7320098500 auth_method 0 2024-09-16T05:26:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 1 -- 172.21.15.114:0/4273011435 learned_addr learned my addr 172.21.15.114:0/4273011435 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7324bfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7324bfa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7324bfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 10 monclient(hunting): get_auth_request con 0x7f732009d4d0 auth_method 0 2024-09-16T05:26:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:38.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7324bfa640 1 -- 172.21.15.114:0/4273011435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7320098500 msgr2=0x7f73201cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T05:26:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7324bfa640 1 -- 172.21.15.114:0/4273011435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7320098500 msgr2=0x7f73201cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:26:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7324bfa640 1 --2- 172.21.15.114:0/4273011435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7320098500 0x7f73201cf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7324bfa640 1 --2- 172.21.15.114:0/4273011435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7320098500 0x7f73201cf070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:26:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731ffff640 1 --2- 172.21.15.114:0/4273011435 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f732009c9b0 0x7f73201cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731ffff640 10 monclient(hunting): get_auth_request con 0x7f732009c9b0 auth_method 0 2024-09-16T05:26:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:38.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 10 monclient(hunting): handle_auth_done global_id 104357 payload 824 2024-09-16T05:26:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 1 -- 172.21.15.114:0/4273011435 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f732009c9b0 msgr2=0x7f73201cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 1 --2- 172.21.15.114:0/4273011435 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f732009c9b0 0x7f73201cf6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 1 -- 172.21.15.114:0/4273011435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7320098500 msgr2=0x7f73201cf070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:26:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 1 --2- 172.21.15.114:0/4273011435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7320098500 0x7f73201cf070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 10 monclient: _finish_hunting 0 2024-09-16T05:26:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 1 monclient: found mon.noname-c 2024-09-16T05:26:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:26:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 1 -- 172.21.15.114:0/4273011435 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7320072b90 con 0x7f732009d4d0 2024-09-16T05:26:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731ffff640 1 --2- 172.21.15.114:0/4273011435 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f732009c9b0 0x7f73201cf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:26:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f73253fb640 1 --2- 172.21.15.114:0/4273011435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f732009d4d0 0x7f73201cfbe0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f73180193d0 tx=0x7f7318019560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 10 client.?.objecter ms_handle_connect 0x7f732009d4d0 2024-09-16T05:26:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:26:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 1 -- 172.21.15.114:0/4273011435 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73180b2050 con 0x7f732009d4d0 2024-09-16T05:26:38.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:26:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 10 monclient: dump: 2024-09-16T05:26:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:38.159 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 10 monclient: _finish_auth 0 2024-09-16T05:26:38.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:26:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 1 -- 172.21.15.114:0/4273011435 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73180be070 con 0x7f732009d4d0 2024-09-16T05:26:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 5 monclient: authenticate success, global_id 104357 2024-09-16T05:26:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient: _renew_subs 2024-09-16T05:26:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 -- 172.21.15.114:0/4273011435 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f732009c6a0 con 0x7f732009d4d0 2024-09-16T05:26:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 1 -- 172.21.15.114:0/4273011435 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73180ada90 con 0x7f732009d4d0 2024-09-16T05:26:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 10 monclient: dump: 2024-09-16T05:26:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:38.161 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731effd640 4 set_mon_vals no callback set 2024-09-16T05:26:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 client.104357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:26:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient: _renew_subs 2024-09-16T05:26:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 -- 172.21.15.114:0/4273011435 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f73200965e0 con 0x7f732009d4d0 2024-09-16T05:26:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 librados: init done 2024-09-16T05:26:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 1 -- 172.21.15.114:0/4273011435 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f73180adc30 con 0x7f732009d4d0 2024-09-16T05:26:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 1 --2- 172.21.15.114:0/4273011435 >> v2:172.21.15.149:6812/1185531135 conn(0x7f730c037810 0x7f730c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7326e5e640 1 -- 172.21.15.114:0/4273011435 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73200988e0 con 0x7f732009d4d0 2024-09-16T05:26:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 1 -- 172.21.15.114:0/4273011435 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7f73180e3db0 con 0x7f732009d4d0 2024-09-16T05:26:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 10 client.104357.objecter ms_dispatch 0x7f7320000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:26:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 3 client.104357.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:26:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 3 client.104357.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:26:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f731d7fa640 20 client.104357.objecter dump_active .. 0 homeless 2024-09-16T05:26:38.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7324bfa640 1 --2- 172.21.15.114:0/4273011435 >> v2:172.21.15.149:6812/1185531135 conn(0x7f730c037810 0x7f730c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.147+0000 7f7324bfa640 10 monclient: get_auth_request con 0x7f730c037810 auth_method 0 2024-09-16T05:26:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.151+0000 7f731d7fa640 10 client.104357.objecter ms_handle_connect 0x7f730c037810 2024-09-16T05:26:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.151+0000 7f7324bfa640 1 --2- 172.21.15.114:0/4273011435 >> v2:172.21.15.149:6812/1185531135 conn(0x7f730c037810 0x7f730c039cd0 secure :-1 s=READY pgs=3619 cs=0 l=1 rev1=1 crypto rx=0x7f7314009100 tx=0x7f7314008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.151+0000 7f731d7fa640 1 -- 172.21.15.114:0/4273011435 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7318016500 con 0x7f732009d4d0 2024-09-16T05:26:38.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.155+0000 7f731d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:38.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.155+0000 7f731d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:26:38.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:26:38.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:26:38.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:38.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 1 -- 172.21.15.114:0/4273011435 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f73200686f0 con 0x7f732009d4d0 2024-09-16T05:26:38.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f731d7fa640 1 -- 172.21.15.114:0/4273011435 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f73180ba030 con 0x7f732009d4d0 2024-09-16T05:26:38.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f731d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:26:38.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f731d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:26:38.373 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":72,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:26:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 10 librados: watch_flush enter 2024-09-16T05:26:38.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 10 librados: watch_flush exit 2024-09-16T05:26:38.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 20 client.104357.objecter shutdown clearing up homeless session... 2024-09-16T05:26:38.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 10 client.104357.objecter successfully canceled tick 2024-09-16T05:26:38.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 1 -- 172.21.15.114:0/4273011435 >> v2:172.21.15.149:6812/1185531135 conn(0x7f730c037810 msgr2=0x7f730c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:38.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 1 --2- 172.21.15.114:0/4273011435 >> v2:172.21.15.149:6812/1185531135 conn(0x7f730c037810 0x7f730c039cd0 secure :-1 s=READY pgs=3619 cs=0 l=1 rev1=1 crypto rx=0x7f7314009100 tx=0x7f7314008040 comp rx=0 tx=0).stop 2024-09-16T05:26:38.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 10 monclient: shutdown 2024-09-16T05:26:38.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:38.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 1 -- 172.21.15.114:0/4273011435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f732009d4d0 msgr2=0x7f73201cfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:38.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 1 --2- 172.21.15.114:0/4273011435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f732009d4d0 0x7f73201cfbe0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f73180193d0 tx=0x7f7318019560 comp rx=0 tx=0).stop 2024-09-16T05:26:38.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 1 -- 172.21.15.114:0/4273011435 shutdown_connections 2024-09-16T05:26:38.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 1 --2- 172.21.15.114:0/4273011435 >> v2:172.21.15.149:6812/1185531135 conn(0x7f730c037810 0x7f730c039cd0 unknown :-1 s=CLOSED pgs=3619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:38.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 1 --2- 172.21.15.114:0/4273011435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f732009d4d0 0x7f73201cfbe0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:38.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 1 --2- 172.21.15.114:0/4273011435 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f732009c9b0 0x7f73201cf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:38.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 1 --2- 172.21.15.114:0/4273011435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7320098500 0x7f73201cf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:38.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 1 -- 172.21.15.114:0/4273011435 >> 172.21.15.114:0/4273011435 conn(0x7f73200a1fb0 msgr2=0x7f7320072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:38.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 1 -- 172.21.15.114:0/4273011435 shutdown_connections 2024-09-16T05:26:38.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.375+0000 7f7326e5e640 1 -- 172.21.15.114:0/4273011435 wait complete. 2024-09-16T05:26:38.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:38.379+0000 7f7326e5e640 1 librados: shutdown 2024-09-16T05:26:41.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:26:41.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 10 monclient: get_monmap_and_config 2024-09-16T05:26:41.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 10 monclient: build_initial_monmap 2024-09-16T05:26:41.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 10 monclient: monmap: 2024-09-16T05:26:41.520 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:41.520 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:41.520 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:26:41.525786+0000 2024-09-16T05:26:41.521 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:26:41.525786+0000 2024-09-16T05:26:41.521 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:41.521 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:41.521 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:26:41.521 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:26:41.521 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:26:41.521 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:26:41.521 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:26:41.521 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:41.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 1 Processor -- start 2024-09-16T05:26:41.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 1 -- start start 2024-09-16T05:26:41.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 10 monclient: init 2024-09-16T05:26:41.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:41.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-16T05:26:41.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb4d0098500 0x7fb4d00988e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:41.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 10 monclient(hunting): picked mon.noname-a con 0x7fb4d0098500 addr v2:172.21.15.114:3300/0 2024-09-16T05:26:41.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb4d009c9b0 addr v1:172.21.15.149:6789/0 2024-09-16T05:26:41.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4d009d4d0 addr v1:172.21.15.114:6789/0 2024-09-16T05:26:41.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4d009db80 con 0x7fb4d009d4d0 2024-09-16T05:26:41.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4d009e2d0 con 0x7fb4d009c9b0 2024-09-16T05:26:41.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:41.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb4d009e070 con 0x7fb4d0098500 2024-09-16T05:26:41.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:41.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d7185640 10 monclient(hunting): authenticate will time out at 2086.154926s 2024-09-16T05:26:41.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d4f21640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb4d0098500 0x7fb4d00988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:41.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d5722640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb4d009d4d0 0x7fb4d013b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34194/0 (socket says 172.21.15.114:34194) 2024-09-16T05:26:41.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d5722640 1 -- 172.21.15.114:0/674801911 learned_addr learned my addr 172.21.15.114:0/674801911 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:26:41.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d4f21640 10 monclient(hunting): get_auth_request con 0x7fb4d0098500 auth_method 0 2024-09-16T05:26:41.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d4f21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:41.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d4f21640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:41.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d4f21640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:41.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d4f21640 1 -- 172.21.15.114:0/674801911 >> v2:172.21.15.114:3300/0 conn(0x7fb4d0098500 msgr2=0x7fb4d00988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:26:41.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d4f21640 1 -- 172.21.15.114:0/674801911 >> v2:172.21.15.114:3300/0 conn(0x7fb4d0098500 msgr2=0x7fb4d00988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:41.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d4f21640 1 --2- 172.21.15.114:0/674801911 >> v2:172.21.15.114:3300/0 conn(0x7fb4d0098500 0x7fb4d00988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:41.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4d4f21640 1 --2- 172.21.15.114:0/674801911 >> v2:172.21.15.114:3300/0 conn(0x7fb4d0098500 0x7fb4d00988e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:26:41.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4cf7fe640 1 -- 172.21.15.114:0/674801911 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2118077340 0 0) 0x7fb4d009e2d0 con 0x7fb4d009c9b0 2024-09-16T05:26:41.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4cf7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:26:41.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:41.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:41.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4cf7fe640 10 monclient(hunting): my global_id is 124162 2024-09-16T05:26:41.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.523+0000 7fb4cf7fe640 1 -- 172.21.15.114:0/674801911 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4b8004640 con 0x7fb4d009c9b0 2024-09-16T05:26:41.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 -- 172.21.15.114:0/674801911 <== mon.1 v1:172.21.15.149:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2730976321 0 0) 0x7fb4b8004640 con 0x7fb4d009c9b0 2024-09-16T05:26:41.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 -- 172.21.15.114:0/674801911 >> v2:172.21.15.114:3300/0 conn(0x7fb4d0098500 msgr2=0x7fb4d00988e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:26:41.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 --2- 172.21.15.114:0/674801911 >> v2:172.21.15.114:3300/0 conn(0x7fb4d0098500 0x7fb4d00988e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:41.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 -- 172.21.15.114:0/674801911 >> v1:172.21.15.114:6789/0 conn(0x7fb4d009d4d0 legacy=0x7fb4d013b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:41.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:26:41.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 monclient: found mon.noname-c-legacy 2024-09-16T05:26:41.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:41.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.149:6789/0 2024-09-16T05:26:41.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 -- 172.21.15.114:0/674801911 --> v1:172.21.15.149:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d009f270 con 0x7fb4d009c9b0 2024-09-16T05:26:41.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: _finish_auth 0 2024-09-16T05:26:41.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:26:41.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:41.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 -- 172.21.15.114:0/674801911 <== mon.1 v1:172.21.15.149:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7fb4c40016d0 con 0x7fb4d009c9b0 2024-09-16T05:26:41.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:41.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T05:26:41.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: dump: 2024-09-16T05:26:41.528 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:41.528 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:41.528 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:41.528 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:41.528 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:41.529 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:41.529 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:41.529 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:41.529 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:41.529 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:41.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] but i'm connected to v1:172.21.15.149:6789/0, reconnecting 2024-09-16T05:26:41.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:41.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 -- 172.21.15.114:0/674801911 >> v1:172.21.15.149:6789/0 conn(0x7fb4d009c9b0 legacy=0x7fb4d009cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:41.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:26:41.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 --2- 172.21.15.114:0/674801911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b8004ad0 0x7fb4b8004ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:41.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient(hunting): picked mon.a con 0x7fb4b8004ad0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:26:41.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 --2- 172.21.15.114:0/674801911 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb4b8005920 0x7fb4b8005da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:41.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient(hunting): picked mon.b con 0x7fb4b8005920 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:26:41.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 --2- 172.21.15.114:0/674801911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4b80062e0 0x7fb4b8003530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:41.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient(hunting): picked mon.c con 0x7fb4b80062e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:41.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:41.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 -- 172.21.15.114:0/674801911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b8003c50 con 0x7fb4b8004ad0 2024-09-16T05:26:41.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:41.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 -- 172.21.15.114:0/674801911 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b8003e00 con 0x7fb4b8005920 2024-09-16T05:26:41.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:41.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 -- 172.21.15.114:0/674801911 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb4b8004000 con 0x7fb4b80062e0 2024-09-16T05:26:41.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:41.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:26:41.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d4f21640 1 --2- 172.21.15.114:0/674801911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b8004ad0 0x7fb4b8004ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:41.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d4f21640 10 monclient(hunting): get_auth_request con 0x7fb4b8004ad0 auth_method 0 2024-09-16T05:26:41.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 1 --2- 172.21.15.114:0/674801911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4b80062e0 0x7fb4b8003530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:41.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d4f21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:41.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d4f21640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:41.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d4f21640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:41.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 10 monclient(hunting): get_auth_request con 0x7fb4b80062e0 auth_method 0 2024-09-16T05:26:41.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:41.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:41.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:41.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:41.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:41.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cffff640 1 --2- 172.21.15.114:0/674801911 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb4b8005920 0x7fb4b8005da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:41.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:26:41.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cffff640 10 monclient(hunting): get_auth_request con 0x7fb4b8005920 auth_method 0 2024-09-16T05:26:41.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:41.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:41.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:41.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d4f21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:41.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d4f21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:41.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d4f21640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:26:41.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:41.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:41.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:26:41.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 10 monclient(hunting): handle_auth_done global_id 124162 payload 293 2024-09-16T05:26:41.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 1 -- 172.21.15.114:0/674801911 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb4b8005920 msgr2=0x7fb4b8005da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:26:41.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 1 --2- 172.21.15.114:0/674801911 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb4b8005920 0x7fb4b8005da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:41.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 1 -- 172.21.15.114:0/674801911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b8004ad0 msgr2=0x7fb4b8004ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:26:41.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 1 --2- 172.21.15.114:0/674801911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b8004ad0 0x7fb4b8004ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:41.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 10 monclient: _finish_hunting 0 2024-09-16T05:26:41.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 1 monclient: found mon.c 2024-09-16T05:26:41.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:41.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:41.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 1 -- 172.21.15.114:0/674801911 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b8004640 con 0x7fb4b80062e0 2024-09-16T05:26:41.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 10 monclient: _finish_auth 0 2024-09-16T05:26:41.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 10 monclient: _check_auth_tickets 2024-09-16T05:26:41.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:41.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d7185640 5 monclient: authenticate success, global_id 124162 2024-09-16T05:26:41.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d7185640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:26:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 1 -- 172.21.15.114:0/674801911 reap_dead start 2024-09-16T05:26:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d5722640 1 --2- 172.21.15.114:0/674801911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4b80062e0 0x7fb4b8003530 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800c190 tx=0x7fb4c801cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a0b57bcf52dc20f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 -- 172.21.15.114:0/674801911 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8001ef0 con 0x7fb4b80062e0 2024-09-16T05:26:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: dump: 2024-09-16T05:26:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:41.539 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:41.540 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:41.540 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:41.540 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:41.540 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:41.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 -- 172.21.15.114:0/674801911 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4c80a9070 con 0x7fb4b80062e0 2024-09-16T05:26:41.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d7185640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:26:41.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:41.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d7185640 10 monclient: get_monmap_and_config success 2024-09-16T05:26:41.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4d7185640 4 set_mon_vals no callback set 2024-09-16T05:26:41.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 1 -- 172.21.15.114:0/674801911 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c80a5a60 con 0x7fb4b80062e0 2024-09-16T05:26:41.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:41.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:41.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.527+0000 7fb4cf7fe640 10 monclient: dump: 2024-09-16T05:26:41.541 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:41.541 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:41.541 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:41.541 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:41.541 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:41.541 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:41.541 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:41.541 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:41.541 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:41.541 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:41.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient: shutdown 2024-09-16T05:26:41.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:41.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 -- 172.21.15.114:0/674801911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4b80062e0 msgr2=0x7fb4b8003530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 --2- 172.21.15.114:0/674801911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4b80062e0 0x7fb4b8003530 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800c190 tx=0x7fb4c801cbe0 comp rx=0 tx=0).stop 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 -- 172.21.15.114:0/674801911 shutdown_connections 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 --2- 172.21.15.114:0/674801911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4b80062e0 0x7fb4b8003530 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 -- 172.21.15.114:0/674801911 >> 172.21.15.114:0/674801911 conn(0x7fb4d00a1fb0 msgr2=0x7fb4d00a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 -- 172.21.15.114:0/674801911 shutdown_connections 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 -- 172.21.15.114:0/674801911 wait complete. 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient: build_initial_monmap 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient: monmap: 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:26:41.542 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:26:41.543 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:26:41.543 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:41.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 librados: starting msgr at 2024-09-16T05:26:41.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 librados: starting objecter 2024-09-16T05:26:41.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 Processor -- start 2024-09-16T05:26:41.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 -- start start 2024-09-16T05:26:41.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 librados: setting wanted keys 2024-09-16T05:26:41.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 librados: calling monclient init 2024-09-16T05:26:41.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient: init 2024-09-16T05:26:41.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:41.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:26:41.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b80062e0 0x7fb4d0136890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:41.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient(hunting): picked mon.noname-a con 0x7fb4b80062e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:26:41.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4d0136dd0 0x7fb4d00706b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient(hunting): picked mon.noname-c con 0x7fb4d0136dd0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb4d0137210 0x7fb4d0070bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient(hunting): picked mon.noname-b con 0x7fb4d0137210 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:26:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d009cdc0 con 0x7fb4b80062e0 2024-09-16T05:26:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d009a010 con 0x7fb4d0137210 2024-09-16T05:26:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb4d009cf40 con 0x7fb4d0136dd0 2024-09-16T05:26:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient(hunting): authenticate will time out at 2086.162256s 2024-09-16T05:26:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d4f21640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b80062e0 0x7fb4d0136890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:41.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4d0136dd0 0x7fb4d00706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d4f21640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b80062e0 0x7fb4d0136890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37044/0 (socket says 172.21.15.114:37044) 2024-09-16T05:26:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4d0136dd0 0x7fb4d00706b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56012/0 (socket says 172.21.15.114:56012) 2024-09-16T05:26:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d4f21640 1 -- 172.21.15.114:0/3866554242 learned_addr learned my addr 172.21.15.114:0/3866554242 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d4f21640 10 monclient(hunting): get_auth_request con 0x7fb4b80062e0 auth_method 0 2024-09-16T05:26:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d4f21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d4f21640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d4f21640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 10 monclient(hunting): get_auth_request con 0x7fb4d0136dd0 auth_method 0 2024-09-16T05:26:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:41.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:41.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:41.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d4f21640 1 -- 172.21.15.114:0/3866554242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b80062e0 msgr2=0x7fb4d0136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:26:41.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d4f21640 1 -- 172.21.15.114:0/3866554242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b80062e0 msgr2=0x7fb4d0136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:26:41.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d4f21640 1 --2- 172.21.15.114:0/3866554242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b80062e0 0x7fb4d0136890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:41.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d5722640 1 --2- 172.21.15.114:0/3866554242 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb4d0137210 0x7fb4d0070bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:41.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d5722640 10 monclient(hunting): get_auth_request con 0x7fb4d0137210 auth_method 0 2024-09-16T05:26:41.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d4f21640 1 --2- 172.21.15.114:0/3866554242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b80062e0 0x7fb4d0136890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:26:41.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d5722640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:41.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d5722640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:41.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d5722640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:41.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 10 monclient(hunting): handle_auth_done global_id 104360 payload 824 2024-09-16T05:26:41.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 1 -- 172.21.15.114:0/3866554242 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb4d0137210 msgr2=0x7fb4d0070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:41.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 1 --2- 172.21.15.114:0/3866554242 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb4d0137210 0x7fb4d0070bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:41.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 1 -- 172.21.15.114:0/3866554242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b80062e0 msgr2=0x7fb4d0136890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:26:41.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 1 --2- 172.21.15.114:0/3866554242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b80062e0 0x7fb4d0136890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:41.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 10 monclient: _finish_hunting 0 2024-09-16T05:26:41.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 1 monclient: found mon.noname-c 2024-09-16T05:26:41.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:41.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:26:41.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 1 -- 172.21.15.114:0/3866554242 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d008d5a0 con 0x7fb4d0136dd0 2024-09-16T05:26:41.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d5722640 1 --2- 172.21.15.114:0/3866554242 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb4d0137210 0x7fb4d0070bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:26:41.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cffff640 1 --2- 172.21.15.114:0/3866554242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4d0136dd0 0x7fb4d00706b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4000fc0 tx=0x7fb4c400fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:41.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cd7fa640 10 client.?.objecter ms_handle_connect 0x7fb4d0136dd0 2024-09-16T05:26:41.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cd7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:26:41.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cd7fa640 1 -- 172.21.15.114:0/3866554242 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c40052e0 con 0x7fb4d0136dd0 2024-09-16T05:26:41.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:41.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:26:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cd7fa640 10 monclient: dump: 2024-09-16T05:26:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:41.548 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cd7fa640 10 monclient: _finish_auth 0 2024-09-16T05:26:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cd7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:26:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:41.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cd7fa640 1 -- 172.21.15.114:0/3866554242 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4c4018070 con 0x7fb4d0136dd0 2024-09-16T05:26:41.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 5 monclient: authenticate success, global_id 104360 2024-09-16T05:26:41.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient: _renew_subs 2024-09-16T05:26:41.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:41.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 -- 172.21.15.114:0/3866554242 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4d00726a0 con 0x7fb4d0136dd0 2024-09-16T05:26:41.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:41.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cd7fa640 1 -- 172.21.15.114:0/3866554242 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c400d940 con 0x7fb4d0136dd0 2024-09-16T05:26:41.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:41.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:41.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4cd7fa640 10 monclient: dump: 2024-09-16T05:26:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:41.550 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 client.104360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:26:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient: _renew_subs 2024-09-16T05:26:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 -- 172.21.15.114:0/3866554242 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4d009c6a0 con 0x7fb4d0136dd0 2024-09-16T05:26:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4ce7fc640 4 set_mon_vals no callback set 2024-09-16T05:26:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.531+0000 7fb4d7185640 1 librados: init done 2024-09-16T05:26:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.535+0000 7fb4d7185640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.535+0000 7fb4d7185640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.535+0000 7fb4d7185640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.535+0000 7fb4cd7fa640 1 -- 172.21.15.114:0/3866554242 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb4c400dae0 con 0x7fb4d0136dd0 2024-09-16T05:26:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.535+0000 7fb4d7185640 1 -- 172.21.15.114:0/3866554242 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4d00988e0 con 0x7fb4d0136dd0 2024-09-16T05:26:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.535+0000 7fb4cd7fa640 1 --2- 172.21.15.114:0/3866554242 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb4bc037810 0x7fb4bc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.535+0000 7fb4cd7fa640 1 -- 172.21.15.114:0/3866554242 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7fb4c4048b00 con 0x7fb4d0136dd0 2024-09-16T05:26:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.535+0000 7fb4cd7fa640 10 client.104360.objecter ms_dispatch 0x7fb4d0000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:26:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.535+0000 7fb4cd7fa640 3 client.104360.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:26:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.535+0000 7fb4cd7fa640 3 client.104360.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:26:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.535+0000 7fb4cd7fa640 20 client.104360.objecter dump_active .. 0 homeless 2024-09-16T05:26:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.535+0000 7fb4d4f21640 1 --2- 172.21.15.114:0/3866554242 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb4bc037810 0x7fb4bc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.535+0000 7fb4d4f21640 10 monclient: get_auth_request con 0x7fb4bc037810 auth_method 0 2024-09-16T05:26:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.535+0000 7fb4d4f21640 1 --2- 172.21.15.114:0/3866554242 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb4bc037810 0x7fb4bc039cd0 secure :-1 s=READY pgs=3620 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0006fd0 tx=0x7fb4c0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.535+0000 7fb4cd7fa640 10 client.104360.objecter ms_handle_connect 0x7fb4bc037810 2024-09-16T05:26:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.539+0000 7fb4cd7fa640 1 -- 172.21.15.114:0/3866554242 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb4c401a310 con 0x7fb4d0136dd0 2024-09-16T05:26:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.539+0000 7fb4cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.539+0000 7fb4cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:26:41.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:26:41.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:26:41.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:41.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 1 -- 172.21.15.114:0/3866554242 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb4d008d000 con 0x7fb4d0136dd0 2024-09-16T05:26:41.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4cd7fa640 1 -- 172.21.15.114:0/3866554242 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7fb4c4005480 con 0x7fb4d0136dd0 2024-09-16T05:26:41.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:26:41.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4cd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:26:41.760 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":72,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:26:41.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 10 librados: watch_flush enter 2024-09-16T05:26:41.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 10 librados: watch_flush exit 2024-09-16T05:26:41.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 20 client.104360.objecter shutdown clearing up homeless session... 2024-09-16T05:26:41.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 10 client.104360.objecter successfully canceled tick 2024-09-16T05:26:41.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 1 -- 172.21.15.114:0/3866554242 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb4bc037810 msgr2=0x7fb4bc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:41.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 1 --2- 172.21.15.114:0/3866554242 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb4bc037810 0x7fb4bc039cd0 secure :-1 s=READY pgs=3620 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0006fd0 tx=0x7fb4c0008040 comp rx=0 tx=0).stop 2024-09-16T05:26:41.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 10 monclient: shutdown 2024-09-16T05:26:41.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:41.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 1 -- 172.21.15.114:0/3866554242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4d0136dd0 msgr2=0x7fb4d00706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:41.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 1 --2- 172.21.15.114:0/3866554242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4d0136dd0 0x7fb4d00706b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4000fc0 tx=0x7fb4c400fc00 comp rx=0 tx=0).stop 2024-09-16T05:26:41.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 1 -- 172.21.15.114:0/3866554242 shutdown_connections 2024-09-16T05:26:41.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 1 --2- 172.21.15.114:0/3866554242 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb4bc037810 0x7fb4bc039cd0 unknown :-1 s=CLOSED pgs=3620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:41.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 1 --2- 172.21.15.114:0/3866554242 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb4d0137210 0x7fb4d0070bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:41.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 1 --2- 172.21.15.114:0/3866554242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb4d0136dd0 0x7fb4d00706b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:41.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 1 --2- 172.21.15.114:0/3866554242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4b80062e0 0x7fb4d0136890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:41.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 1 -- 172.21.15.114:0/3866554242 >> 172.21.15.114:0/3866554242 conn(0x7fb4d00a1fb0 msgr2=0x7fb4d01397e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:41.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 1 -- 172.21.15.114:0/3866554242 shutdown_connections 2024-09-16T05:26:41.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 1 -- 172.21.15.114:0/3866554242 wait complete. 2024-09-16T05:26:41.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:41.763+0000 7fb4d7185640 1 librados: shutdown 2024-09-16T05:26:44.778 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:26:44.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 10 monclient: get_monmap_and_config 2024-09-16T05:26:44.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 10 monclient: build_initial_monmap 2024-09-16T05:26:44.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 10 monclient: monmap: 2024-09-16T05:26:44.908 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:44.908 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:26:44.913509+0000 2024-09-16T05:26:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:26:44.913509+0000 2024-09-16T05:26:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:26:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:26:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:26:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:26:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:26:44.909 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:44.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 1 Processor -- start 2024-09-16T05:26:44.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 1 -- start start 2024-09-16T05:26:44.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 10 monclient: init 2024-09-16T05:26:44.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:44.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-16T05:26:44.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 0x7fdbf0072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:44.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 10 monclient(hunting): picked mon.noname-b con 0x7fdbf0071da0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:44.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 0x7fdbf00726c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:44.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 10 monclient(hunting): picked mon.noname-c con 0x7fdbf00a1ab0 addr v2:172.21.15.149:3300/0 2024-09-16T05:26:44.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdbf0099cc0 addr v1:172.21.15.149:6789/0 2024-09-16T05:26:44.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbf0099910 con 0x7fdbf0099cc0 2024-09-16T05:26:44.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:44.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fdbf00a1870 con 0x7fdbf00a1ab0 2024-09-16T05:26:44.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:44.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fdbf0072c00 con 0x7fdbf0071da0 2024-09-16T05:26:44.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:44.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf7089640 10 monclient(hunting): authenticate will time out at 2089.542699s 2024-09-16T05:26:44.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf4e25640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 0x7fdbf0072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:44.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf4e25640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 0x7fdbf0072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:56034/0 (socket says 172.21.15.114:56034) 2024-09-16T05:26:44.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf4e25640 1 -- 172.21.15.114:0/3521228281 learned_addr learned my addr 172.21.15.114:0/3521228281 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:44.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf4e25640 10 monclient(hunting): get_auth_request con 0x7fdbf0071da0 auth_method 0 2024-09-16T05:26:44.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf4e25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:44.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf4e25640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:44.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf4e25640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:44.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf4e25640 1 -- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 msgr2=0x7fdbf0072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:26:44.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf4e25640 1 -- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 msgr2=0x7fdbf0072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:44.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf4e25640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 0x7fdbf0072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:44.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbf4e25640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 0x7fdbf0072180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:26:44.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 0x7fdbf00726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:44.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbeffff640 10 monclient(hunting): get_auth_request con 0x7fdbf00a1ab0 auth_method 0 2024-09-16T05:26:44.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:44.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:44.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:44.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 msgr2=0x7fdbf00726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:26:44.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 msgr2=0x7fdbf00726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:44.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 0x7fdbf00726c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:44.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:44.911+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 0x7fdbf00726c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:26:45.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbf4e25640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 0x7fdbf0072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:45.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbf4e25640 10 monclient(hunting): get_auth_request con 0x7fdbf0071da0 auth_method 0 2024-09-16T05:26:45.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbf4e25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:45.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbf4e25640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:45.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbf4e25640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:45.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 0x7fdbf00726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:45.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbeffff640 10 monclient(hunting): get_auth_request con 0x7fdbf00a1ab0 auth_method 0 2024-09-16T05:26:45.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:45.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:45.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbeffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:45.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 msgr2=0x7fdbf00726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:26:45.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbf4e25640 1 -- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 msgr2=0x7fdbf0072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:26:45.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 msgr2=0x7fdbf00726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:45.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbf4e25640 1 -- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 msgr2=0x7fdbf0072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:45.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 0x7fdbf00726c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:45.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbf4e25640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 0x7fdbf0072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:45.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 0x7fdbf00726c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:26:45.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.115+0000 7fdbf4e25640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 0x7fdbf0072180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:26:45.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbf4e25640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 0x7fdbf0072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 0x7fdbf00726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbf4e25640 10 monclient(hunting): get_auth_request con 0x7fdbf0071da0 auth_method 0 2024-09-16T05:26:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbf4e25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbf4e25640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbf4e25640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbeffff640 10 monclient(hunting): get_auth_request con 0x7fdbf00a1ab0 auth_method 0 2024-09-16T05:26:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbeffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 msgr2=0x7fdbf00726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:26:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 msgr2=0x7fdbf00726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:45.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 0x7fdbf00726c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:45.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbf4e25640 1 -- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 msgr2=0x7fdbf0072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:26:45.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbf4e25640 1 -- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 msgr2=0x7fdbf0072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:45.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbf4e25640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 0x7fdbf0072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:45.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 0x7fdbf00726c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:26:45.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:45.519+0000 7fdbf4e25640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 0x7fdbf0072180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:26:46.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbf4e25640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 0x7fdbf0072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:46.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbf4e25640 10 monclient(hunting): get_auth_request con 0x7fdbf0071da0 auth_method 0 2024-09-16T05:26:46.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbf4e25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:46.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbf4e25640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:46.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbf4e25640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:46.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 0x7fdbf00726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:46.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbeffff640 10 monclient(hunting): get_auth_request con 0x7fdbf00a1ab0 auth_method 0 2024-09-16T05:26:46.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:46.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:46.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbeffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:46.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbf4e25640 1 -- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 msgr2=0x7fdbf0072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:26:46.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbf4e25640 1 -- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 msgr2=0x7fdbf0072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:46.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbf4e25640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 0x7fdbf0072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:46.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbf4e25640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 0x7fdbf0072180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:26:46.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 msgr2=0x7fdbf00726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:26:46.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 msgr2=0x7fdbf00726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:46.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 0x7fdbf00726c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:46.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:46.323+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 0x7fdbf00726c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:26:47.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 10 monclient(hunting): tick 2024-09-16T05:26:47.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:26:47.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 1 monclient(hunting): continuing hunt 2024-09-16T05:26:47.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:26:47.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 1 -- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 msgr2=0x7fdbf0072180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:26:47.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbf0071da0 0x7fdbf0072180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:47.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 msgr2=0x7fdbf00726c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:26:47.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.149:3300/0 conn(0x7fdbf00a1ab0 0x7fdbf00726c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:47.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 1 -- 172.21.15.114:0/3521228281 >> v1:172.21.15.149:6789/0 conn(0x7fdbf0099cc0 legacy=0x7fdbf0137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:47.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 10 monclient: _add_conns ranks=[0,3] 2024-09-16T05:26:47.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdbdc0036d0 addr v1:172.21.15.114:6789/0 2024-09-16T05:26:47.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 0x7fdbdc0098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:47.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fdbdc007420 addr v2:172.21.15.114:3300/0 2024-09-16T05:26:47.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 1 -- 172.21.15.114:0/3521228281 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbf0073e20 con 0x7fdbdc0036d0 2024-09-16T05:26:47.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:47.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 1 -- 172.21.15.114:0/3521228281 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fdbf00a1870 con 0x7fdbdc007420 2024-09-16T05:26:47.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbee7fc640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:47.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 0x7fdbdc0098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:47.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbeffff640 10 monclient(hunting): get_auth_request con 0x7fdbdc007420 auth_method 0 2024-09-16T05:26:47.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:47.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:47.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:47.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 msgr2=0x7fdbdc0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:26:47.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 msgr2=0x7fdbdc0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:47.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 0x7fdbdc0098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:47.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:47.911+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 0x7fdbdc0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:26:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.115+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 0x7fdbdc0098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.115+0000 7fdbeffff640 10 monclient(hunting): get_auth_request con 0x7fdbdc007420 auth_method 0 2024-09-16T05:26:48.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.115+0000 7fdbeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.115+0000 7fdbeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.115+0000 7fdbeffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.115+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 msgr2=0x7fdbdc0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.115+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 msgr2=0x7fdbdc0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.115+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 0x7fdbdc0098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.115+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 0x7fdbdc0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:26:48.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.519+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 0x7fdbdc0098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:48.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.519+0000 7fdbeffff640 10 monclient(hunting): get_auth_request con 0x7fdbdc007420 auth_method 0 2024-09-16T05:26:48.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.519+0000 7fdbeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:48.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.519+0000 7fdbeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:48.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.519+0000 7fdbeffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:48.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.519+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 msgr2=0x7fdbdc0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T05:26:48.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.519+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 msgr2=0x7fdbdc0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:48.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.519+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 0x7fdbdc0098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:48.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:48.519+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 0x7fdbdc0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:26:49.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 -- 172.21.15.114:0/3521228281 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 374026419 0 0) 0x7fdbf0073e20 con 0x7fdbdc0036d0 2024-09-16T05:26:49.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:26:49.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:49.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:49.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient(hunting): my global_id is 134097 2024-09-16T05:26:49.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 -- 172.21.15.114:0/3521228281 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbd8003530 con 0x7fdbdc0036d0 2024-09-16T05:26:49.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 -- 172.21.15.114:0/3521228281 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 290100208 0 0) 0x7fdbd8003530 con 0x7fdbdc0036d0 2024-09-16T05:26:49.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 -- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 msgr2=0x7fdbdc0098c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:26:49.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 --2- 172.21.15.114:0/3521228281 >> v2:172.21.15.114:3300/0 conn(0x7fdbdc007420 0x7fdbdc0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:49.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:26:49.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:26:49.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:49.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:26:49.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 -- 172.21.15.114:0/3521228281 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbdc00b090 con 0x7fdbdc0036d0 2024-09-16T05:26:49.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient: _finish_auth 0 2024-09-16T05:26:49.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:26:49.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:49.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 -- 172.21.15.114:0/3521228281 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7fdbe000f040 con 0x7fdbdc0036d0 2024-09-16T05:26:49.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:49.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:26:49.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient: dump: 2024-09-16T05:26:49.183 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:49.183 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:49.183 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:49.183 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:49.183 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:49.183 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:49.183 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:49.183 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:49.183 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:49.183 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:49.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:26:49.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:49.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 -- 172.21.15.114:0/3521228281 >> v1:172.21.15.114:6789/0 conn(0x7fdbdc0036d0 legacy=0x7fdbdc005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:49.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf5626640 1 -- 172.21.15.114:0/3521228281 reap_dead start 2024-09-16T05:26:49.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:26:49.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdbd8003a10 0x7fdbd8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:49.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient(hunting): picked mon.b con 0x7fdbd8003a10 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:26:49.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbd8004810 0x7fdbd8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:49.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient(hunting): picked mon.c con 0x7fdbd8004810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:49.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd80053d0 0x7fdbd800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:49.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient(hunting): picked mon.a con 0x7fdbd80053d0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:26:49.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:49.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 -- 172.21.15.114:0/3521228281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf00a1870 con 0x7fdbd80053d0 2024-09-16T05:26:49.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:49.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 -- 172.21.15.114:0/3521228281 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd800c290 con 0x7fdbd8003a10 2024-09-16T05:26:49.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:49.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 1 -- 172.21.15.114:0/3521228281 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fdbd800c490 con 0x7fdbd8004810 2024-09-16T05:26:49.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:49.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbef7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:26:49.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbd8004810 0x7fdbd8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:49.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf5626640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd80053d0 0x7fdbd800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:49.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 10 monclient(hunting): get_auth_request con 0x7fdbd8004810 auth_method 0 2024-09-16T05:26:49.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:49.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:49.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:49.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf4e25640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdbd8003a10 0x7fdbd8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:49.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf5626640 10 monclient(hunting): get_auth_request con 0x7fdbd80053d0 auth_method 0 2024-09-16T05:26:49.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf5626640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:49.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf5626640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:49.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf5626640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:49.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf4e25640 10 monclient(hunting): get_auth_request con 0x7fdbd8003a10 auth_method 0 2024-09-16T05:26:49.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf4e25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:49.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf4e25640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:49.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf4e25640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:49.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:49.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:49.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:26:49.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf5626640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:49.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf5626640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:49.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf5626640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:26:49.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf4e25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:49.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf4e25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:49.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf4e25640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:26:49.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 10 monclient(hunting): handle_auth_done global_id 134097 payload 293 2024-09-16T05:26:49.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdbd8003a10 msgr2=0x7fdbd8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:26:49.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdbd8003a10 0x7fdbd8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:49.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd80053d0 msgr2=0x7fdbd800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:26:49.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd80053d0 0x7fdbd800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:49.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 10 monclient: _finish_hunting 0 2024-09-16T05:26:49.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 1 monclient: found mon.c 2024-09-16T05:26:49.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:49.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:49.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 1 -- 172.21.15.114:0/3521228281 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd8003530 con 0x7fdbd8004810 2024-09-16T05:26:49.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 10 monclient: _finish_auth 0 2024-09-16T05:26:49.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 10 monclient: _check_auth_tickets 2024-09-16T05:26:49.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:49.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf7089640 5 monclient: authenticate success, global_id 134097 2024-09-16T05:26:49.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.183+0000 7fdbf7089640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:26:49.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.187+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbd8004810 0x7fdbd8004c90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4006f80 tx=0x7fdbe401df10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f28177f7d9d46093 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:49.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.187+0000 7fdbef7fe640 1 -- 172.21.15.114:0/3521228281 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe40ac070 con 0x7fdbd8004810 2024-09-16T05:26:49.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.187+0000 7fdbef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:49.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.187+0000 7fdbef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:49.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.187+0000 7fdbef7fe640 10 monclient: dump: 2024-09-16T05:26:49.197 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:49.197 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:49.198 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:49.198 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:49.198 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:49.198 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:49.198 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:49.198 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:49.198 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:49.198 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:49.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.187+0000 7fdbef7fe640 1 -- 172.21.15.114:0/3521228281 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbe40a6b00 con 0x7fdbd8004810 2024-09-16T05:26:49.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.187+0000 7fdbf7089640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:26:49.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.187+0000 7fdbef7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:49.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.187+0000 7fdbf7089640 10 monclient: get_monmap_and_config success 2024-09-16T05:26:49.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.187+0000 7fdbef7fe640 1 -- 172.21.15.114:0/3521228281 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe40178d0 con 0x7fdbd8004810 2024-09-16T05:26:49.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.187+0000 7fdbf7089640 4 set_mon_vals no callback set 2024-09-16T05:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.187+0000 7fdbef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.187+0000 7fdbef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.187+0000 7fdbef7fe640 10 monclient: dump: 2024-09-16T05:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:49.200 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:49.200 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:49.200 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:49.200 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:49.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 10 monclient: shutdown 2024-09-16T05:26:49.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:49.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 -- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbd8004810 msgr2=0x7fdbd8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:49.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbd8004810 0x7fdbd8004c90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4006f80 tx=0x7fdbe401df10 comp rx=0 tx=0).stop 2024-09-16T05:26:49.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 -- 172.21.15.114:0/3521228281 shutdown_connections 2024-09-16T05:26:49.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd80053d0 0x7fdbd800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:49.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbd8004810 0x7fdbd8004c90 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:49.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 --2- 172.21.15.114:0/3521228281 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdbd8003a10 0x7fdbd8003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:49.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 -- 172.21.15.114:0/3521228281 >> 172.21.15.114:0/3521228281 conn(0x7fdbf00a1fb0 msgr2=0x7fdbf00a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:49.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 -- 172.21.15.114:0/3521228281 shutdown_connections 2024-09-16T05:26:49.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 -- 172.21.15.114:0/3521228281 wait complete. 2024-09-16T05:26:49.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 10 monclient: build_initial_monmap 2024-09-16T05:26:49.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 10 monclient: monmap: 2024-09-16T05:26:49.201 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:49.201 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:26:49.201 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:26:49.201 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:26:49.201 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:49.201 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:49.201 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:26:49.202 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:26:49.202 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:26:49.202 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:49.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 librados: starting msgr at 2024-09-16T05:26:49.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 librados: starting objecter 2024-09-16T05:26:49.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 Processor -- start 2024-09-16T05:26:49.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 -- start start 2024-09-16T05:26:49.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 librados: setting wanted keys 2024-09-16T05:26:49.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 librados: calling monclient init 2024-09-16T05:26:49.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 10 monclient: init 2024-09-16T05:26:49.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:49.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:26:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8003a10 0x7fdbf01cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 10 monclient(hunting): picked mon.noname-a con 0x7fdbd8003a10 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:26:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdbd8004810 0x7fdbf00983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 10 monclient(hunting): picked mon.noname-b con 0x7fdbd8004810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:26:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbd80053d0 0x7fdbf0098930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 10 monclient(hunting): picked mon.noname-c con 0x7fdbd80053d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf00735a0 con 0x7fdbd8003a10 2024-09-16T05:26:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf0073420 con 0x7fdbd8004810 2024-09-16T05:26:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fdbf0072180 con 0x7fdbd80053d0 2024-09-16T05:26:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf7089640 10 monclient(hunting): authenticate will time out at 2093.822891s 2024-09-16T05:26:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8003a10 0x7fdbf01cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:49.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf5626640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbd80053d0 0x7fdbf0098930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:49.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8003a10 0x7fdbf01cbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59586/0 (socket says 172.21.15.114:59586) 2024-09-16T05:26:49.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 1 -- 172.21.15.114:0/3741830111 learned_addr learned my addr 172.21.15.114:0/3741830111 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:49.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 10 monclient(hunting): get_auth_request con 0x7fdbd8003a10 auth_method 0 2024-09-16T05:26:49.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:49.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:49.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:49.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf5626640 10 monclient(hunting): get_auth_request con 0x7fdbd80053d0 auth_method 0 2024-09-16T05:26:49.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf5626640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:49.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf5626640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:49.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf5626640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:49.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:49.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:49.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:49.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf5626640 1 -- 172.21.15.114:0/3741830111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbd80053d0 msgr2=0x7fdbf0098930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T05:26:49.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf5626640 1 -- 172.21.15.114:0/3741830111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbd80053d0 msgr2=0x7fdbf0098930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:26:49.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf5626640 1 --2- 172.21.15.114:0/3741830111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbd80053d0 0x7fdbf0098930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:49.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3741830111 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdbd8004810 0x7fdbf00983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:49.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf5626640 1 --2- 172.21.15.114:0/3741830111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbd80053d0 0x7fdbf0098930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:26:49.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbeffff640 10 monclient(hunting): get_auth_request con 0x7fdbd8004810 auth_method 0 2024-09-16T05:26:49.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:49.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:49.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:49.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 10 monclient(hunting): handle_auth_done global_id 134100 payload 824 2024-09-16T05:26:49.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 1 -- 172.21.15.114:0/3741830111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbd80053d0 msgr2=0x7fdbf0098930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:26:49.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 1 --2- 172.21.15.114:0/3741830111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbd80053d0 0x7fdbf0098930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:49.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 1 -- 172.21.15.114:0/3741830111 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdbd8004810 msgr2=0x7fdbf00983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:49.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 1 --2- 172.21.15.114:0/3741830111 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdbd8004810 0x7fdbf00983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:49.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 10 monclient: _finish_hunting 0 2024-09-16T05:26:49.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 1 monclient: found mon.noname-a 2024-09-16T05:26:49.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:49.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:26:49.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 1 -- 172.21.15.114:0/3741830111 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbf0091030 con 0x7fdbd8003a10 2024-09-16T05:26:49.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbf4e25640 1 --2- 172.21.15.114:0/3741830111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8003a10 0x7fdbf01cbea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdbe000c5c0 tx=0x7fdbe0018ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:49.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbed7fa640 10 client.?.objecter ms_handle_connect 0x7fdbd8003a10 2024-09-16T05:26:49.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.191+0000 7fdbed7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:26:49.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 1 -- 172.21.15.114:0/3741830111 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe000f040 con 0x7fdbd8003a10 2024-09-16T05:26:49.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:49.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:26:49.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 10 monclient: dump: 2024-09-16T05:26:49.206 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:49.207 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 10 monclient: _finish_auth 0 2024-09-16T05:26:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:26:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:49.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 1 -- 172.21.15.114:0/3741830111 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbe0000e50 con 0x7fdbd8003a10 2024-09-16T05:26:49.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:49.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 1 -- 172.21.15.114:0/3741830111 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0003860 con 0x7fdbd8003a10 2024-09-16T05:26:49.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:49.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:26:49.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 10 monclient: dump: 2024-09-16T05:26:49.208 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:49.208 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:49.208 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:49.208 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:49.208 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:49.208 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:49.208 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:49.209 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:49.209 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:49.209 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:49.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbeeffd640 4 set_mon_vals no callback set 2024-09-16T05:26:49.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbf7089640 5 monclient: authenticate success, global_id 134100 2024-09-16T05:26:49.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbf7089640 10 monclient: _renew_subs 2024-09-16T05:26:49.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbf7089640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:49.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbf7089640 1 -- 172.21.15.114:0/3741830111 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbf0092ba0 con 0x7fdbd8003a10 2024-09-16T05:26:49.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbf7089640 10 client.134100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:26:49.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbf7089640 10 monclient: _renew_subs 2024-09-16T05:26:49.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbf7089640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:49.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbf7089640 1 -- 172.21.15.114:0/3741830111 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbf00a45a0 con 0x7fdbd8003a10 2024-09-16T05:26:49.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbf7089640 1 librados: init done 2024-09-16T05:26:49.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 1 -- 172.21.15.114:0/3741830111 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fdbe001f020 con 0x7fdbd8003a10 2024-09-16T05:26:49.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbf7089640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:49.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbf7089640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:49.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbf7089640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:49.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbf7089640 1 -- 172.21.15.114:0/3741830111 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbf009c620 con 0x7fdbd8003a10 2024-09-16T05:26:49.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 1 --2- 172.21.15.114:0/3741830111 >> v2:172.21.15.149:6812/1185531135 conn(0x7fdbdc038020 0x7fdbdc03a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:49.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 1 -- 172.21.15.114:0/3741830111 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7fdbe004c040 con 0x7fdbd8003a10 2024-09-16T05:26:49.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 10 client.134100.objecter ms_dispatch 0x7fdbf0000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:26:49.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 3 client.134100.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:26:49.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 3 client.134100.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:26:49.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 20 client.134100.objecter dump_active .. 0 homeless 2024-09-16T05:26:49.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3741830111 >> v2:172.21.15.149:6812/1185531135 conn(0x7fdbdc038020 0x7fdbdc03a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:49.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbeffff640 10 monclient: get_auth_request con 0x7fdbdc038020 auth_method 0 2024-09-16T05:26:49.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbeffff640 1 --2- 172.21.15.114:0/3741830111 >> v2:172.21.15.149:6812/1185531135 conn(0x7fdbdc038020 0x7fdbdc03a4e0 secure :-1 s=READY pgs=3621 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40ae000 tx=0x7fdbe40b2040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:49.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.195+0000 7fdbed7fa640 10 client.134100.objecter ms_handle_connect 0x7fdbdc038020 2024-09-16T05:26:49.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.199+0000 7fdbed7fa640 1 -- 172.21.15.114:0/3741830111 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fdbe00293a0 con 0x7fdbd8003a10 2024-09-16T05:26:49.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.199+0000 7fdbed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:49.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.199+0000 7fdbed7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:26:49.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.439+0000 7fdbf7089640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:26:49.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.439+0000 7fdbf7089640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:26:49.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.439+0000 7fdbf7089640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:49.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.439+0000 7fdbf7089640 1 -- 172.21.15.114:0/3741830111 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdbf0072560 con 0x7fdbd8003a10 2024-09-16T05:26:49.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.439+0000 7fdbed7fa640 1 -- 172.21.15.114:0/3741830111 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fdbe0003c90 con 0x7fdbd8003a10 2024-09-16T05:26:49.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.439+0000 7fdbed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:26:49.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.439+0000 7fdbed7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:26:49.438 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":76,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:26:49.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 10 librados: watch_flush enter 2024-09-16T05:26:49.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 10 librados: watch_flush exit 2024-09-16T05:26:49.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 20 client.134100.objecter shutdown clearing up homeless session... 2024-09-16T05:26:49.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 10 client.134100.objecter successfully canceled tick 2024-09-16T05:26:49.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 1 -- 172.21.15.114:0/3741830111 >> v2:172.21.15.149:6812/1185531135 conn(0x7fdbdc038020 msgr2=0x7fdbdc03a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:49.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 1 --2- 172.21.15.114:0/3741830111 >> v2:172.21.15.149:6812/1185531135 conn(0x7fdbdc038020 0x7fdbdc03a4e0 secure :-1 s=READY pgs=3621 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40ae000 tx=0x7fdbe40b2040 comp rx=0 tx=0).stop 2024-09-16T05:26:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 10 monclient: shutdown 2024-09-16T05:26:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 1 -- 172.21.15.114:0/3741830111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8003a10 msgr2=0x7fdbf01cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 1 --2- 172.21.15.114:0/3741830111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8003a10 0x7fdbf01cbea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdbe000c5c0 tx=0x7fdbe0018ac0 comp rx=0 tx=0).stop 2024-09-16T05:26:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 1 -- 172.21.15.114:0/3741830111 shutdown_connections 2024-09-16T05:26:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 1 --2- 172.21.15.114:0/3741830111 >> v2:172.21.15.149:6812/1185531135 conn(0x7fdbdc038020 0x7fdbdc03a4e0 unknown :-1 s=CLOSED pgs=3621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 1 --2- 172.21.15.114:0/3741830111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdbd80053d0 0x7fdbf0098930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 1 --2- 172.21.15.114:0/3741830111 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdbd8004810 0x7fdbf00983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 1 --2- 172.21.15.114:0/3741830111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbd8003a10 0x7fdbf01cbea0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 1 -- 172.21.15.114:0/3741830111 >> 172.21.15.114:0/3741830111 conn(0x7fdbf00a1fb0 msgr2=0x7fdbf01bf910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 1 -- 172.21.15.114:0/3741830111 shutdown_connections 2024-09-16T05:26:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 1 -- 172.21.15.114:0/3741830111 wait complete. 2024-09-16T05:26:49.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:49.443+0000 7fdbf7089640 1 librados: shutdown 2024-09-16T05:26:52.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:26:52.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 10 monclient: get_monmap_and_config 2024-09-16T05:26:52.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 10 monclient: build_initial_monmap 2024-09-16T05:26:52.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 10 monclient: monmap: 2024-09-16T05:26:52.589 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:52.589 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:52.589 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:26:52.594324+0000 2024-09-16T05:26:52.589 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:26:52.594324+0000 2024-09-16T05:26:52.589 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:52.589 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:52.589 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:26:52.590 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:26:52.590 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:26:52.590 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:26:52.590 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:26:52.590 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:52.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 1 Processor -- start 2024-09-16T05:26:52.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 1 -- start start 2024-09-16T05:26:52.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 10 monclient: init 2024-09-16T05:26:52.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:52.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-16T05:26:52.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c0a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:52.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 10 monclient(hunting): picked mon.noname-b con 0x7fb52c09b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:52.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb52c0a1400 addr v1:172.21.15.114:6789/0 2024-09-16T05:26:52.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 0x7fb52c13cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:52.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 10 monclient(hunting): picked mon.noname-a con 0x7fb52c0a17e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:26:52.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb52c09d4e0 con 0x7fb52c0a1400 2024-09-16T05:26:52.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb52c09e390 con 0x7fb52c0a17e0 2024-09-16T05:26:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb52c09ece0 con 0x7fb52c09b6c0 2024-09-16T05:26:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb5315df640 10 monclient(hunting): authenticate will time out at 2097.223491s 2024-09-16T05:26:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52affd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c0a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52b7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 0x7fb52c13cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52affd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c0a0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:37634/0 (socket says 172.21.15.114:37634) 2024-09-16T05:26:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52affd640 1 -- 172.21.15.114:0/152603252 learned_addr learned my addr 172.21.15.114:0/152603252 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52affd640 10 monclient(hunting): get_auth_request con 0x7fb52c09b6c0 auth_method 0 2024-09-16T05:26:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52b7fe640 10 monclient(hunting): get_auth_request con 0x7fb52c0a17e0 auth_method 0 2024-09-16T05:26:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52affd640 1 -- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 msgr2=0x7fb52c0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:26:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52affd640 1 -- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 msgr2=0x7fb52c0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52affd640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c0a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52b7fe640 1 -- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 msgr2=0x7fb52c13cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:26:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52b7fe640 1 -- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 msgr2=0x7fb52c13cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 0x7fb52c13cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52affd640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c0a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:26:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.591+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 0x7fb52c13cd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:26:52.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 0x7fb52c13cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:52.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52affd640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c0a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:52.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52affd640 10 monclient(hunting): get_auth_request con 0x7fb52c09b6c0 auth_method 0 2024-09-16T05:26:52.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:52.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:52.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:52.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52b7fe640 10 monclient(hunting): get_auth_request con 0x7fb52c0a17e0 auth_method 0 2024-09-16T05:26:52.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:52.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:52.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:52.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52b7fe640 1 -- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 msgr2=0x7fb52c13cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:26:52.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52b7fe640 1 -- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 msgr2=0x7fb52c13cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:52.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 0x7fb52c13cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:52.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52affd640 1 -- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 msgr2=0x7fb52c0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:26:52.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52affd640 1 -- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 msgr2=0x7fb52c0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:52.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52affd640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c0a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:52.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 0x7fb52c13cd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:26:52.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:52.795+0000 7fb52affd640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c0a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T05:26:53.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 0x7fb52c13cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:53.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52affd640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c0a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:53.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52affd640 10 monclient(hunting): get_auth_request con 0x7fb52c09b6c0 auth_method 0 2024-09-16T05:26:53.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:53.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:53.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:53.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52b7fe640 10 monclient(hunting): get_auth_request con 0x7fb52c0a17e0 auth_method 0 2024-09-16T05:26:53.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:53.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:53.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:53.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52affd640 1 -- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 msgr2=0x7fb52c0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:26:53.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52affd640 1 -- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 msgr2=0x7fb52c0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:53.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52b7fe640 1 -- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 msgr2=0x7fb52c13cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:26:53.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52affd640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c0a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:53.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52b7fe640 1 -- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 msgr2=0x7fb52c13cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:53.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 0x7fb52c13cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:53.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52affd640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c0a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:26:53.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:53.199+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 0x7fb52c13cd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T05:26:53.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 0x7fb52c13cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:53.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52affd640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c0a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:54.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52affd640 10 monclient(hunting): get_auth_request con 0x7fb52c09b6c0 auth_method 0 2024-09-16T05:26:54.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:54.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:54.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:54.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52b7fe640 10 monclient(hunting): get_auth_request con 0x7fb52c0a17e0 auth_method 0 2024-09-16T05:26:54.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:54.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:54.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:54.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52b7fe640 1 -- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 msgr2=0x7fb52c13cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T05:26:54.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52affd640 1 -- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 msgr2=0x7fb52c0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T05:26:54.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52b7fe640 1 -- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 msgr2=0x7fb52c13cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:54.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52affd640 1 -- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 msgr2=0x7fb52c0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T05:26:54.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 0x7fb52c13cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:54.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52affd640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c0a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:54.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52affd640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c0a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:26:54.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:54.003+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 0x7fb52c13cd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T05:26:55.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 1 -- 172.21.15.114:0/152603252 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 535194467 0 0) 0x7fb52c09d4e0 con 0x7fb52c0a1400 2024-09-16T05:26:55.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:26:55.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:55.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:55.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 10 monclient(hunting): my global_id is 134103 2024-09-16T05:26:55.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 1 -- 172.21.15.114:0/152603252 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb518003530 con 0x7fb52c0a1400 2024-09-16T05:26:55.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 1 -- 172.21.15.114:0/152603252 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3705667497 0 0) 0x7fb518003530 con 0x7fb52c0a1400 2024-09-16T05:26:55.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 1 -- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 msgr2=0x7fb52c0a0ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:26:55.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c0a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:55.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 1 -- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 msgr2=0x7fb52c13cd70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:26:55.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 1 --2- 172.21.15.114:0/152603252 >> v2:172.21.15.114:3300/0 conn(0x7fb52c0a17e0 0x7fb52c13cd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:55.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:26:55.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:26:55.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:55.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:26:55.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 1 -- 172.21.15.114:0/152603252 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52c09f430 con 0x7fb52c0a1400 2024-09-16T05:26:55.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 10 monclient: _finish_auth 0 2024-09-16T05:26:55.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:26:55.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:55.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 1 -- 172.21.15.114:0/152603252 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7fb5140016f0 con 0x7fb52c0a1400 2024-09-16T05:26:55.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:55.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:26:55.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 10 monclient: dump: 2024-09-16T05:26:55.177 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:55.177 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:55.178 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:55.178 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:55.178 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:55.178 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:55.178 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:55.178 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:55.178 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:55.178 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:55.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:26:55.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:55.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 1 -- 172.21.15.114:0/152603252 >> v1:172.21.15.114:6789/0 conn(0x7fb52c0a1400 legacy=0x7fb52c139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:55.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.175+0000 7fb529ffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:26:55.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb529ffb640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5180039c0 0x7fb518003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:55.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb529ffb640 10 monclient(hunting): picked mon.a con 0x7fb5180039c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:26:55.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb529ffb640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb518004810 0x7fb518004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:55.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb529ffb640 10 monclient(hunting): picked mon.b con 0x7fb518004810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:26:55.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb529ffb640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb5180052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:55.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb529ffb640 10 monclient(hunting): picked mon.c con 0x7fb52c09b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:55.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb529ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:55.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb529ffb640 1 -- 172.21.15.114:0/152603252 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c09e390 con 0x7fb5180039c0 2024-09-16T05:26:55.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb529ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:55.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb529ffb640 1 -- 172.21.15.114:0/152603252 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c09ece0 con 0x7fb518004810 2024-09-16T05:26:55.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb529ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:55.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb529ffb640 1 -- 172.21.15.114:0/152603252 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb51800bc10 con 0x7fb52c09b6c0 2024-09-16T05:26:55.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb529ffb640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:55.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb529ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:26:55.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb5180052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:55.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52affd640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5180039c0 0x7fb518003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:55.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 10 monclient(hunting): get_auth_request con 0x7fb52c09b6c0 auth_method 0 2024-09-16T05:26:55.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:55.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:55.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:55.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52affd640 10 monclient(hunting): get_auth_request con 0x7fb5180039c0 auth_method 0 2024-09-16T05:26:55.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:55.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:55.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:55.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:55.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:55.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:26:55.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52a7fc640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb518004810 0x7fb518004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:55.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:55.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:55.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:26:55.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52a7fc640 10 monclient(hunting): get_auth_request con 0x7fb518004810 auth_method 0 2024-09-16T05:26:55.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:55.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:55.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:26:55.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:55.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:55.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:26:55.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 10 monclient(hunting): handle_auth_done global_id 134103 payload 293 2024-09-16T05:26:55.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 1 -- 172.21.15.114:0/152603252 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb518004810 msgr2=0x7fb518004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:26:55.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb518004810 0x7fb518004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:55.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 1 -- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5180039c0 msgr2=0x7fb518003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:26:55.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5180039c0 0x7fb518003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:55.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:26:55.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 1 monclient: found mon.c 2024-09-16T05:26:55.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:55.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:55.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 1 -- 172.21.15.114:0/152603252 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb518003530 con 0x7fb52c09b6c0 2024-09-16T05:26:55.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:26:55.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:26:55.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:55.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb52affd640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5180039c0 0x7fb518003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:26:55.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb5315df640 5 monclient: authenticate success, global_id 134103 2024-09-16T05:26:55.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.179+0000 7fb5315df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:26:55.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.183+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb5180052a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb51c0129d0 tx=0x7fb51c01d0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d72d41d2dbc76d33 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:55.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.183+0000 7fb529ffb640 1 -- 172.21.15.114:0/152603252 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c0ab070 con 0x7fb52c09b6c0 2024-09-16T05:26:55.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.183+0000 7fb529ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:55.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.183+0000 7fb529ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:55.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.183+0000 7fb529ffb640 10 monclient: dump: 2024-09-16T05:26:55.184 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:55.184 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:55.184 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:55.185 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:55.185 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:55.185 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:55.185 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:55.185 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:55.185 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:55.185 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:55.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.183+0000 7fb5315df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:26:55.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.183+0000 7fb529ffb640 1 -- 172.21.15.114:0/152603252 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb51c0a5da0 con 0x7fb52c09b6c0 2024-09-16T05:26:55.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.183+0000 7fb529ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:55.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.183+0000 7fb5315df640 10 monclient: get_monmap_and_config success 2024-09-16T05:26:55.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.183+0000 7fb5315df640 4 set_mon_vals no callback set 2024-09-16T05:26:55.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.183+0000 7fb529ffb640 1 -- 172.21.15.114:0/152603252 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c00f040 con 0x7fb52c09b6c0 2024-09-16T05:26:55.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.183+0000 7fb529ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:55.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.183+0000 7fb529ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:55.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.183+0000 7fb529ffb640 10 monclient: dump: 2024-09-16T05:26:55.186 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:55.186 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:55.186 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:55.186 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:55.186 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:55.186 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:55.186 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:55.186 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:55.186 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:55.186 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:55.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 10 monclient: shutdown 2024-09-16T05:26:55.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:55.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 1 -- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 msgr2=0x7fb5180052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:55.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 1 --2- 172.21.15.114:0/152603252 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb5180052a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb51c0129d0 tx=0x7fb51c01d0b0 comp rx=0 tx=0).stop 2024-09-16T05:26:55.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb52b7fe640 1 -- 172.21.15.114:0/152603252 reap_dead start 2024-09-16T05:26:55.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 1 -- 172.21.15.114:0/152603252 shutdown_connections 2024-09-16T05:26:55.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 1 -- 172.21.15.114:0/152603252 >> 172.21.15.114:0/152603252 conn(0x7fb52c0a1fb0 msgr2=0x7fb52c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:55.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 1 -- 172.21.15.114:0/152603252 shutdown_connections 2024-09-16T05:26:55.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 1 -- 172.21.15.114:0/152603252 wait complete. 2024-09-16T05:26:55.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 10 monclient: build_initial_monmap 2024-09-16T05:26:55.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 10 monclient: monmap: 2024-09-16T05:26:55.191 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:55.191 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:26:55.191 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:26:55.191 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:26:55.191 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:55.191 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:55.191 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:26:55.192 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:26:55.192 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:26:55.192 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:55.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 1 librados: starting msgr at 2024-09-16T05:26:55.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 1 librados: starting objecter 2024-09-16T05:26:55.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 1 Processor -- start 2024-09-16T05:26:55.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 1 -- start start 2024-09-16T05:26:55.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 1 librados: setting wanted keys 2024-09-16T05:26:55.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 1 librados: calling monclient init 2024-09-16T05:26:55.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.187+0000 7fb5315df640 10 monclient: init 2024-09-16T05:26:55.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:55.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:26:55.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c12a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:55.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient(hunting): picked mon.noname-c con 0x7fb52c09b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:55.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb52c12aab0 0x7fb52c12af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:55.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient(hunting): picked mon.noname-a con 0x7fb52c12aab0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:26:55.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb52c12b470 0x7fb52c1d5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:55.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient(hunting): picked mon.noname-b con 0x7fb52c12b470 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:26:55.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:55.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c0a1bf0 con 0x7fb52c12aab0 2024-09-16T05:26:55.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:55.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c09cad0 con 0x7fb52c12b470 2024-09-16T05:26:55.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:55.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb52c13dac0 con 0x7fb52c09b6c0 2024-09-16T05:26:55.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:55.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient(hunting): authenticate will time out at 2099.820969s 2024-09-16T05:26:55.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52affd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c12a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:55.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb52c12aab0 0x7fb52c12af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:55.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52affd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c12a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:37680/0 (socket says 172.21.15.114:37680) 2024-09-16T05:26:55.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52affd640 1 -- 172.21.15.114:0/1229650557 learned_addr learned my addr 172.21.15.114:0/1229650557 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:55.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52affd640 10 monclient(hunting): get_auth_request con 0x7fb52c09b6c0 auth_method 0 2024-09-16T05:26:55.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:55.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:55.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:55.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 10 monclient(hunting): get_auth_request con 0x7fb52c12aab0 auth_method 0 2024-09-16T05:26:55.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:55.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:55.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:55.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52affd640 1 -- 172.21.15.114:0/1229650557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 msgr2=0x7fb52c12a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:26:55.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52affd640 1 -- 172.21.15.114:0/1229650557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 msgr2=0x7fb52c12a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:26:55.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52affd640 1 --2- 172.21.15.114:0/1229650557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c12a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T05:26:55.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:55.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:55.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:55.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52affd640 1 --2- 172.21.15.114:0/1229650557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c12a570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T05:26:55.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52b7fe640 1 --2- 172.21.15.114:0/1229650557 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb52c12b470 0x7fb52c1d5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:55.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52b7fe640 10 monclient(hunting): get_auth_request con 0x7fb52c12b470 auth_method 0 2024-09-16T05:26:55.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:55.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:55.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:55.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:55.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:55.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:55.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 10 monclient(hunting): handle_auth_done global_id 134106 payload 824 2024-09-16T05:26:55.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 1 -- 172.21.15.114:0/1229650557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 msgr2=0x7fb52c12a570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:26:55.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 1 --2- 172.21.15.114:0/1229650557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c12a570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:55.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 1 -- 172.21.15.114:0/1229650557 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb52c12b470 msgr2=0x7fb52c1d5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:55.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 1 --2- 172.21.15.114:0/1229650557 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb52c12b470 0x7fb52c1d5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:55.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:26:55.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 1 monclient: found mon.noname-a 2024-09-16T05:26:55.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:55.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:26:55.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 1 -- 172.21.15.114:0/1229650557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52c12d530 con 0x7fb52c12aab0 2024-09-16T05:26:55.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb52a7fc640 1 --2- 172.21.15.114:0/1229650557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb52c12aab0 0x7fb52c12af30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb51400f850 tx=0x7fb51400fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:55.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 10 client.?.objecter ms_handle_connect 0x7fb52c12aab0 2024-09-16T05:26:55.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:26:55.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 1 -- 172.21.15.114:0/1229650557 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5140052e0 con 0x7fb52c12aab0 2024-09-16T05:26:55.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:55.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:26:55.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 10 monclient: dump: 2024-09-16T05:26:55.197 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:55.197 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:55.197 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:55.197 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:55.197 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:55.197 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:55.197 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:55.197 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:55.197 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:55.198 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:55.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 10 monclient: _finish_auth 0 2024-09-16T05:26:55.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 10 monclient: _check_auth_tickets 2024-09-16T05:26:55.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:55.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 1 -- 172.21.15.114:0/1229650557 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb514018070 con 0x7fb52c12aab0 2024-09-16T05:26:55.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 5 monclient: authenticate success, global_id 134106 2024-09-16T05:26:55.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:55.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient: _renew_subs 2024-09-16T05:26:55.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 1 -- 172.21.15.114:0/1229650557 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51400d970 con 0x7fb52c12aab0 2024-09-16T05:26:55.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:55.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 1 -- 172.21.15.114:0/1229650557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52c098140 con 0x7fb52c12aab0 2024-09-16T05:26:55.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:55.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:26:55.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 10 monclient: dump: 2024-09-16T05:26:55.199 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:55.199 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:55.199 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:55.199 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:55.199 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:55.199 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:55.199 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:55.199 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:55.199 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:55.199 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:55.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5297fa640 4 set_mon_vals no callback set 2024-09-16T05:26:55.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 client.134106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:26:55.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient: _renew_subs 2024-09-16T05:26:55.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:55.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 1 -- 172.21.15.114:0/1229650557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb52c12cc20 con 0x7fb52c12aab0 2024-09-16T05:26:55.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 1 librados: init done 2024-09-16T05:26:55.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:55.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:55.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:55.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5315df640 1 -- 172.21.15.114:0/1229650557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb52c137630 con 0x7fb52c12aab0 2024-09-16T05:26:55.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 1 -- 172.21.15.114:0/1229650557 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb51400db10 con 0x7fb52c12aab0 2024-09-16T05:26:55.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 1 --2- 172.21.15.114:0/1229650557 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb50c037810 0x7fb50c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:55.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 1 -- 172.21.15.114:0/1229650557 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7fb514048c00 con 0x7fb52c12aab0 2024-09-16T05:26:55.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 10 client.134106.objecter ms_dispatch 0x7fb52c000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:26:55.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 3 client.134106.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:26:55.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 3 client.134106.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:26:55.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.191+0000 7fb5237fe640 20 client.134106.objecter dump_active .. 0 homeless 2024-09-16T05:26:55.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.195+0000 7fb52affd640 1 --2- 172.21.15.114:0/1229650557 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb50c037810 0x7fb50c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:55.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.195+0000 7fb52affd640 10 monclient: get_auth_request con 0x7fb50c037810 auth_method 0 2024-09-16T05:26:55.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.195+0000 7fb52affd640 1 --2- 172.21.15.114:0/1229650557 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb50c037810 0x7fb50c039cd0 secure :-1 s=READY pgs=3622 cs=0 l=1 rev1=1 crypto rx=0x7fb5240049e0 tx=0x7fb52400b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:55.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.195+0000 7fb5237fe640 10 client.134106.objecter ms_handle_connect 0x7fb50c037810 2024-09-16T05:26:55.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.199+0000 7fb5237fe640 1 -- 172.21.15.114:0/1229650557 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb51401c3c0 con 0x7fb52c12aab0 2024-09-16T05:26:55.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.199+0000 7fb5237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:55.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.199+0000 7fb5237fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:26:55.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.431+0000 7fb5315df640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:26:55.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.431+0000 7fb5315df640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:26:55.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.431+0000 7fb5315df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:55.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.431+0000 7fb5315df640 1 -- 172.21.15.114:0/1229650557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb52c12d270 con 0x7fb52c12aab0 2024-09-16T05:26:55.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.431+0000 7fb5237fe640 1 -- 172.21.15.114:0/1229650557 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fb514003b20 con 0x7fb52c12aab0 2024-09-16T05:26:55.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.431+0000 7fb5237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:26:55.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.431+0000 7fb5237fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:26:55.429 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":80,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:26:55.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.431+0000 7fb5315df640 10 librados: watch_flush enter 2024-09-16T05:26:55.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.431+0000 7fb5315df640 10 librados: watch_flush exit 2024-09-16T05:26:55.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.431+0000 7fb5315df640 20 client.134106.objecter shutdown clearing up homeless session... 2024-09-16T05:26:55.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.431+0000 7fb5315df640 10 client.134106.objecter successfully canceled tick 2024-09-16T05:26:55.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.431+0000 7fb5315df640 1 -- 172.21.15.114:0/1229650557 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb50c037810 msgr2=0x7fb50c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:55.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.435+0000 7fb5315df640 1 --2- 172.21.15.114:0/1229650557 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb50c037810 0x7fb50c039cd0 secure :-1 s=READY pgs=3622 cs=0 l=1 rev1=1 crypto rx=0x7fb5240049e0 tx=0x7fb52400b040 comp rx=0 tx=0).stop 2024-09-16T05:26:55.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.435+0000 7fb5315df640 10 monclient: shutdown 2024-09-16T05:26:55.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.435+0000 7fb5315df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:55.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.435+0000 7fb5315df640 1 -- 172.21.15.114:0/1229650557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb52c12aab0 msgr2=0x7fb52c12af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:55.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.435+0000 7fb5315df640 1 --2- 172.21.15.114:0/1229650557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb52c12aab0 0x7fb52c12af30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb51400f850 tx=0x7fb51400fd10 comp rx=0 tx=0).stop 2024-09-16T05:26:55.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.435+0000 7fb5315df640 1 -- 172.21.15.114:0/1229650557 shutdown_connections 2024-09-16T05:26:55.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.435+0000 7fb5315df640 1 --2- 172.21.15.114:0/1229650557 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb50c037810 0x7fb50c039cd0 unknown :-1 s=CLOSED pgs=3622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:55.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.435+0000 7fb5315df640 1 --2- 172.21.15.114:0/1229650557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb52c12aab0 0x7fb52c12af30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:55.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.435+0000 7fb5315df640 1 --2- 172.21.15.114:0/1229650557 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb52c12b470 0x7fb52c1d5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:55.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.435+0000 7fb5315df640 1 --2- 172.21.15.114:0/1229650557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb52c09b6c0 0x7fb52c12a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:55.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.435+0000 7fb5315df640 1 -- 172.21.15.114:0/1229650557 >> 172.21.15.114:0/1229650557 conn(0x7fb52c0a1fb0 msgr2=0x7fb52c134a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:55.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.435+0000 7fb5315df640 1 -- 172.21.15.114:0/1229650557 shutdown_connections 2024-09-16T05:26:55.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.435+0000 7fb5315df640 1 -- 172.21.15.114:0/1229650557 wait complete. 2024-09-16T05:26:55.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:55.435+0000 7fb5315df640 1 librados: shutdown 2024-09-16T05:26:58.448 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:26:58.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 10 monclient: get_monmap_and_config 2024-09-16T05:26:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 10 monclient: build_initial_monmap 2024-09-16T05:26:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 10 monclient: monmap: 2024-09-16T05:26:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:26:58.585192+0000 2024-09-16T05:26:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:26:58.585192+0000 2024-09-16T05:26:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:26:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:26:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:26:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:26:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:26:58.581 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:58.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 1 Processor -- start 2024-09-16T05:26:58.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 1 -- start start 2024-09-16T05:26:58.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 10 monclient: init 2024-09-16T05:26:58.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:58.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-16T05:26:58.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f744c09b6c0 0x7f744c0a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:58.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 10 monclient(hunting): picked mon.noname-a con 0x7f744c09b6c0 addr v2:172.21.15.114:3300/0 2024-09-16T05:26:58.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f744c0a1400 addr v1:172.21.15.114:6789/0 2024-09-16T05:26:58.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f744c0a1810 0x7f744c13cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:58.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 10 monclient(hunting): picked mon.noname-b con 0x7f744c0a1810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:58.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f744c09d4e0 con 0x7f744c0a1400 2024-09-16T05:26:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f744c09e390 con 0x7f744c09b6c0 2024-09-16T05:26:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f744c09ece0 con 0x7f744c0a1810 2024-09-16T05:26:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7453a3b640 10 monclient(hunting): authenticate will time out at 2103.214541s 2024-09-16T05:26:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f74517d7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f744c09b6c0 0x7f744c0a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f744c0a1810 0x7f744c13cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f74517d7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f744c09b6c0 0x7f744c0a0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34986/0 (socket says 172.21.15.114:34986) 2024-09-16T05:26:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f744c0a1810 0x7f744c13cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51598/0 (socket says 172.21.15.114:51598) 2024-09-16T05:26:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f74517d7640 1 -- 172.21.15.114:0/4238689691 learned_addr learned my addr 172.21.15.114:0/4238689691 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f74517d7640 10 monclient(hunting): get_auth_request con 0x7f744c09b6c0 auth_method 0 2024-09-16T05:26:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f74517d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f74517d7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f74517d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 10 monclient(hunting): get_auth_request con 0x7f744c0a1810 auth_method 0 2024-09-16T05:26:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f74517d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f74517d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f74517d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:58.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 10 monclient(hunting): handle_auth_done global_id 104363 payload 274 2024-09-16T05:26:58.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 1 -- 172.21.15.114:0/4238689691 >> v2:172.21.15.114:3300/0 conn(0x7f744c09b6c0 msgr2=0x7f744c0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:26:58.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 1 --2- 172.21.15.114:0/4238689691 >> v2:172.21.15.114:3300/0 conn(0x7f744c09b6c0 0x7f744c0a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:58.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 1 -- 172.21.15.114:0/4238689691 >> v1:172.21.15.114:6789/0 conn(0x7f744c0a1400 legacy=0x7f744c139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:58.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 10 monclient: _finish_hunting 0 2024-09-16T05:26:58.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 1 monclient: found mon.noname-b 2024-09-16T05:26:58.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:58.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:26:58.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.583+0000 7f7451fd8640 1 -- 172.21.15.114:0/4238689691 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744c09f430 con 0x7f744c0a1810 2024-09-16T05:26:58.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7451fd8640 1 --2- 172.21.15.114:0/4238689691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f744c0a1810 0x7f744c13cd70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f743c0097c0 tx=0x7f743c01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fff8942cb1b1b4b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:58.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f744bfff640 1 -- 172.21.15.114:0/4238689691 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c0b0070 con 0x7f744c0a1810 2024-09-16T05:26:58.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f744bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:58.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f744bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:26:58.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f744bfff640 10 monclient: dump: 2024-09-16T05:26:58.592 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:58.593 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:58.593 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.593 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.593 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:58.593 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:58.593 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:58.594 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:58.594 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:58.594 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:58.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f744bfff640 10 monclient: _finish_auth 0 2024-09-16T05:26:58.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f744bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:26:58.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f744bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:58.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 5 monclient: authenticate success, global_id 104363 2024-09-16T05:26:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f744bfff640 1 -- 172.21.15.114:0/4238689691 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f743c0ab600 con 0x7f744c0a1810 2024-09-16T05:26:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:26:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f744bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 10 monclient: get_monmap_and_config success 2024-09-16T05:26:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 4 set_mon_vals no callback set 2024-09-16T05:26:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f744bfff640 1 -- 172.21.15.114:0/4238689691 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c0ab9d0 con 0x7f744c0a1810 2024-09-16T05:26:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f744bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f744bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f744bfff640 10 monclient: dump: 2024-09-16T05:26:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:58.596 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 10 monclient: shutdown 2024-09-16T05:26:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 -- 172.21.15.114:0/4238689691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f744c0a1810 msgr2=0x7f744c13cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 --2- 172.21.15.114:0/4238689691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f744c0a1810 0x7f744c13cd70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f743c0097c0 tx=0x7f743c01c360 comp rx=0 tx=0).stop 2024-09-16T05:26:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 -- 172.21.15.114:0/4238689691 shutdown_connections 2024-09-16T05:26:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 --2- 172.21.15.114:0/4238689691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f744c0a1810 0x7f744c13cd70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 --2- 172.21.15.114:0/4238689691 >> v2:172.21.15.114:3300/0 conn(0x7f744c09b6c0 0x7f744c0a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 -- 172.21.15.114:0/4238689691 >> 172.21.15.114:0/4238689691 conn(0x7f744c0a1fb0 msgr2=0x7f744c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 -- 172.21.15.114:0/4238689691 shutdown_connections 2024-09-16T05:26:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 -- 172.21.15.114:0/4238689691 wait complete. 2024-09-16T05:26:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 10 monclient: build_initial_monmap 2024-09-16T05:26:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 10 monclient: monmap: 2024-09-16T05:26:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:26:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:26:58.598 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:26:58.598 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:58.598 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:58.598 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:26:58.598 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:26:58.598 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:26:58.598 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:58.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 librados: starting msgr at 2024-09-16T05:26:58.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 librados: starting objecter 2024-09-16T05:26:58.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 Processor -- start 2024-09-16T05:26:58.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 -- start start 2024-09-16T05:26:58.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 librados: setting wanted keys 2024-09-16T05:26:58.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 librados: calling monclient init 2024-09-16T05:26:58.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 10 monclient: init 2024-09-16T05:26:58.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:58.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:26:58.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f744c09b6c0 0x7f744c13cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:58.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 10 monclient(hunting): picked mon.noname-b con 0x7f744c09b6c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:26:58.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f744c0a1400 0x7f744c091e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:58.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 10 monclient(hunting): picked mon.noname-c con 0x7f744c0a1400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:58.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744c0a1810 0x7f744c092360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:58.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 10 monclient(hunting): picked mon.noname-a con 0x7f744c0a1810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:26:58.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:58.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f744c09cad0 con 0x7f744c0a1810 2024-09-16T05:26:58.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:58.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f744c09dd40 con 0x7f744c09b6c0 2024-09-16T05:26:58.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:58.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f744c13fad0 con 0x7f744c0a1400 2024-09-16T05:26:58.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:58.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7453a3b640 10 monclient(hunting): authenticate will time out at 2103.219615s 2024-09-16T05:26:58.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7451fd8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744c0a1810 0x7f744c092360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:58.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7450fd6640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f744c0a1400 0x7f744c091e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:58.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7451fd8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744c0a1810 0x7f744c092360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34994/0 (socket says 172.21.15.114:34994) 2024-09-16T05:26:58.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7450fd6640 10 monclient(hunting): get_auth_request con 0x7f744c0a1400 auth_method 0 2024-09-16T05:26:58.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7451fd8640 1 -- 172.21.15.114:0/694087080 learned_addr learned my addr 172.21.15.114:0/694087080 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:58.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7450fd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:58.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7450fd6640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:58.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7450fd6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:58.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7451fd8640 10 monclient(hunting): get_auth_request con 0x7f744c0a1810 auth_method 0 2024-09-16T05:26:58.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7451fd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:58.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7451fd8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:58.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7451fd8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:58.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7450fd6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:58.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7450fd6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:58.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7450fd6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:58.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7451fd8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:58.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7451fd8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:58.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f7451fd8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:58.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f74517d7640 1 --2- 172.21.15.114:0/694087080 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f744c09b6c0 0x7f744c13cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:58.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.587+0000 7f74517d7640 10 monclient(hunting): get_auth_request con 0x7f744c09b6c0 auth_method 0 2024-09-16T05:26:58.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f74517d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:58.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f74517d7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:58.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f74517d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:58.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7450fd6640 10 monclient(hunting): handle_auth_done global_id 104366 payload 824 2024-09-16T05:26:58.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7450fd6640 1 -- 172.21.15.114:0/694087080 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f744c09b6c0 msgr2=0x7f744c13cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:58.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7450fd6640 1 --2- 172.21.15.114:0/694087080 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f744c09b6c0 0x7f744c13cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:58.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7450fd6640 1 -- 172.21.15.114:0/694087080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744c0a1810 msgr2=0x7f744c092360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:58.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7450fd6640 1 --2- 172.21.15.114:0/694087080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744c0a1810 0x7f744c092360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:58.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7450fd6640 10 monclient: _finish_hunting 0 2024-09-16T05:26:58.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7450fd6640 1 monclient: found mon.noname-c 2024-09-16T05:26:58.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7450fd6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:58.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7450fd6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:26:58.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7450fd6640 1 -- 172.21.15.114:0/694087080 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744c0943a0 con 0x7f744c0a1400 2024-09-16T05:26:58.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7451fd8640 1 --2- 172.21.15.114:0/694087080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744c0a1810 0x7f744c092360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:26:58.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7450fd6640 1 --2- 172.21.15.114:0/694087080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f744c0a1400 0x7f744c091e20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7440002240 tx=0x7f744000cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:58.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 10 client.?.objecter ms_handle_connect 0x7f744c0a1400 2024-09-16T05:26:58.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:26:58.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 1 -- 172.21.15.114:0/694087080 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7440017070 con 0x7f744c0a1400 2024-09-16T05:26:58.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:58.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:26:58.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 10 monclient: dump: 2024-09-16T05:26:58.606 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:58.606 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:58.606 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.606 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.607 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:58.607 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:58.607 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:58.607 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:58.607 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:58.607 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:58.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 10 monclient: _finish_auth 0 2024-09-16T05:26:58.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:26:58.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:58.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 1 -- 172.21.15.114:0/694087080 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7440003ba0 con 0x7f744c0a1400 2024-09-16T05:26:58.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:58.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 1 -- 172.21.15.114:0/694087080 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7440005070 con 0x7f744c0a1400 2024-09-16T05:26:58.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7453a3b640 5 monclient: authenticate success, global_id 104366 2024-09-16T05:26:58.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7453a3b640 10 monclient: _renew_subs 2024-09-16T05:26:58.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7453a3b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:58.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7453a3b640 1 -- 172.21.15.114:0/694087080 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f744c095ad0 con 0x7f744c0a1400 2024-09-16T05:26:58.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f744b7fe640 4 set_mon_vals no callback set 2024-09-16T05:26:58.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:58.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:58.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 10 monclient: dump: 2024-09-16T05:26:58.608 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:58.608 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:58.608 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.608 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.609 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:58.609 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:58.609 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:58.609 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:58.609 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:58.609 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:58.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7453a3b640 10 client.104366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:26:58.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7453a3b640 10 monclient: _renew_subs 2024-09-16T05:26:58.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7453a3b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:58.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7453a3b640 1 -- 172.21.15.114:0/694087080 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f744c06e220 con 0x7f744c0a1400 2024-09-16T05:26:58.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7453a3b640 1 librados: init done 2024-09-16T05:26:58.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 1 -- 172.21.15.114:0/694087080 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f744001a070 con 0x7f744c0a1400 2024-09-16T05:26:58.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7453a3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:58.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7453a3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:58.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7453a3b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:58.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7453a3b640 1 -- 172.21.15.114:0/694087080 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f744c06dce0 con 0x7f744c0a1400 2024-09-16T05:26:58.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 1 --2- 172.21.15.114:0/694087080 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7438037810 0x7f7438039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:58.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 1 -- 172.21.15.114:0/694087080 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7f7440048240 con 0x7f744c0a1400 2024-09-16T05:26:58.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 10 client.104366.objecter ms_dispatch 0x7f744c000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:26:58.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 3 client.104366.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:26:58.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 3 client.104366.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:26:58.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 20 client.104366.objecter dump_active .. 0 homeless 2024-09-16T05:26:58.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f74517d7640 1 --2- 172.21.15.114:0/694087080 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7438037810 0x7f7438039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:58.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f74517d7640 10 monclient: get_auth_request con 0x7f7438037810 auth_method 0 2024-09-16T05:26:58.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f7449ffb640 10 client.104366.objecter ms_handle_connect 0x7f7438037810 2024-09-16T05:26:58.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.591+0000 7f74517d7640 1 --2- 172.21.15.114:0/694087080 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7438037810 0x7f7438039cd0 secure :-1 s=READY pgs=3623 cs=0 l=1 rev1=1 crypto rx=0x7f7444009a40 tx=0x7f7444008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:58.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.595+0000 7f7449ffb640 1 -- 172.21.15.114:0/694087080 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f744001a520 con 0x7f744c0a1400 2024-09-16T05:26:58.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.595+0000 7f7449ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:58.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.595+0000 7f7449ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:26:58.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.819+0000 7f7453a3b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:26:58.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.819+0000 7f7453a3b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:26:58.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.819+0000 7f7453a3b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:26:58.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.819+0000 7f7453a3b640 1 -- 172.21.15.114:0/694087080 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f744c0008e0 con 0x7f744c0a1400 2024-09-16T05:26:58.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.819+0000 7f7449ffb640 1 -- 172.21.15.114:0/694087080 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f744000a640 con 0x7f744c0a1400 2024-09-16T05:26:58.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.819+0000 7f7449ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:26:58.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.819+0000 7f7449ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:26:58.818 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":84,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:26:58.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 10 librados: watch_flush enter 2024-09-16T05:26:58.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 10 librados: watch_flush exit 2024-09-16T05:26:58.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 20 client.104366.objecter shutdown clearing up homeless session... 2024-09-16T05:26:58.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 10 client.104366.objecter successfully canceled tick 2024-09-16T05:26:58.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 1 -- 172.21.15.114:0/694087080 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7438037810 msgr2=0x7f7438039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:58.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 1 --2- 172.21.15.114:0/694087080 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7438037810 0x7f7438039cd0 secure :-1 s=READY pgs=3623 cs=0 l=1 rev1=1 crypto rx=0x7f7444009a40 tx=0x7f7444008040 comp rx=0 tx=0).stop 2024-09-16T05:26:58.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 10 monclient: shutdown 2024-09-16T05:26:58.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:58.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 1 -- 172.21.15.114:0/694087080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f744c0a1400 msgr2=0x7f744c091e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:58.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 1 --2- 172.21.15.114:0/694087080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f744c0a1400 0x7f744c091e20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7440002240 tx=0x7f744000cb90 comp rx=0 tx=0).stop 2024-09-16T05:26:58.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 1 -- 172.21.15.114:0/694087080 shutdown_connections 2024-09-16T05:26:58.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 1 --2- 172.21.15.114:0/694087080 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7438037810 0x7f7438039cd0 unknown :-1 s=CLOSED pgs=3623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:58.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 1 --2- 172.21.15.114:0/694087080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744c0a1810 0x7f744c092360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:58.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 1 --2- 172.21.15.114:0/694087080 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f744c0a1400 0x7f744c091e20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:58.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 1 --2- 172.21.15.114:0/694087080 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f744c09b6c0 0x7f744c13cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:58.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 1 -- 172.21.15.114:0/694087080 >> 172.21.15.114:0/694087080 conn(0x7f744c0a1fb0 msgr2=0x7f744c0713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:58.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 1 -- 172.21.15.114:0/694087080 shutdown_connections 2024-09-16T05:26:58.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 1 -- 172.21.15.114:0/694087080 wait complete. 2024-09-16T05:26:58.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.823+0000 7f7453a3b640 1 librados: shutdown 2024-09-16T05:26:58.836 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:26:58.836 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:26:58.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:26:58.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 10 monclient: get_monmap_and_config 2024-09-16T05:26:58.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 10 monclient: build_initial_monmap 2024-09-16T05:26:58.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 10 monclient: monmap: 2024-09-16T05:26:58.959 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:58.959 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:58.959 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:26:58.964647+0000 2024-09-16T05:26:58.959 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:26:58.964647+0000 2024-09-16T05:26:58.960 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:58.960 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:58.960 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:26:58.960 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:26:58.960 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:26:58.960 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:26:58.960 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:26:58.960 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:58.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 1 Processor -- start 2024-09-16T05:26:58.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 1 -- start start 2024-09-16T05:26:58.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 10 monclient: init 2024-09-16T05:26:58.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:58.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-16T05:26:58.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ecc09b6c0 0x7f1ecc0a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:58.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 10 monclient(hunting): picked mon.noname-b con 0x7f1ecc09b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:58.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ecc0a1400 addr v1:172.21.15.114:6789/0 2024-09-16T05:26:58.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ecc0a17e0 addr v1:172.21.15.149:6789/0 2024-09-16T05:26:58.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ecc09d4e0 con 0x7f1ecc0a1400 2024-09-16T05:26:58.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ecc09f430 con 0x7f1ecc0a17e0 2024-09-16T05:26:58.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:58.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1ecc09e390 con 0x7f1ecc09b6c0 2024-09-16T05:26:58.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:58.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed39c2640 10 monclient(hunting): authenticate will time out at 2103.593791s 2024-09-16T05:26:58.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed0f5d640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f1ecc0a1400 0x7f1ecc139640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37566/0 (socket says 172.21.15.114:37566) 2024-09-16T05:26:58.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ecc09b6c0 0x7f1ecc0a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:58.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed0f5d640 1 -- 172.21.15.114:0/247865215 learned_addr learned my addr 172.21.15.114:0/247865215 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:26:58.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 10 monclient(hunting): get_auth_request con 0x7f1ecc09b6c0 auth_method 0 2024-09-16T05:26:58.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:58.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 10 monclient(hunting): handle_auth_done global_id 104369 payload 274 2024-09-16T05:26:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ec3fff640 1 -- 172.21.15.114:0/247865215 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 964867133 0 0) 0x7f1ecc09d4e0 con 0x7f1ecc0a1400 2024-09-16T05:26:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 1 -- 172.21.15.114:0/247865215 >> v1:172.21.15.149:6789/0 conn(0x7f1ecc0a17e0 legacy=0x7f1ecc13cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 1 -- 172.21.15.114:0/247865215 >> v1:172.21.15.114:6789/0 conn(0x7f1ecc0a1400 legacy=0x7f1ecc139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 10 monclient: _finish_hunting 0 2024-09-16T05:26:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 1 monclient: found mon.noname-b 2024-09-16T05:26:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:26:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ed175e640 1 -- 172.21.15.114:0/247865215 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ecc09c2f0 con 0x7f1ecc09b6c0 2024-09-16T05:26:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.963+0000 7f1ec3fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:26:58.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed175e640 1 --2- 172.21.15.114:0/247865215 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ecc09b6c0 0x7f1ecc0a0ec0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0097f0 tx=0x7f1ebc01c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=769392604a917345 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:58.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ec3fff640 1 -- 172.21.15.114:0/247865215 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc0b0070 con 0x7f1ecc09b6c0 2024-09-16T05:26:58.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ec3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:58.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ec3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:26:58.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ec3fff640 10 monclient: dump: 2024-09-16T05:26:58.971 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:58.972 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ec3fff640 10 monclient: _finish_auth 0 2024-09-16T05:26:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ec3fff640 10 monclient: _check_auth_tickets 2024-09-16T05:26:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ec3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 5 monclient: authenticate success, global_id 104369 2024-09-16T05:26:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:26:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ec3fff640 1 -- 172.21.15.114:0/247865215 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ebc0ab600 con 0x7f1ecc09b6c0 2024-09-16T05:26:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ec3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 10 monclient: get_monmap_and_config success 2024-09-16T05:26:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ec3fff640 1 -- 172.21.15.114:0/247865215 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc0ab9d0 con 0x7f1ecc09b6c0 2024-09-16T05:26:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 4 set_mon_vals no callback set 2024-09-16T05:26:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ec3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ec3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ec3fff640 10 monclient: dump: 2024-09-16T05:26:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:58.974 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 10 monclient: shutdown 2024-09-16T05:26:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 1 -- 172.21.15.114:0/247865215 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ecc09b6c0 msgr2=0x7f1ecc0a0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 1 --2- 172.21.15.114:0/247865215 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ecc09b6c0 0x7f1ecc0a0ec0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0097f0 tx=0x7f1ebc01c3a0 comp rx=0 tx=0).stop 2024-09-16T05:26:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 1 -- 172.21.15.114:0/247865215 shutdown_connections 2024-09-16T05:26:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 1 --2- 172.21.15.114:0/247865215 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ecc09b6c0 0x7f1ecc0a0ec0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 1 -- 172.21.15.114:0/247865215 >> 172.21.15.114:0/247865215 conn(0x7f1ecc0a1fb0 msgr2=0x7f1ecc0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 1 -- 172.21.15.114:0/247865215 shutdown_connections 2024-09-16T05:26:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 1 -- 172.21.15.114:0/247865215 wait complete. 2024-09-16T05:26:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 10 monclient: build_initial_monmap 2024-09-16T05:26:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 10 monclient: monmap: 2024-09-16T05:26:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:26:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:26:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:26:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:26:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:26:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:26:58.975 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 1 librados: starting msgr at 2024-09-16T05:26:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 1 librados: starting objecter 2024-09-16T05:26:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.967+0000 7f1ed39c2640 1 Processor -- start 2024-09-16T05:26:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 1 -- start start 2024-09-16T05:26:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 1 librados: setting wanted keys 2024-09-16T05:26:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 1 librados: calling monclient init 2024-09-16T05:26:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient: init 2024-09-16T05:26:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:26:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc09b6c0 0x7f1ecc098190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient(hunting): picked mon.noname-a con 0x7f1ecc09b6c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:26:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1ecc0a1400 0x7f1ecc091ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient(hunting): picked mon.noname-b con 0x7f1ecc0a1400 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:26:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ecc0a17e0 0x7f1ecc092020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient(hunting): picked mon.noname-c con 0x7f1ecc0a17e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1ecc140cb0 con 0x7f1ecc09b6c0 2024-09-16T05:26:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f1ecc0a1bf0 con 0x7f1ecc0a1400 2024-09-16T05:26:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1ecc140e30 con 0x7f1ecc0a17e0 2024-09-16T05:26:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient(hunting): authenticate will time out at 2103.600531s 2024-09-16T05:26:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc09b6c0 0x7f1ecc098190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc09b6c0 0x7f1ecc098190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35002/0 (socket says 172.21.15.114:35002) 2024-09-16T05:26:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 1 -- 172.21.15.114:0/230605925 learned_addr learned my addr 172.21.15.114:0/230605925 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 10 monclient(hunting): get_auth_request con 0x7f1ecc09b6c0 auth_method 0 2024-09-16T05:26:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed1f5f640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ecc0a17e0 0x7f1ecc092020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed1f5f640 10 monclient(hunting): get_auth_request con 0x7f1ecc0a17e0 auth_method 0 2024-09-16T05:26:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed1f5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed1f5f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed1f5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed1f5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed1f5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed1f5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 10 monclient(hunting): handle_auth_done global_id 134121 payload 824 2024-09-16T05:26:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed0f5d640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1ecc0a1400 0x7f1ecc091ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 1 -- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ecc0a17e0 msgr2=0x7f1ecc092020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ecc0a17e0 0x7f1ecc092020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 1 -- 172.21.15.114:0/230605925 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1ecc0a1400 msgr2=0x7f1ecc091ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1ecc0a1400 0x7f1ecc091ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 10 monclient: _finish_hunting 0 2024-09-16T05:26:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 1 monclient: found mon.noname-a 2024-09-16T05:26:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:26:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 1 -- 172.21.15.114:0/230605925 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ecc094290 con 0x7f1ecc09b6c0 2024-09-16T05:26:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed0f5d640 10 monclient: get_auth_request con 0x7f1ecc0a1400 auth_method 0 2024-09-16T05:26:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed0f5d640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1ecc0a1400 0x7f1ecc091ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:26:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed175e640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc09b6c0 0x7f1ecc098190 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0ae040 tx=0x7f1ebc01abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 client.?.objecter ms_handle_connect 0x7f1ecc09b6c0 2024-09-16T05:26:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:26:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 1 -- 172.21.15.114:0/230605925 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc0bc070 con 0x7f1ecc09b6c0 2024-09-16T05:26:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:26:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 monclient: dump: 2024-09-16T05:26:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:58.980 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 monclient: _finish_auth 0 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 1 -- 172.21.15.114:0/230605925 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ebc0b1040 con 0x7f1ecc09b6c0 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 5 monclient: authenticate success, global_id 134121 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient: _renew_subs 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 1 -- 172.21.15.114:0/230605925 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ecc095740 con 0x7f1ecc09b6c0 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 1 -- 172.21.15.114:0/230605925 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc0b0050 con 0x7f1ecc09b6c0 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 monclient: dump: 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.982 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.982 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:58.982 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:58.982 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:58.982 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:58.982 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:58.982 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:58.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec37fe640 4 set_mon_vals no callback set 2024-09-16T05:26:58.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 client.134121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:26:58.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient: _renew_subs 2024-09-16T05:26:58.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:58.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 1 -- 172.21.15.114:0/230605925 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ecc095180 con 0x7f1ecc09b6c0 2024-09-16T05:26:58.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 1 librados: init done 2024-09-16T05:26:58.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc0986d0 0x7f1ecc092560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed39c2640 1 -- 172.21.15.114:0/230605925 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1ecc0a0ec0 con 0x7f1ecc0986d0 2024-09-16T05:26:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 1 -- 172.21.15.114:0/230605925 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1ebc0bf070 con 0x7f1ecc09b6c0 2024-09-16T05:26:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed0f5d640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc0986d0 0x7f1ecc092560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed0f5d640 10 monclient: get_auth_request con 0x7f1ecc0986d0 auth_method 0 2024-09-16T05:26:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed0f5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed0f5d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed0f5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed0f5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed0f5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed0f5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 1 --2- 172.21.15.114:0/230605925 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1eb0037810 0x7f1eb0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 1 -- 172.21.15.114:0/230605925 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7f1ebc0e2890 con 0x7f1ecc09b6c0 2024-09-16T05:26:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 client.134121.objecter ms_dispatch 0x7f1ecc000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:26:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 3 client.134121.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:26:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 3 client.134121.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:26:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed0f5d640 10 monclient(hunting): handle_auth_done global_id 134124 payload 824 2024-09-16T05:26:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed0f5d640 1 -- 172.21.15.114:0/230605925 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ecc094010 con 0x7f1ecc0986d0 2024-09-16T05:26:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 20 client.134121.objecter dump_active .. 0 homeless 2024-09-16T05:26:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed0f5d640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc0986d0 0x7f1ecc092560 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc093330 tx=0x7f1ec800db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 client.134121.objecter ms_handle_connect 0x7f1ecc0986d0 2024-09-16T05:26:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 client.134121.objecter resend_mon_ops 2024-09-16T05:26:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed1f5f640 1 --2- 172.21.15.114:0/230605925 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1eb0037810 0x7f1eb0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed1f5f640 10 monclient: get_auth_request con 0x7f1eb0037810 auth_method 0 2024-09-16T05:26:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 1 -- 172.21.15.114:0/230605925 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8007270 con 0x7f1ecc0986d0 2024-09-16T05:26:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:26:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 monclient: dump: 2024-09-16T05:26:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:58.985 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 1 -- 172.21.15.114:0/230605925 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1ec8017070 con 0x7f1ecc0986d0 2024-09-16T05:26:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:58.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:26:58.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 1 -- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc0986d0 msgr2=0x7f1ecc092560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:58.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc0986d0 0x7f1ecc092560 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc093330 tx=0x7f1ec800db80 comp rx=0 tx=0).stop 2024-09-16T05:26:58.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 1 -- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc0986d0 msgr2=0x7f1ecc092560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:26:58.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc0986d0 0x7f1ecc092560 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc093330 tx=0x7f1ec800db80 comp rx=0 tx=0).stop 2024-09-16T05:26:58.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ec1ffb640 10 client.134121.objecter ms_handle_connect 0x7f1eb0037810 2024-09-16T05:26:58.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.971+0000 7f1ed1f5f640 1 --2- 172.21.15.114:0/230605925 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1eb0037810 0x7f1eb0039cd0 secure :-1 s=READY pgs=3624 cs=0 l=1 rev1=1 crypto rx=0x7f1ec40012f0 tx=0x7f1ec4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:58.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:26:58.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc08e010 0x7f1ecc08e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:58.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:58.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 1 -- 172.21.15.114:0/230605925 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1ecc0686f0 con 0x7f1ecc08e010 2024-09-16T05:26:58.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed0f5d640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc08e010 0x7f1ecc08e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:58.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed0f5d640 10 monclient: get_auth_request con 0x7f1ecc08e010 auth_method 0 2024-09-16T05:26:58.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed0f5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:58.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed0f5d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:58.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed0f5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:58.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed0f5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:58.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed0f5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:58.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed0f5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:58.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed0f5d640 10 monclient(hunting): handle_auth_done global_id 134127 payload 824 2024-09-16T05:26:58.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed0f5d640 1 -- 172.21.15.114:0/230605925 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1ecc095980 con 0x7f1ecc08e010 2024-09-16T05:26:58.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ec1ffb640 10 client.134121.objecter ms_handle_connect 0x7f1ecc08e010 2024-09-16T05:26:58.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ec1ffb640 10 client.134121.objecter resend_mon_ops 2024-09-16T05:26:58.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed0f5d640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc08e010 0x7f1ecc08e420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8016000 tx=0x7f1ec8000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:58.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ec1ffb640 1 -- 172.21.15.114:0/230605925 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec80057b0 con 0x7f1ecc08e010 2024-09-16T05:26:58.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ec1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:58.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ec1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:26:58.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ec1ffb640 10 monclient: dump: 2024-09-16T05:26:58.989 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:58.989 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:58.990 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.990 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:58.990 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:58.990 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:58.990 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:58.990 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:58.990 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:58.990 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:58.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ec1ffb640 1 -- 172.21.15.114:0/230605925 <== mon.0 v2:172.21.15.114:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f1ec800b6d0 con 0x7f1ecc08e010 2024-09-16T05:26:58.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ec1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:26:58.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ec1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:26:58.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ec1ffb640 1 -- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc08e010 msgr2=0x7f1ecc08e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:58.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ec1ffb640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc08e010 0x7f1ecc08e420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8016000 tx=0x7f1ec8000bd0 comp rx=0 tx=0).stop 2024-09-16T05:26:58.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ec1ffb640 1 -- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc08e010 msgr2=0x7f1ecc08e420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:26:58.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ec1ffb640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc08e010 0x7f1ecc08e420 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8016000 tx=0x7f1ec8000bd0 comp rx=0 tx=0).stop 2024-09-16T05:26:58.991 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:26:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:26:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:26:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "leader", 2024-09-16T05:26:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 84, 2024-09-16T05:26:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 27667, 2024-09-16T05:26:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:26:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:26:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:26:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:26:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:26:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:26:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:26:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:26:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:26:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:26:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:26:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:26:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:26:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:26:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:26:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:26:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:26:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:26:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:26:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:26:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:26:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:26:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:26:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:26:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:26:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:26:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:26:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:26:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:26:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:26:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:26:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:26:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:26:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:26:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:26:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:26:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:26:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:26:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:26:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:26:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:26:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:26:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:26:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:26:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:26:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:26:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:26:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:26:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:26:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:26:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:26:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:26:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:26:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:26:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:26:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:26:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:26:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:26:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:26:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:26:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:26:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:26:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:26:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:26:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:26:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:26:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:26:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:26:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:26:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:26:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:26:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:26:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:26:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:26:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:26:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:26:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:26:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:26:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:26:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:26:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:26:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:26:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:26:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:26:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:26:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:26:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:26:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:26:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:26:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:26:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:26:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:26:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:26:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:26:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:26:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:26:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:26:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:26:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:26:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:26:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:26:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:26:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:26:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:26:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:26:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:26:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:26:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:26:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:26:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:26:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:26:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:26:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:26:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-09-16T05:26:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:26:59.003 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:26:59.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 10 librados: watch_flush enter 2024-09-16T05:26:59.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 10 librados: watch_flush exit 2024-09-16T05:26:59.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 20 client.134121.objecter shutdown clearing up homeless session... 2024-09-16T05:26:59.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 10 client.134121.objecter successfully canceled tick 2024-09-16T05:26:59.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 1 -- 172.21.15.114:0/230605925 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1eb0037810 msgr2=0x7f1eb0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 1 --2- 172.21.15.114:0/230605925 >> v2:172.21.15.149:6812/1185531135 conn(0x7f1eb0037810 0x7f1eb0039cd0 secure :-1 s=READY pgs=3624 cs=0 l=1 rev1=1 crypto rx=0x7f1ec40012f0 tx=0x7f1ec4009040 comp rx=0 tx=0).stop 2024-09-16T05:26:59.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 10 monclient: shutdown 2024-09-16T05:26:59.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:59.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 1 -- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc09b6c0 msgr2=0x7f1ecc098190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 1 --2- 172.21.15.114:0/230605925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ecc09b6c0 0x7f1ecc098190 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0ae040 tx=0x7f1ebc01abe0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed1f5f640 1 -- 172.21.15.114:0/230605925 reap_dead start 2024-09-16T05:26:59.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed1f5f640 1 -- 172.21.15.114:0/230605925 reap_dead start 2024-09-16T05:26:59.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 1 -- 172.21.15.114:0/230605925 shutdown_connections 2024-09-16T05:26:59.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 1 -- 172.21.15.114:0/230605925 >> 172.21.15.114:0/230605925 conn(0x7f1ecc0a1fb0 msgr2=0x7f1ecc071320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:59.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 1 -- 172.21.15.114:0/230605925 shutdown_connections 2024-09-16T05:26:59.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 1 -- 172.21.15.114:0/230605925 wait complete. 2024-09-16T05:26:59.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:58.979+0000 7f1ed39c2640 1 librados: shutdown 2024-09-16T05:26:59.010 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:26:59.010 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:26:59.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:26:59.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 10 monclient: get_monmap_and_config 2024-09-16T05:26:59.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 10 monclient: build_initial_monmap 2024-09-16T05:26:59.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 10 monclient: monmap: 2024-09-16T05:26:59.124 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:59.124 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.124 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:26:59.129320+0000 2024-09-16T05:26:59.124 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:26:59.129320+0000 2024-09-16T05:26:59.124 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:59.124 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.125 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:26:59.125 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:26:59.125 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:26:59.125 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:26:59.125 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:26:59.125 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 1 Processor -- start 2024-09-16T05:26:59.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 1 -- start start 2024-09-16T05:26:59.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 10 monclient: init 2024-09-16T05:26:59.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:59.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-16T05:26:59.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da80a0ec0 0x7f0da809dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 10 monclient(hunting): picked mon.noname-b con 0x7f0da80a0ec0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:59.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0da809e000 addr v1:172.21.15.114:6789/0 2024-09-16T05:26:59.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f0da8139640 0x7f0da813ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 10 monclient(hunting): picked mon.noname-c con 0x7f0da8139640 addr v2:172.21.15.149:3300/0 2024-09-16T05:26:59.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0da809a760 con 0x7f0da809e000 2024-09-16T05:26:59.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f0da80a12a0 con 0x7f0da8139640 2024-09-16T05:26:59.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f0da80994b0 con 0x7f0da80a0ec0 2024-09-16T05:26:59.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:59.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dae88e640 10 monclient(hunting): authenticate will time out at 2103.758978s 2024-09-16T05:26:59.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da80a0ec0 0x7f0da809dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da77fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0da809e000 0x7f0da809e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37570/0 (socket says 172.21.15.114:37570) 2024-09-16T05:26:59.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da80a0ec0 0x7f0da809dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51630/0 (socket says 172.21.15.114:51630) 2024-09-16T05:26:59.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da77fe640 1 -- 172.21.15.114:0/4185693082 learned_addr learned my addr 172.21.15.114:0/4185693082 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:26:59.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 10 monclient(hunting): get_auth_request con 0x7f0da80a0ec0 auth_method 0 2024-09-16T05:26:59.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:59.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:59.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:59.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dace2b640 1 --2- 172.21.15.114:0/4185693082 >> v2:172.21.15.149:3300/0 conn(0x7f0da8139640 0x7f0da813ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dace2b640 10 monclient(hunting): get_auth_request con 0x7f0da8139640 auth_method 0 2024-09-16T05:26:59.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dace2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dace2b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0dace2b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 10 monclient(hunting): handle_auth_done global_id 104375 payload 274 2024-09-16T05:26:59.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 1 -- 172.21.15.114:0/4185693082 >> v2:172.21.15.149:3300/0 conn(0x7f0da8139640 msgr2=0x7f0da813ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:26:59.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 1 --2- 172.21.15.114:0/4185693082 >> v2:172.21.15.149:3300/0 conn(0x7f0da8139640 0x7f0da813ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 1 -- 172.21.15.114:0/4185693082 >> v1:172.21.15.114:6789/0 conn(0x7f0da809e000 legacy=0x7f0da809e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da6ffd640 1 -- 172.21.15.114:0/4185693082 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4139313136 0 0) 0x7f0da809a760 con 0x7f0da809e000 2024-09-16T05:26:59.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:26:59.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 1 monclient: found mon.noname-b 2024-09-16T05:26:59.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:59.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:26:59.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da7fff640 1 -- 172.21.15.114:0/4185693082 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da80a1870 con 0x7f0da80a0ec0 2024-09-16T05:26:59.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.127+0000 7f0da6ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:26:59.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0da7fff640 1 --2- 172.21.15.114:0/4185693082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da80a0ec0 0x7f0da809dac0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0d980097c0 tx=0x7f0d9801c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c2e929c5a7ed22 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:59.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0da6ffd640 1 -- 172.21.15.114:0/4185693082 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d980b0070 con 0x7f0da80a0ec0 2024-09-16T05:26:59.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0da6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0da6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:26:59.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0da6ffd640 10 monclient: dump: 2024-09-16T05:26:59.138 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.138 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.139 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.139 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.139 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.139 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.139 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.139 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.139 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.139 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0da6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:26:59.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0da6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:26:59.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0da6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:59.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0da6ffd640 1 -- 172.21.15.114:0/4185693082 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d980ab600 con 0x7f0da80a0ec0 2024-09-16T05:26:59.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0da6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:59.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0da6ffd640 1 -- 172.21.15.114:0/4185693082 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d980ab9d0 con 0x7f0da80a0ec0 2024-09-16T05:26:59.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 5 monclient: authenticate success, global_id 104375 2024-09-16T05:26:59.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 10 monclient: get_monmap_and_config success 2024-09-16T05:26:59.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 4 set_mon_vals no callback set 2024-09-16T05:26:59.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0da6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0da6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:59.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0da6ffd640 10 monclient: dump: 2024-09-16T05:26:59.140 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.140 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.140 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.140 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.140 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.140 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.141 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.141 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.141 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.141 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 10 monclient: shutdown 2024-09-16T05:26:59.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:59.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 1 -- 172.21.15.114:0/4185693082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da80a0ec0 msgr2=0x7f0da809dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 1 --2- 172.21.15.114:0/4185693082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da80a0ec0 0x7f0da809dac0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0d980097c0 tx=0x7f0d9801c3f0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 1 -- 172.21.15.114:0/4185693082 shutdown_connections 2024-09-16T05:26:59.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 1 --2- 172.21.15.114:0/4185693082 >> v2:172.21.15.149:3300/0 conn(0x7f0da8139640 0x7f0da813ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 1 --2- 172.21.15.114:0/4185693082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da80a0ec0 0x7f0da809dac0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 1 -- 172.21.15.114:0/4185693082 >> 172.21.15.114:0/4185693082 conn(0x7f0da80a1fb0 msgr2=0x7f0da80a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:59.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 1 -- 172.21.15.114:0/4185693082 shutdown_connections 2024-09-16T05:26:59.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 1 -- 172.21.15.114:0/4185693082 wait complete. 2024-09-16T05:26:59.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 10 monclient: build_initial_monmap 2024-09-16T05:26:59.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 10 monclient: monmap: 2024-09-16T05:26:59.142 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:59.142 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:26:59.142 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:26:59.142 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:26:59.142 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:59.142 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.142 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:26:59.142 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:26:59.142 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:26:59.142 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 1 librados: starting msgr at 2024-09-16T05:26:59.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 1 librados: starting objecter 2024-09-16T05:26:59.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 1 Processor -- start 2024-09-16T05:26:59.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 1 -- start start 2024-09-16T05:26:59.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 1 librados: setting wanted keys 2024-09-16T05:26:59.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 1 librados: calling monclient init 2024-09-16T05:26:59.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.131+0000 7f0dae88e640 10 monclient: init 2024-09-16T05:26:59.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:59.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:26:59.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0da809e000 0x7f0da813e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 monclient(hunting): picked mon.noname-b con 0x7f0da809e000 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:26:59.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0da80a0ec0 0x7f0da813ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 monclient(hunting): picked mon.noname-a con 0x7f0da80a0ec0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:26:59.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da8139640 0x7f0da813f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 monclient(hunting): picked mon.noname-c con 0x7f0da8139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:59.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0da809cb30 con 0x7f0da80a0ec0 2024-09-16T05:26:59.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f0da809c9b0 con 0x7f0da809e000 2024-09-16T05:26:59.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f0da809ccb0 con 0x7f0da8139640 2024-09-16T05:26:59.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:59.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 monclient(hunting): authenticate will time out at 2103.764168s 2024-09-16T05:26:59.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0da80a0ec0 0x7f0da813ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dace2b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da8139640 0x7f0da813f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0da80a0ec0 0x7f0da813ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35026/0 (socket says 172.21.15.114:35026) 2024-09-16T05:26:59.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dace2b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da8139640 0x7f0da813f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51640/0 (socket says 172.21.15.114:51640) 2024-09-16T05:26:59.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 1 -- 172.21.15.114:0/1170151773 learned_addr learned my addr 172.21.15.114:0/1170151773 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:59.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 10 monclient(hunting): get_auth_request con 0x7f0da80a0ec0 auth_method 0 2024-09-16T05:26:59.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dace2b640 10 monclient(hunting): get_auth_request con 0x7f0da8139640 auth_method 0 2024-09-16T05:26:59.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dace2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dace2b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dace2b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dace2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:59.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dace2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:59.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dace2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:59.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:59.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:59.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:59.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0da809e000 0x7f0da813e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 10 monclient(hunting): get_auth_request con 0x7f0da809e000 auth_method 0 2024-09-16T05:26:59.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 10 monclient(hunting): handle_auth_done global_id 134133 payload 824 2024-09-16T05:26:59.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 1 -- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da8139640 msgr2=0x7f0da813f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da8139640 0x7f0da813f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 1 -- 172.21.15.114:0/1170151773 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0da809e000 msgr2=0x7f0da813e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0da809e000 0x7f0da813e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 10 monclient: _finish_hunting 0 2024-09-16T05:26:59.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 1 monclient: found mon.noname-a 2024-09-16T05:26:59.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:59.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:26:59.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 1 -- 172.21.15.114:0/1170151773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da812ab80 con 0x7f0da80a0ec0 2024-09-16T05:26:59.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dace2b640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da8139640 0x7f0da813f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:26:59.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0da809e000 0x7f0da813e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:26:59.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da77fe640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0da80a0ec0 0x7f0da813ecd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00c3e0 tx=0x7f0d9c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:59.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 client.?.objecter ms_handle_connect 0x7f0da80a0ec0 2024-09-16T05:26:59.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:26:59.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 1 -- 172.21.15.114:0/1170151773 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c0074e0 con 0x7f0da80a0ec0 2024-09-16T05:26:59.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:26:59.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 monclient: dump: 2024-09-16T05:26:59.148 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.148 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.148 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.148 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.148 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.148 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.148 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.148 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.149 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.149 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 monclient: _finish_auth 0 2024-09-16T05:26:59.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:26:59.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:59.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 5 monclient: authenticate success, global_id 134133 2024-09-16T05:26:59.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 1 -- 172.21.15.114:0/1170151773 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d9c0047f0 con 0x7f0da80a0ec0 2024-09-16T05:26:59.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 monclient: _renew_subs 2024-09-16T05:26:59.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:59.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 1 -- 172.21.15.114:0/1170151773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0da80736d0 con 0x7f0da80a0ec0 2024-09-16T05:26:59.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:59.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 1 -- 172.21.15.114:0/1170151773 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c0076a0 con 0x7f0da80a0ec0 2024-09-16T05:26:59.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:26:59.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 monclient: dump: 2024-09-16T05:26:59.150 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.150 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.150 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.150 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.150 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.150 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.150 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.150 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.150 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.150 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 client.134133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:26:59.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da67fc640 4 set_mon_vals no callback set 2024-09-16T05:26:59.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 monclient: _renew_subs 2024-09-16T05:26:59.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:59.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 1 -- 172.21.15.114:0/1170151773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0da812a570 con 0x7f0da80a0ec0 2024-09-16T05:26:59.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 1 librados: init done 2024-09-16T05:26:59.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:59.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da81cff10 0x7f0da802c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dae88e640 1 -- 172.21.15.114:0/1170151773 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f0da80a3d40 con 0x7f0da81cff10 2024-09-16T05:26:59.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 1 -- 172.21.15.114:0/1170151773 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0d9c011070 con 0x7f0da80a0ec0 2024-09-16T05:26:59.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da81cff10 0x7f0da802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 10 monclient: get_auth_request con 0x7f0da81cff10 auth_method 0 2024-09-16T05:26:59.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:59.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:59.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:59.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 1 --2- 172.21.15.114:0/1170151773 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0d94037840 0x7f0d94039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 10 monclient(hunting): handle_auth_done global_id 104381 payload 824 2024-09-16T05:26:59.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 1 -- 172.21.15.114:0/1170151773 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7f0d9c048640 con 0x7f0da80a0ec0 2024-09-16T05:26:59.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 client.134133.objecter ms_dispatch 0x7f0da8000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:26:59.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 1 -- 172.21.15.114:0/1170151773 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0da812a880 con 0x7f0da81cff10 2024-09-16T05:26:59.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 3 client.134133.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:26:59.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 3 client.134133.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:26:59.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 20 client.134133.objecter dump_active .. 0 homeless 2024-09-16T05:26:59.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 client.134133.objecter ms_handle_connect 0x7f0da81cff10 2024-09-16T05:26:59.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 client.134133.objecter resend_mon_ops 2024-09-16T05:26:59.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da7fff640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da81cff10 0x7f0da802c650 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0da806cf30 tx=0x7f0d9801ac30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:59.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dace2b640 1 --2- 172.21.15.114:0/1170151773 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0d94037840 0x7f0d94039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 1 -- 172.21.15.114:0/1170151773 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d980b0040 con 0x7f0da81cff10 2024-09-16T05:26:59.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:59.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 monclient: dump: 2024-09-16T05:26:59.153 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.153 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.153 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.153 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.153 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.153 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.153 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.153 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.154 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.154 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dace2b640 10 monclient: get_auth_request con 0x7f0d94037840 auth_method 0 2024-09-16T05:26:59.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 1 -- 172.21.15.114:0/1170151773 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0d980abdc0 con 0x7f0da81cff10 2024-09-16T05:26:59.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:59.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:26:59.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 1 -- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da81cff10 msgr2=0x7f0da802c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da81cff10 0x7f0da802c650 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0da806cf30 tx=0x7f0d9801ac30 comp rx=0 tx=0).stop 2024-09-16T05:26:59.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 1 -- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da81cff10 msgr2=0x7f0da802c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:26:59.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da81cff10 0x7f0da802c650 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0da806cf30 tx=0x7f0d9801ac30 comp rx=0 tx=0).stop 2024-09-16T05:26:59.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0da4ff9640 10 client.134133.objecter ms_handle_connect 0x7f0d94037840 2024-09-16T05:26:59.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.135+0000 7f0dace2b640 1 --2- 172.21.15.114:0/1170151773 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0d94037840 0x7f0d94039d00 secure :-1 s=READY pgs=3625 cs=0 l=1 rev1=1 crypto rx=0x7f0da00077c0 tx=0x7f0da0009820 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:59.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:26:59.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da81d02f0 0x7f0da802cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 1 -- 172.21.15.114:0/1170151773 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f0da8069390 con 0x7f0da81d02f0 2024-09-16T05:26:59.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da7fff640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da81d02f0 0x7f0da802cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da7fff640 10 monclient: get_auth_request con 0x7f0da81d02f0 auth_method 0 2024-09-16T05:26:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da7fff640 10 monclient(hunting): handle_auth_done global_id 104384 payload 824 2024-09-16T05:26:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da7fff640 1 -- 172.21.15.114:0/1170151773 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0da809e3e0 con 0x7f0da81d02f0 2024-09-16T05:26:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da7fff640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da81d02f0 0x7f0da802cb90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0d9801aaa0 tx=0x7f0d98019190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da4ff9640 10 client.134133.objecter ms_handle_connect 0x7f0da81d02f0 2024-09-16T05:26:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da4ff9640 10 client.134133.objecter resend_mon_ops 2024-09-16T05:26:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da4ff9640 1 -- 172.21.15.114:0/1170151773 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d980b0040 con 0x7f0da81d02f0 2024-09-16T05:26:59.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:26:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da4ff9640 10 monclient: dump: 2024-09-16T05:26:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.158 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da4ff9640 1 -- 172.21.15.114:0/1170151773 <== mon.2 v2:172.21.15.114:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f0d980b6e40 con 0x7f0da81d02f0 2024-09-16T05:26:59.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:26:59.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:26:59.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da4ff9640 1 -- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da81d02f0 msgr2=0x7f0da802cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da4ff9640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da81d02f0 0x7f0da802cb90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0d9801aaa0 tx=0x7f0d98019190 comp rx=0 tx=0).stop 2024-09-16T05:26:59.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da4ff9640 1 -- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da81d02f0 msgr2=0x7f0da802cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:26:59.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0da4ff9640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0da81d02f0 0x7f0da802cb90 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0d9801aaa0 tx=0x7f0d98019190 comp rx=0 tx=0).stop 2024-09-16T05:26:59.159 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:26:59.159 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:26:59.159 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:26:59.159 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:26:59.159 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 84, 2024-09-16T05:26:59.159 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 395307, 2024-09-16T05:26:59.160 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:26:59.160 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:26:59.160 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:26:59.160 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:26:59.160 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:26:59.160 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:26:59.160 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:26:59.160 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:26:59.160 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:26:59.160 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:26:59.160 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:26:59.160 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:26:59.160 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:26:59.161 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:26:59.161 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:26:59.161 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:26:59.161 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:26:59.161 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:26:59.161 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:26:59.161 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:26:59.161 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:26:59.161 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:26:59.161 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:26:59.161 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:26:59.161 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:26:59.161 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:26:59.161 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:26:59.162 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:26:59.162 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:26:59.162 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:26:59.162 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:26:59.162 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:26:59.162 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:26:59.162 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:26:59.162 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.162 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:26:59.162 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:26:59.162 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:26:59.163 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:26:59.163 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:26:59.163 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:26:59.163 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:26:59.163 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:26:59.163 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:26:59.163 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:26:59.163 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:26:59.163 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:26:59.163 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:26:59.163 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:26:59.163 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:26:59.163 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:26:59.163 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:26:59.164 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:26:59.164 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:26:59.164 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:26:59.164 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:26:59.164 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:26:59.164 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:26:59.164 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:26:59.164 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:26:59.164 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:26:59.164 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:26:59.164 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:26:59.165 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:26:59.165 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:26:59.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.165 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:26:59.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.165 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:26:59.165 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:26:59.165 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:26:59.165 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:26:59.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.165 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:26:59.165 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:26:59.165 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:59.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.166 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:26:59.166 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:26:59.166 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:59.166 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.166 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.166 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:26:59.166 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:26:59.166 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:26:59.166 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:26:59.166 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:26:59.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.167 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:26:59.167 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:26:59.167 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:26:59.167 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:26:59.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.167 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:26:59.167 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:26:59.167 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:59.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.167 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:26:59.167 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:26:59.167 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:59.167 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.168 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.168 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:26:59.168 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:26:59.168 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:26:59.168 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:26:59.168 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:26:59.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.168 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:26:59.168 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:26:59.168 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:26:59.168 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:26:59.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.169 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:26:59.169 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:26:59.169 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:59.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.169 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:26:59.169 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:26:59.169 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:59.169 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.169 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.169 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:26:59.169 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:26:59.170 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:26:59.170 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:26:59.170 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:26:59.170 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.170 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.170 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.170 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:26:59.170 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:26:59.170 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.170 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:26:59.170 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:26:59.170 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:26:59.170 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.170 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:26:59.170 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:26:59.171 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.171 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:26:59.171 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:26:59.171 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:26:59.171 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.171 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:26:59.171 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:26:59.171 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.171 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:26:59.171 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:26:59.171 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 5 2024-09-16T05:26:59.171 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.171 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:26:59.171 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": [ 2024-09-16T05:26:59.172 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.172 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:26:59.172 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:26:59.172 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:26:59.172 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.172 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.172 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:26:59.172 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:26:59.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 10 librados: watch_flush enter 2024-09-16T05:26:59.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 10 librados: watch_flush exit 2024-09-16T05:26:59.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 20 client.134133.objecter shutdown clearing up homeless session... 2024-09-16T05:26:59.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 10 client.134133.objecter successfully canceled tick 2024-09-16T05:26:59.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 1 -- 172.21.15.114:0/1170151773 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0d94037840 msgr2=0x7f0d94039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 1 --2- 172.21.15.114:0/1170151773 >> v2:172.21.15.149:6812/1185531135 conn(0x7f0d94037840 0x7f0d94039d00 secure :-1 s=READY pgs=3625 cs=0 l=1 rev1=1 crypto rx=0x7f0da00077c0 tx=0x7f0da0009820 comp rx=0 tx=0).stop 2024-09-16T05:26:59.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 10 monclient: shutdown 2024-09-16T05:26:59.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:59.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 1 -- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0da80a0ec0 msgr2=0x7f0da813ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 1 --2- 172.21.15.114:0/1170151773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0da80a0ec0 0x7f0da813ecd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00c3e0 tx=0x7f0d9c00c8a0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dace2b640 1 -- 172.21.15.114:0/1170151773 reap_dead start 2024-09-16T05:26:59.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dace2b640 1 -- 172.21.15.114:0/1170151773 reap_dead start 2024-09-16T05:26:59.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 1 -- 172.21.15.114:0/1170151773 shutdown_connections 2024-09-16T05:26:59.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 1 -- 172.21.15.114:0/1170151773 >> 172.21.15.114:0/1170151773 conn(0x7f0da80a1fb0 msgr2=0x7f0da8138dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:59.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 1 -- 172.21.15.114:0/1170151773 shutdown_connections 2024-09-16T05:26:59.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 1 -- 172.21.15.114:0/1170151773 wait complete. 2024-09-16T05:26:59.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.143+0000 7f0dae88e640 1 librados: shutdown 2024-09-16T05:26:59.175 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:26:59.175 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:26:59.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:26:59.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 10 monclient: get_monmap_and_config 2024-09-16T05:26:59.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 10 monclient: build_initial_monmap 2024-09-16T05:26:59.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 10 monclient: monmap: 2024-09-16T05:26:59.291 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:59.291 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.291 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:26:59.296450+0000 2024-09-16T05:26:59.291 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:26:59.296450+0000 2024-09-16T05:26:59.291 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:59.291 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.291 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:26:59.291 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:26:59.292 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:26:59.292 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:26:59.292 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:26:59.292 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 1 Processor -- start 2024-09-16T05:26:59.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 1 -- start start 2024-09-16T05:26:59.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 10 monclient: init 2024-09-16T05:26:59.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:59.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T05:26:59.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fabcc0a7d60 0x7fabcc0a8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 10 monclient(hunting): picked mon.noname-a con 0x7fabcc0a7d60 addr v2:172.21.15.114:3300/0 2024-09-16T05:26:59.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fabcc0a9190 addr v1:172.21.15.114:6789/0 2024-09-16T05:26:59.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fabcc0a7600 addr v1:172.21.15.149:6789/0 2024-09-16T05:26:59.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabcc137f50 con 0x7fabcc0a9190 2024-09-16T05:26:59.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabcc0a8a70 con 0x7fabcc0a7600 2024-09-16T05:26:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fabcc0a88a0 con 0x7fabcc0a7d60 2024-09-16T05:26:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabd161c640 10 monclient(hunting): authenticate will time out at 2103.925666s 2024-09-16T05:26:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fabcc0a7d60 0x7fabcc0a8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fabcc0a7d60 0x7fabcc0a8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35042/0 (socket says 172.21.15.114:35042) 2024-09-16T05:26:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 1 -- 172.21.15.114:0/788057649 learned_addr learned my addr 172.21.15.114:0/788057649 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 10 monclient(hunting): get_auth_request con 0x7fabcc0a7d60 auth_method 0 2024-09-16T05:26:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabca7fc640 1 --1- 172.21.15.114:0/788057649 >> v1:172.21.15.114:6789/0 conn(0x7fabcc0a9190 0x7fabcc137130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37574/0 (socket says 172.21.15.114:37574) 2024-09-16T05:26:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:59.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:59.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:59.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 10 monclient(hunting): handle_auth_done global_id 134136 payload 274 2024-09-16T05:26:59.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 1 -- 172.21.15.114:0/788057649 >> v1:172.21.15.149:6789/0 conn(0x7fabcc0a7600 legacy=0x7fabcc137840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:26:59.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 1 -- 172.21.15.114:0/788057649 >> v1:172.21.15.114:6789/0 conn(0x7fabcc0a9190 legacy=0x7fabcc137130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 10 monclient: _finish_hunting 0 2024-09-16T05:26:59.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 1 monclient: found mon.noname-a 2024-09-16T05:26:59.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:59.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:26:59.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.295+0000 7fabcaffd640 1 -- 172.21.15.114:0/788057649 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabcc0a3710 con 0x7fabcc0a7d60 2024-09-16T05:26:59.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabcaffd640 1 --2- 172.21.15.114:0/788057649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabcc0a7d60 0x7fabcc0a8140 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009870 tx=0x7fabc001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a58d62f734a91af server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:59.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabc9ffb640 1 -- 172.21.15.114:0/788057649 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc00b0070 con 0x7fabcc0a7d60 2024-09-16T05:26:59.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabc9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:26:59.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabc9ffb640 10 monclient: dump: 2024-09-16T05:26:59.304 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.304 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.304 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.305 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.305 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.305 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.305 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.305 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.305 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.305 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabc9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:26:59.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabc9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:26:59.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:59.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabc9ffb640 1 -- 172.21.15.114:0/788057649 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabc00ab600 con 0x7fabcc0a7d60 2024-09-16T05:26:59.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabc9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:59.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabc9ffb640 1 -- 172.21.15.114:0/788057649 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc00ab9d0 con 0x7fabcc0a7d60 2024-09-16T05:26:59.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabc9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:26:59.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabc9ffb640 10 monclient: dump: 2024-09-16T05:26:59.306 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.306 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.306 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.306 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.306 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.306 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.306 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.306 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.306 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.306 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 5 monclient: authenticate success, global_id 134136 2024-09-16T05:26:59.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 10 monclient: get_monmap_and_config success 2024-09-16T05:26:59.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 4 set_mon_vals no callback set 2024-09-16T05:26:59.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 10 monclient: shutdown 2024-09-16T05:26:59.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:59.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 1 -- 172.21.15.114:0/788057649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabcc0a7d60 msgr2=0x7fabcc0a8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 1 --2- 172.21.15.114:0/788057649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabcc0a7d60 0x7fabcc0a8140 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009870 tx=0x7fabc001c3e0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 1 -- 172.21.15.114:0/788057649 shutdown_connections 2024-09-16T05:26:59.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 1 --2- 172.21.15.114:0/788057649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabcc0a7d60 0x7fabcc0a8140 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 1 -- 172.21.15.114:0/788057649 >> 172.21.15.114:0/788057649 conn(0x7fabcc0a9e00 msgr2=0x7fabcc0a6300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:59.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 1 -- 172.21.15.114:0/788057649 shutdown_connections 2024-09-16T05:26:59.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 1 -- 172.21.15.114:0/788057649 wait complete. 2024-09-16T05:26:59.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 10 monclient: build_initial_monmap 2024-09-16T05:26:59.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 10 monclient: monmap: 2024-09-16T05:26:59.307 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:59.308 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:26:59.308 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:26:59.308 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:26:59.308 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:59.308 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.308 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:26:59.308 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:26:59.308 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:26:59.308 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 1 librados: starting msgr at 2024-09-16T05:26:59.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 1 librados: starting objecter 2024-09-16T05:26:59.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 1 Processor -- start 2024-09-16T05:26:59.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 1 -- start start 2024-09-16T05:26:59.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 1 librados: setting wanted keys 2024-09-16T05:26:59.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 1 librados: calling monclient init 2024-09-16T05:26:59.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.299+0000 7fabd161c640 10 monclient: init 2024-09-16T05:26:59.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:59.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:26:59.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc0a7600 0x7fabcc1cf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 monclient(hunting): picked mon.noname-b con 0x7fabcc0a7600 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:26:59.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabcc0a7d60 0x7fabcc1371c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 monclient(hunting): picked mon.noname-c con 0x7fabcc0a7d60 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:59.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabcc0a9190 0x7fabcc137700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 monclient(hunting): picked mon.noname-a con 0x7fabcc0a9190 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:26:59.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc142e20 con 0x7fabcc0a9190 2024-09-16T05:26:59.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc142ca0 con 0x7fabcc0a7600 2024-09-16T05:26:59.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fabcc142fa0 con 0x7fabcc0a7d60 2024-09-16T05:26:59.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:59.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 monclient(hunting): authenticate will time out at 2103.932177s 2024-09-16T05:26:59.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabcc0a9190 0x7fabcc137700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabcc0a9190 0x7fabcc137700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35048/0 (socket says 172.21.15.114:35048) 2024-09-16T05:26:59.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 1 -- 172.21.15.114:0/2153665819 learned_addr learned my addr 172.21.15.114:0/2153665819 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:59.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 10 monclient(hunting): get_auth_request con 0x7fabcc0a9190 auth_method 0 2024-09-16T05:26:59.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabca7fc640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabcc0a7d60 0x7fabcc1371c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabca7fc640 10 monclient(hunting): get_auth_request con 0x7fabcc0a7d60 auth_method 0 2024-09-16T05:26:59.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:59.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:59.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc0a7600 0x7fabcc1cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:59.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 10 monclient(hunting): get_auth_request con 0x7fabcc0a7600 auth_method 0 2024-09-16T05:26:59.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:59.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:59.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:59.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:59.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:59.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:59.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 10 monclient(hunting): handle_auth_done global_id 134142 payload 824 2024-09-16T05:26:59.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 1 -- 172.21.15.114:0/2153665819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabcc0a7d60 msgr2=0x7fabcc1371c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fabcc0a7d60 0x7fabcc1371c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 1 -- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc0a7600 msgr2=0x7fabcc1cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc0a7600 0x7fabcc1cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:26:59.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 1 monclient: found mon.noname-a 2024-09-16T05:26:59.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:59.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:26:59.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 1 -- 172.21.15.114:0/2153665819 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabcc120b40 con 0x7fabcc0a9190 2024-09-16T05:26:59.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcb7fe640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabcc0a9190 0x7fabcc137700 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fabc400c780 tx=0x7fabc400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:59.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 client.?.objecter ms_handle_connect 0x7fabcc0a9190 2024-09-16T05:26:59.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:26:59.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 1 -- 172.21.15.114:0/2153665819 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc4004b70 con 0x7fabcc0a9190 2024-09-16T05:26:59.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:26:59.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 monclient: dump: 2024-09-16T05:26:59.314 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.314 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.314 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.314 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.314 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.314 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.314 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.314 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.315 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 monclient: _finish_auth 0 2024-09-16T05:26:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:26:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 1 -- 172.21.15.114:0/2153665819 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabc4004d10 con 0x7fabcc0a9190 2024-09-16T05:26:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 5 monclient: authenticate success, global_id 134142 2024-09-16T05:26:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 monclient: _renew_subs 2024-09-16T05:26:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 1 -- 172.21.15.114:0/2153665819 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabcc09d2f0 con 0x7fabcc0a9190 2024-09-16T05:26:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 1 -- 172.21.15.114:0/2153665819 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc400aa90 con 0x7fabcc0a9190 2024-09-16T05:26:59.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:26:59.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 monclient: dump: 2024-09-16T05:26:59.316 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.316 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.316 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.316 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.316 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.316 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.316 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.316 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.316 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.316 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 client.134142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:26:59.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 monclient: _renew_subs 2024-09-16T05:26:59.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:59.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 1 -- 172.21.15.114:0/2153665819 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabcc13ec00 con 0x7fabcc0a9190 2024-09-16T05:26:59.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 1 librados: init done 2024-09-16T05:26:59.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabc97fa640 4 set_mon_vals no callback set 2024-09-16T05:26:59.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:59.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc13d720 0x7fabcc02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabd161c640 1 -- 172.21.15.114:0/2153665819 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc1204e0 con 0x7fabcc13d720 2024-09-16T05:26:59.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 1 -- 172.21.15.114:0/2153665819 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fabc400ac30 con 0x7fabcc0a9190 2024-09-16T05:26:59.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc13d720 0x7fabcc02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 1 --2- 172.21.15.114:0/2153665819 >> v2:172.21.15.149:6812/1185531135 conn(0x7fabb8037810 0x7fabb8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 10 monclient: get_auth_request con 0x7fabcc13d720 auth_method 0 2024-09-16T05:26:59.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 1 -- 172.21.15.114:0/2153665819 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7fabc4049030 con 0x7fabcc0a9190 2024-09-16T05:26:59.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 client.134142.objecter ms_dispatch 0x7fabcc000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:26:59.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 3 client.134142.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:26:59.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 3 client.134142.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:26:59.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 20 client.134142.objecter dump_active .. 0 homeless 2024-09-16T05:26:59.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:59.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:59.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:59.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabca7fc640 1 --2- 172.21.15.114:0/2153665819 >> v2:172.21.15.149:6812/1185531135 conn(0x7fabb8037810 0x7fabb8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabca7fc640 10 monclient: get_auth_request con 0x7fabb8037810 auth_method 0 2024-09-16T05:26:59.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 10 monclient(hunting): handle_auth_done global_id 124186 payload 824 2024-09-16T05:26:59.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 1 -- 172.21.15.114:0/2153665819 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fabcc0a8140 con 0x7fabcc13d720 2024-09-16T05:26:59.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 client.134142.objecter ms_handle_connect 0x7fabcc13d720 2024-09-16T05:26:59.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabcaffd640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc13d720 0x7fabcc02c650 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0a3ce0 tx=0x7fabc001ac20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:59.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 client.134142.objecter resend_mon_ops 2024-09-16T05:26:59.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 1 -- 172.21.15.114:0/2153665819 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc00b0040 con 0x7fabcc13d720 2024-09-16T05:26:59.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:26:59.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 monclient: dump: 2024-09-16T05:26:59.319 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.319 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.320 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabb37fe640 10 client.134142.objecter ms_handle_connect 0x7fabb8037810 2024-09-16T05:26:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.303+0000 7fabca7fc640 1 --2- 172.21.15.114:0/2153665819 >> v2:172.21.15.149:6812/1185531135 conn(0x7fabb8037810 0x7fabb8039cd0 secure :-1 s=READY pgs=3626 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0a2580 tx=0x7fabbc007e50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.307+0000 7fabb37fe640 1 -- 172.21.15.114:0/2153665819 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fabc00b6670 con 0x7fabcc13d720 2024-09-16T05:26:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.307+0000 7fabb37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.307+0000 7fabb37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:26:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.307+0000 7fabb37fe640 1 -- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc13d720 msgr2=0x7fabcc02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.307+0000 7fabb37fe640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc13d720 0x7fabcc02c650 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0a3ce0 tx=0x7fabc001ac20 comp rx=0 tx=0).stop 2024-09-16T05:26:59.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.307+0000 7fabb37fe640 1 -- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc13d720 msgr2=0x7fabcc02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:26:59.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.307+0000 7fabb37fe640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc13d720 0x7fabcc02c650 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0a3ce0 tx=0x7fabc001ac20 comp rx=0 tx=0).stop 2024-09-16T05:26:59.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabd161c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:26:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabd161c640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc13db00 0x7fabcc02cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabd161c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabd161c640 1 -- 172.21.15.114:0/2153665819 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc068f10 con 0x7fabcc13db00 2024-09-16T05:26:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabcaffd640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc13db00 0x7fabcc02cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabcaffd640 10 monclient: get_auth_request con 0x7fabcc13db00 auth_method 0 2024-09-16T05:26:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabcaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabcaffd640 10 monclient(hunting): handle_auth_done global_id 124189 payload 824 2024-09-16T05:26:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabcaffd640 1 -- 172.21.15.114:0/2153665819 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fabcc0a6740 con 0x7fabcc13db00 2024-09-16T05:26:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabcaffd640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc13db00 0x7fabcc02cb90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fabc001c420 tx=0x7fabc0019180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabb37fe640 10 client.134142.objecter ms_handle_connect 0x7fabcc13db00 2024-09-16T05:26:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.311+0000 7fabb37fe640 10 client.134142.objecter resend_mon_ops 2024-09-16T05:26:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabb37fe640 1 -- 172.21.15.114:0/2153665819 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc00b0040 con 0x7fabcc13db00 2024-09-16T05:26:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabb37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:26:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabb37fe640 10 monclient: dump: 2024-09-16T05:26:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.325 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.325 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.325 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabb37fe640 1 -- 172.21.15.114:0/2153665819 <== mon.1 v2:172.21.15.149:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fabc00bfb60 con 0x7fabcc13db00 2024-09-16T05:26:59.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabb37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:26:59.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabb37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:26:59.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabb37fe640 1 -- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc13db00 msgr2=0x7fabcc02cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabb37fe640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc13db00 0x7fabcc02cb90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fabc001c420 tx=0x7fabc0019180 comp rx=0 tx=0).stop 2024-09-16T05:26:59.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabb37fe640 1 -- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc13db00 msgr2=0x7fabcc02cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:26:59.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabb37fe640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fabcc13db00 0x7fabcc02cb90 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fabc001c420 tx=0x7fabc0019180 comp rx=0 tx=0).stop 2024-09-16T05:26:59.326 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:26:59.326 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:26:59.326 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:26:59.326 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "peon", 2024-09-16T05:26:59.326 INFO:teuthology.orchestra.run.smithi114.stdout: "election_epoch": 84, 2024-09-16T05:26:59.326 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 141341, 2024-09-16T05:26:59.326 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum": [ 2024-09-16T05:26:59.326 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-16T05:26:59.326 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-16T05:26:59.326 INFO:teuthology.orchestra.run.smithi114.stdout: 2 2024-09-16T05:26:59.327 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:26:59.327 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_age": 2, 2024-09-16T05:26:59.327 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:26:59.327 INFO:teuthology.orchestra.run.smithi114.stdout: "required_con": "2450521708007866372", 2024-09-16T05:26:59.327 INFO:teuthology.orchestra.run.smithi114.stdout: "required_mon": [ 2024-09-16T05:26:59.327 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:26:59.327 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:26:59.327 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:26:59.327 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:26:59.327 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:26:59.327 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:26:59.327 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:26:59.327 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:26:59.327 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:26:59.328 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:26:59.328 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:26:59.328 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:26:59.328 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:26:59.328 INFO:teuthology.orchestra.run.smithi114.stdout: "quorum_mon": [ 2024-09-16T05:26:59.328 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:26:59.328 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:26:59.328 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:26:59.328 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:26:59.328 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:26:59.328 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:26:59.328 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:26:59.328 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:26:59.328 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:26:59.329 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:26:59.329 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:26:59.329 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.329 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.329 INFO:teuthology.orchestra.run.smithi114.stdout: "outside_quorum": [], 2024-09-16T05:26:59.329 INFO:teuthology.orchestra.run.smithi114.stdout: "extra_probe_peers": [], 2024-09-16T05:26:59.329 INFO:teuthology.orchestra.run.smithi114.stdout: "sync_provider": [], 2024-09-16T05:26:59.329 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": { 2024-09-16T05:26:59.329 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 1, 2024-09-16T05:26:59.329 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "c6dbfe9e-ab8c-4165-ae76-3f2273766e2c", 2024-09-16T05:26:59.329 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:26:59.329 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-16T05:06:54.422314Z", 2024-09-16T05:26:59.329 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release": 19, 2024-09-16T05:26:59.330 INFO:teuthology.orchestra.run.smithi114.stdout: "min_mon_release_name": "squid", 2024-09-16T05:26:59.330 INFO:teuthology.orchestra.run.smithi114.stdout: "election_strategy": 1, 2024-09-16T05:26:59.330 INFO:teuthology.orchestra.run.smithi114.stdout: "disallowed_leaders: ": "", 2024-09-16T05:26:59.330 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false, 2024-09-16T05:26:59.330 INFO:teuthology.orchestra.run.smithi114.stdout: "tiebreaker_mon": "", 2024-09-16T05:26:59.330 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_ranks: ": "", 2024-09-16T05:26:59.330 INFO:teuthology.orchestra.run.smithi114.stdout: "features": { 2024-09-16T05:26:59.330 INFO:teuthology.orchestra.run.smithi114.stdout: "persistent": [ 2024-09-16T05:26:59.330 INFO:teuthology.orchestra.run.smithi114.stdout: "kraken", 2024-09-16T05:26:59.330 INFO:teuthology.orchestra.run.smithi114.stdout: "luminous", 2024-09-16T05:26:59.330 INFO:teuthology.orchestra.run.smithi114.stdout: "mimic", 2024-09-16T05:26:59.330 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap-prune", 2024-09-16T05:26:59.330 INFO:teuthology.orchestra.run.smithi114.stdout: "nautilus", 2024-09-16T05:26:59.330 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus", 2024-09-16T05:26:59.331 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific", 2024-09-16T05:26:59.331 INFO:teuthology.orchestra.run.smithi114.stdout: "elector-pinging", 2024-09-16T05:26:59.331 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy", 2024-09-16T05:26:59.331 INFO:teuthology.orchestra.run.smithi114.stdout: "reef", 2024-09-16T05:26:59.331 INFO:teuthology.orchestra.run.smithi114.stdout: "squid" 2024-09-16T05:26:59.331 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:26:59.331 INFO:teuthology.orchestra.run.smithi114.stdout: "optional": [] 2024-09-16T05:26:59.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.331 INFO:teuthology.orchestra.run.smithi114.stdout: "mons": [ 2024-09-16T05:26:59.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.331 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 0, 2024-09-16T05:26:59.331 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "a", 2024-09-16T05:26:59.331 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:26:59.332 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:26:59.332 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.332 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:26:59.332 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3300", 2024-09-16T05:26:59.332 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:59.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.332 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.332 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:26:59.332 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789", 2024-09-16T05:26:59.332 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:59.332 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.332 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.333 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6789/0", 2024-09-16T05:26:59.333 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6789/0", 2024-09-16T05:26:59.333 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:26:59.333 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:26:59.333 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:26:59.333 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.333 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 1, 2024-09-16T05:26:59.333 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "b", 2024-09-16T05:26:59.333 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:26:59.333 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:26:59.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.333 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:26:59.333 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:3300", 2024-09-16T05:26:59.333 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:59.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.334 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.334 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:26:59.334 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789", 2024-09-16T05:26:59.334 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:59.334 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.334 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.334 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.149:6789/0", 2024-09-16T05:26:59.334 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.149:6789/0", 2024-09-16T05:26:59.334 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:26:59.334 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:26:59.334 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:26:59.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.335 INFO:teuthology.orchestra.run.smithi114.stdout: "rank": 2, 2024-09-16T05:26:59.335 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "c", 2024-09-16T05:26:59.335 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-16T05:26:59.335 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-16T05:26:59.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.335 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-16T05:26:59.335 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:3301", 2024-09-16T05:26:59.335 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:59.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.335 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-16T05:26:59.335 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790", 2024-09-16T05:26:59.336 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 0 2024-09-16T05:26:59.336 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.336 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.336 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6790/0", 2024-09-16T05:26:59.336 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6790/0", 2024-09-16T05:26:59.336 INFO:teuthology.orchestra.run.smithi114.stdout: "priority": 0, 2024-09-16T05:26:59.336 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-16T05:26:59.336 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "{}" 2024-09-16T05:26:59.336 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.336 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.336 INFO:teuthology.orchestra.run.smithi114.stdout: "feature_map": { 2024-09-16T05:26:59.337 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": [ 2024-09-16T05:26:59.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.337 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:26:59.337 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:26:59.337 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:26:59.337 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.337 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:26:59.337 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": [ 2024-09-16T05:26:59.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.337 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:26:59.337 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:26:59.337 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:26:59.337 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.338 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-16T05:26:59.338 INFO:teuthology.orchestra.run.smithi114.stdout: "client": [ 2024-09-16T05:26:59.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-16T05:26:59.338 INFO:teuthology.orchestra.run.smithi114.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:26:59.338 INFO:teuthology.orchestra.run.smithi114.stdout: "release": "squid", 2024-09-16T05:26:59.338 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-09-16T05:26:59.338 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-16T05:26:59.338 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T05:26:59.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-16T05:26:59.338 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": false 2024-09-16T05:26:59.338 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:26:59.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabd161c640 10 librados: watch_flush enter 2024-09-16T05:26:59.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabd161c640 10 librados: watch_flush exit 2024-09-16T05:26:59.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabd161c640 20 client.134142.objecter shutdown clearing up homeless session... 2024-09-16T05:26:59.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabd161c640 10 client.134142.objecter successfully canceled tick 2024-09-16T05:26:59.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabd161c640 1 -- 172.21.15.114:0/2153665819 >> v2:172.21.15.149:6812/1185531135 conn(0x7fabb8037810 msgr2=0x7fabb8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabd161c640 1 --2- 172.21.15.114:0/2153665819 >> v2:172.21.15.149:6812/1185531135 conn(0x7fabb8037810 0x7fabb8039cd0 secure :-1 s=READY pgs=3626 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0a2580 tx=0x7fabbc007e50 comp rx=0 tx=0).stop 2024-09-16T05:26:59.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabd161c640 10 monclient: shutdown 2024-09-16T05:26:59.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabd161c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:59.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabd161c640 1 -- 172.21.15.114:0/2153665819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabcc0a9190 msgr2=0x7fabcc137700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabd161c640 1 --2- 172.21.15.114:0/2153665819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabcc0a9190 0x7fabcc137700 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fabc400c780 tx=0x7fabc400cc40 comp rx=0 tx=0).stop 2024-09-16T05:26:59.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabcb7fe640 1 -- 172.21.15.114:0/2153665819 reap_dead start 2024-09-16T05:26:59.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabcb7fe640 1 -- 172.21.15.114:0/2153665819 reap_dead start 2024-09-16T05:26:59.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabd161c640 1 -- 172.21.15.114:0/2153665819 shutdown_connections 2024-09-16T05:26:59.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabd161c640 1 -- 172.21.15.114:0/2153665819 >> 172.21.15.114:0/2153665819 conn(0x7fabcc0a9e00 msgr2=0x7fabcc0a12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:59.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabd161c640 1 -- 172.21.15.114:0/2153665819 shutdown_connections 2024-09-16T05:26:59.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabd161c640 1 -- 172.21.15.114:0/2153665819 wait complete. 2024-09-16T05:26:59.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.315+0000 7fabd161c640 1 librados: shutdown 2024-09-16T05:26:59.342 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:26:59.342 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:26:59.342 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:26:59.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 10 monclient: get_monmap_and_config 2024-09-16T05:26:59.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 10 monclient: build_initial_monmap 2024-09-16T05:26:59.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 10 monclient: monmap: 2024-09-16T05:26:59.452 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:59.452 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.452 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:26:59.457303+0000 2024-09-16T05:26:59.452 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:26:59.457303+0000 2024-09-16T05:26:59.452 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:59.452 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.452 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:26:59.452 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:26:59.452 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:26:59.452 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:26:59.453 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:26:59.453 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 1 Processor -- start 2024-09-16T05:26:59.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 1 -- start start 2024-09-16T05:26:59.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 10 monclient: init 2024-09-16T05:26:59.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:59.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T05:26:59.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14b0137730 addr v1:172.21.15.114:6789/0 2024-09-16T05:26:59.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f14b00a5d00 0x7f14b013c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 10 monclient(hunting): picked mon.noname-a con 0x7f14b00a5d00 addr v2:172.21.15.114:3300/0 2024-09-16T05:26:59.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f14b013ca00 0x7f14b013edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 10 monclient(hunting): picked mon.noname-c con 0x7f14b013ca00 addr v2:172.21.15.149:3300/0 2024-09-16T05:26:59.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14b00a55e0 con 0x7f14b0137730 2024-09-16T05:26:59.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f14b009c450 con 0x7f14b00a5d00 2024-09-16T05:26:59.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f14b00a3710 con 0x7f14b013ca00 2024-09-16T05:26:59.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:59.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14b60b5640 10 monclient(hunting): authenticate will time out at 2104.086517s 2024-09-16T05:26:59.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f14b00a5d00 0x7f14b013c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f14b00a5d00 0x7f14b013c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35054/0 (socket says 172.21.15.114:35054) 2024-09-16T05:26:59.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 1 -- 172.21.15.114:0/3792451854 learned_addr learned my addr 172.21.15.114:0/3792451854 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:59.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 10 monclient(hunting): get_auth_request con 0x7f14b00a5d00 auth_method 0 2024-09-16T05:26:59.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:59.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:59.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:59.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14affff640 1 --2- 172.21.15.114:0/3792451854 >> v2:172.21.15.149:3300/0 conn(0x7f14b013ca00 0x7f14b013edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14affff640 10 monclient(hunting): get_auth_request con 0x7f14b013ca00 auth_method 0 2024-09-16T05:26:59.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 10 monclient(hunting): handle_auth_done global_id 134145 payload 274 2024-09-16T05:26:59.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 1 -- 172.21.15.114:0/3792451854 >> v2:172.21.15.149:3300/0 conn(0x7f14b013ca00 msgr2=0x7f14b013edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:26:59.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 1 --2- 172.21.15.114:0/3792451854 >> v2:172.21.15.149:3300/0 conn(0x7f14b013ca00 0x7f14b013edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 1 -- 172.21.15.114:0/3792451854 >> v1:172.21.15.114:6789/0 conn(0x7f14b0137730 legacy=0x7f14b0137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 10 monclient: _finish_hunting 0 2024-09-16T05:26:59.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 1 monclient: found mon.noname-a 2024-09-16T05:26:59.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:59.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:26:59.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.455+0000 7f14aeffd640 1 -- 172.21.15.114:0/3792451854 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b009be00 con 0x7f14b00a5d00 2024-09-16T05:26:59.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14aeffd640 1 --2- 172.21.15.114:0/3792451854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b00a5d00 0x7f14b013c4c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f14b00a60e0 tx=0x7f14a401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1086d6fdc75b89 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:59.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14ae7fc640 1 -- 172.21.15.114:0/3792451854 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a40b0070 con 0x7f14b00a5d00 2024-09-16T05:26:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:26:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14ae7fc640 10 monclient: dump: 2024-09-16T05:26:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.465 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14ae7fc640 10 monclient: _finish_auth 0 2024-09-16T05:26:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14ae7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:26:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14ae7fc640 1 -- 172.21.15.114:0/3792451854 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14a40ab600 con 0x7f14b00a5d00 2024-09-16T05:26:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 5 monclient: authenticate success, global_id 134145 2024-09-16T05:26:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14ae7fc640 1 -- 172.21.15.114:0/3792451854 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a40ab9d0 con 0x7f14b00a5d00 2024-09-16T05:26:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:26:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14ae7fc640 10 monclient: dump: 2024-09-16T05:26:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.466 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 10 monclient: get_monmap_and_config success 2024-09-16T05:26:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 4 set_mon_vals no callback set 2024-09-16T05:26:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 10 monclient: shutdown 2024-09-16T05:26:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 -- 172.21.15.114:0/3792451854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b00a5d00 msgr2=0x7f14b013c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 --2- 172.21.15.114:0/3792451854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b00a5d00 0x7f14b013c4c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f14b00a60e0 tx=0x7f14a401c3e0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 -- 172.21.15.114:0/3792451854 shutdown_connections 2024-09-16T05:26:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 --2- 172.21.15.114:0/3792451854 >> v2:172.21.15.149:3300/0 conn(0x7f14b013ca00 0x7f14b013edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 --2- 172.21.15.114:0/3792451854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b00a5d00 0x7f14b013c4c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 -- 172.21.15.114:0/3792451854 >> 172.21.15.114:0/3792451854 conn(0x7f14b00a9c70 msgr2=0x7f14b00aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 -- 172.21.15.114:0/3792451854 shutdown_connections 2024-09-16T05:26:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 -- 172.21.15.114:0/3792451854 wait complete. 2024-09-16T05:26:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 10 monclient: build_initial_monmap 2024-09-16T05:26:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 10 monclient: monmap: 2024-09-16T05:26:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:26:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:26:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:26:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:26:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:26:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:26:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:26:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:26:59.468 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 librados: starting msgr at 2024-09-16T05:26:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 librados: starting objecter 2024-09-16T05:26:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 Processor -- start 2024-09-16T05:26:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 -- start start 2024-09-16T05:26:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 librados: setting wanted keys 2024-09-16T05:26:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 librados: calling monclient init 2024-09-16T05:26:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 10 monclient: init 2024-09-16T05:26:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 10 monclient: _reopen_session rank -1 2024-09-16T05:26:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:26:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b00a5d00 0x7f14b01cf340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 10 monclient(hunting): picked mon.noname-a con 0x7f14b00a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:26:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14b0137730 0x7f14b00a20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 10 monclient(hunting): picked mon.noname-c con 0x7f14b0137730 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:26:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f14b013ca00 0x7f14b00a25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 10 monclient(hunting): picked mon.noname-b con 0x7f14b013ca00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:26:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f14b0141cd0 con 0x7f14b00a5d00 2024-09-16T05:26:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f14b0141b50 con 0x7f14b013ca00 2024-09-16T05:26:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 10 monclient(hunting): start opening mon connection 2024-09-16T05:26:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f14b0141e50 con 0x7f14b0137730 2024-09-16T05:26:59.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 10 monclient(hunting): _renew_subs 2024-09-16T05:26:59.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14b60b5640 10 monclient(hunting): authenticate will time out at 2104.091356s 2024-09-16T05:26:59.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b00a5d00 0x7f14b01cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14aeffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14b0137730 0x7f14b00a20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b00a5d00 0x7f14b01cf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35066/0 (socket says 172.21.15.114:35066) 2024-09-16T05:26:59.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 1 -- 172.21.15.114:0/1635488883 learned_addr learned my addr 172.21.15.114:0/1635488883 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:26:59.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 10 monclient(hunting): get_auth_request con 0x7f14b00a5d00 auth_method 0 2024-09-16T05:26:59.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14aeffd640 10 monclient(hunting): get_auth_request con 0x7f14b0137730 auth_method 0 2024-09-16T05:26:59.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:59.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:59.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:59.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:26:59.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:26:59.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:26:59.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14affff640 1 --2- 172.21.15.114:0/1635488883 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f14b013ca00 0x7f14b00a25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14affff640 10 monclient(hunting): get_auth_request con 0x7f14b013ca00 auth_method 0 2024-09-16T05:26:59.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:26:59.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:26:59.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:26:59.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 10 monclient(hunting): handle_auth_done global_id 134148 payload 824 2024-09-16T05:26:59.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 1 -- 172.21.15.114:0/1635488883 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14b0137730 msgr2=0x7f14b00a20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 1 --2- 172.21.15.114:0/1635488883 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14b0137730 0x7f14b00a20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 1 -- 172.21.15.114:0/1635488883 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f14b013ca00 msgr2=0x7f14b00a25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 1 --2- 172.21.15.114:0/1635488883 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f14b013ca00 0x7f14b00a25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:26:59.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 1 monclient: found mon.noname-a 2024-09-16T05:26:59.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:26:59.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:26:59.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14af7fe640 1 -- 172.21.15.114:0/1635488883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b00a0890 con 0x7f14b00a5d00 2024-09-16T05:26:59.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.459+0000 7f14aeffd640 1 --2- 172.21.15.114:0/1635488883 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14b0137730 0x7f14b00a20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:26:59.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14af7fe640 1 --2- 172.21.15.114:0/1635488883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b00a5d00 0x7f14b01cf340 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f14a000c790 tx=0x7f14a000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:59.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 10 client.?.objecter ms_handle_connect 0x7f14b00a5d00 2024-09-16T05:26:59.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 10 client.?.objecter resend_mon_ops 2024-09-16T05:26:59.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 1 -- 172.21.15.114:0/1635488883 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a0016020 con 0x7f14b00a5d00 2024-09-16T05:26:59.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:26:59.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 10 monclient: dump: 2024-09-16T05:26:59.473 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.473 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.473 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.473 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.473 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.473 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.473 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.473 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.473 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.474 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 10 monclient: _finish_auth 0 2024-09-16T05:26:59.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 10 monclient: _check_auth_tickets 2024-09-16T05:26:59.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:26:59.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14b60b5640 5 monclient: authenticate success, global_id 134148 2024-09-16T05:26:59.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 1 -- 172.21.15.114:0/1635488883 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14a0006d60 con 0x7f14b00a5d00 2024-09-16T05:26:59.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14b60b5640 10 monclient: _renew_subs 2024-09-16T05:26:59.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14b60b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:59.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14b60b5640 1 -- 172.21.15.114:0/1635488883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b00a0b70 con 0x7f14b00a5d00 2024-09-16T05:26:59.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:26:59.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 1 -- 172.21.15.114:0/1635488883 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a0003bc0 con 0x7f14b00a5d00 2024-09-16T05:26:59.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:26:59.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:26:59.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 10 monclient: dump: 2024-09-16T05:26:59.475 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:26:59.475 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:26:59.475 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.475 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:26:59.475 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:26:59.475 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:26:59.475 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:26:59.475 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:26:59.475 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:26:59.475 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:26:59.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14b60b5640 10 client.134148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:26:59.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14adffb640 4 set_mon_vals no callback set 2024-09-16T05:26:59.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14b60b5640 10 monclient: _renew_subs 2024-09-16T05:26:59.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14b60b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:59.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14b60b5640 1 -- 172.21.15.114:0/1635488883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b00a8b70 con 0x7f14b00a5d00 2024-09-16T05:26:59.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14b60b5640 1 librados: init done 2024-09-16T05:26:59.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 1 -- 172.21.15.114:0/1635488883 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f14a000a850 con 0x7f14b00a5d00 2024-09-16T05:26:59.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14b60b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:59.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14b60b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:59.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14b60b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:59.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14b60b5640 1 -- 172.21.15.114:0/1635488883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14b0137b10 con 0x7f14b00a5d00 2024-09-16T05:26:59.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 1 --2- 172.21.15.114:0/1635488883 >> v2:172.21.15.149:6812/1185531135 conn(0x7f149c037810 0x7f149c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:26:59.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 1 -- 172.21.15.114:0/1635488883 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7f14a0048810 con 0x7f14b00a5d00 2024-09-16T05:26:59.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 10 client.134148.objecter ms_dispatch 0x7f14b0000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:26:59.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 3 client.134148.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:26:59.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 3 client.134148.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:26:59.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 20 client.134148.objecter dump_active .. 0 homeless 2024-09-16T05:26:59.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14aeffd640 1 --2- 172.21.15.114:0/1635488883 >> v2:172.21.15.149:6812/1185531135 conn(0x7f149c037810 0x7f149c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:26:59.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14aeffd640 10 monclient: get_auth_request con 0x7f149c037810 auth_method 0 2024-09-16T05:26:59.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f1497fff640 10 client.134148.objecter ms_handle_connect 0x7f149c037810 2024-09-16T05:26:59.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.463+0000 7f14aeffd640 1 --2- 172.21.15.114:0/1635488883 >> v2:172.21.15.149:6812/1185531135 conn(0x7f149c037810 0x7f149c039cd0 secure :-1 s=READY pgs=3627 cs=0 l=1 rev1=1 crypto rx=0x7f14a40ad000 tx=0x7f14a4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:26:59.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.467+0000 7f1497fff640 1 -- 172.21.15.114:0/1635488883 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f14a0020020 con 0x7f14b00a5d00 2024-09-16T05:26:59.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.467+0000 7f1497fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:26:59.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.467+0000 7f1497fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:26:59.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.683+0000 7f14b60b5640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:26:59.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.683+0000 7f14b60b5640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:26:59.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.683+0000 7f14b60b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:26:59.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.683+0000 7f14b60b5640 1 -- 172.21.15.114:0/1635488883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f14b0073820 con 0x7f14b00a5d00 2024-09-16T05:26:59.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f1497fff640 1 -- 172.21.15.114:0/1635488883 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f14a000ae20 con 0x7f14b00a5d00 2024-09-16T05:26:59.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f1497fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:26:59.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f1497fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:26:59.684 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.687+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.684 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.687+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=2981234857,config=2586702516,health=314794473,kv=2757161437,logm=3629734974}) 2024-09-16T05:26:59.684 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.687+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=12,kv=12,logm=51} crc {auth=279516036,config=2586702516,health=314794473,kv=2757161437,logm=3841646074}) 2024-09-16T05:26:59.684 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.687+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.684 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.687+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=2981234857,config=2586702516,health=314794473,kv=2757161437,logm=3629734974}) 2024-09-16T05:26:59.684 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.687+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=1,health=10,kv=10,logm=56} crc {auth=279516036,config=1228700967,health=679511669,kv=1091748972,logm=1099521245}) 2024-09-16T05:26:59.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 10 librados: watch_flush enter 2024-09-16T05:26:59.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 10 librados: watch_flush exit 2024-09-16T05:26:59.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 20 client.134148.objecter shutdown clearing up homeless session... 2024-09-16T05:26:59.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 10 client.134148.objecter successfully canceled tick 2024-09-16T05:26:59.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 1 -- 172.21.15.114:0/1635488883 >> v2:172.21.15.149:6812/1185531135 conn(0x7f149c037810 msgr2=0x7f149c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 1 --2- 172.21.15.114:0/1635488883 >> v2:172.21.15.149:6812/1185531135 conn(0x7f149c037810 0x7f149c039cd0 secure :-1 s=READY pgs=3627 cs=0 l=1 rev1=1 crypto rx=0x7f14a40ad000 tx=0x7f14a4008040 comp rx=0 tx=0).stop 2024-09-16T05:26:59.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 10 monclient: shutdown 2024-09-16T05:26:59.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:26:59.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 1 -- 172.21.15.114:0/1635488883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b00a5d00 msgr2=0x7f14b01cf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:26:59.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 1 --2- 172.21.15.114:0/1635488883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b00a5d00 0x7f14b01cf340 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f14a000c790 tx=0x7f14a000cc50 comp rx=0 tx=0).stop 2024-09-16T05:26:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 1 -- 172.21.15.114:0/1635488883 shutdown_connections 2024-09-16T05:26:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 1 --2- 172.21.15.114:0/1635488883 >> v2:172.21.15.149:6812/1185531135 conn(0x7f149c037810 0x7f149c039cd0 unknown :-1 s=CLOSED pgs=3627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 1 --2- 172.21.15.114:0/1635488883 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f14b013ca00 0x7f14b00a25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 1 --2- 172.21.15.114:0/1635488883 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14b0137730 0x7f14b00a20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 1 --2- 172.21.15.114:0/1635488883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b00a5d00 0x7f14b01cf340 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:26:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 1 -- 172.21.15.114:0/1635488883 >> 172.21.15.114:0/1635488883 conn(0x7f14b00a9c70 msgr2=0x7f14b013e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:26:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 1 -- 172.21.15.114:0/1635488883 shutdown_connections 2024-09-16T05:26:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 1 -- 172.21.15.114:0/1635488883 wait complete. 2024-09-16T05:26:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:26:59.687+0000 7f14b60b5640 1 librados: shutdown 2024-09-16T05:26:59.688 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.691+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.688 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.691+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=518970111}) 2024-09-16T05:26:59.688 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.691+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1435206810}) 2024-09-16T05:26:59.688 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.691+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.688 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.691+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=518970111}) 2024-09-16T05:26:59.688 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.691+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2197477467}) 2024-09-16T05:26:59.690 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.691+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.690 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.691+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428610789}) 2024-09-16T05:26:59.690 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.691+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1306211111}) 2024-09-16T05:26:59.690 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.691+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.690 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.691+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428610789}) 2024-09-16T05:26:59.690 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.691+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=420227866}) 2024-09-16T05:26:59.693 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.695+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.693 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.695+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1899149073}) 2024-09-16T05:26:59.693 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.695+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3930140144}) 2024-09-16T05:26:59.693 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.695+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.693 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.695+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1899149073}) 2024-09-16T05:26:59.693 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.695+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3611953935}) 2024-09-16T05:26:59.696 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.699+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.696 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.699+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114297590}) 2024-09-16T05:26:59.696 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.699+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1944734193}) 2024-09-16T05:26:59.696 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.699+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.697 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.699+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114297590}) 2024-09-16T05:26:59.697 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.699+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3879554594}) 2024-09-16T05:26:59.699 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.703+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.699 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.703+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783391384}) 2024-09-16T05:26:59.699 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.703+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1947760812}) 2024-09-16T05:26:59.699 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.703+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.699 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.703+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783391384}) 2024-09-16T05:26:59.699 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.703+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=111481256}) 2024-09-16T05:26:59.702 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:26:59.702 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.703+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.703 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.703+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3972323378}) 2024-09-16T05:26:59.703 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.703+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4127423436}) 2024-09-16T05:26:59.703 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.703+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.703 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.703+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3972323378}) 2024-09-16T05:26:59.703 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.703+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2798976033}) 2024-09-16T05:26:59.704 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.707+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.704 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.707+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3725279597}) 2024-09-16T05:26:59.704 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.707+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4174118068}) 2024-09-16T05:26:59.704 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.707+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.704 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.707+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3725279597}) 2024-09-16T05:26:59.704 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.707+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2773843355}) 2024-09-16T05:26:59.706 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.707+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.706 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.707+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186522321}) 2024-09-16T05:26:59.706 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.707+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3939623308}) 2024-09-16T05:26:59.706 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.707+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.706 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.707+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186522321}) 2024-09-16T05:26:59.706 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.707+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3539318419}) 2024-09-16T05:26:59.708 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.711+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.708 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.711+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086794336}) 2024-09-16T05:26:59.708 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.711+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2950362741}) 2024-09-16T05:26:59.708 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.711+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.708 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.711+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086794336}) 2024-09-16T05:26:59.709 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.711+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3087879670}) 2024-09-16T05:26:59.710 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.711+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.710 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.711+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2906336208}) 2024-09-16T05:26:59.710 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.711+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2275340050}) 2024-09-16T05:26:59.710 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.711+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.710 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.711+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2906336208}) 2024-09-16T05:26:59.710 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.711+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2440219755}) 2024-09-16T05:26:59.711 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.715+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.712 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.715+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1566552419}) 2024-09-16T05:26:59.712 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.715+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=730386966}) 2024-09-16T05:26:59.712 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.715+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.712 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.715+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1566552419}) 2024-09-16T05:26:59.712 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.715+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2542111130}) 2024-09-16T05:26:59.713 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.715+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.713 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.715+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494481636}) 2024-09-16T05:26:59.713 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.715+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1675548784}) 2024-09-16T05:26:59.714 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.715+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.714 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.715+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494481636}) 2024-09-16T05:26:59.714 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.715+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2055404339}) 2024-09-16T05:26:59.715 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.719+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.716 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.719+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3399094237}) 2024-09-16T05:26:59.716 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.719+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=954125200}) 2024-09-16T05:26:59.716 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.719+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.716 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.719+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3399094237}) 2024-09-16T05:26:59.716 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.719+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2925998753}) 2024-09-16T05:26:59.717 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.719+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.717 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.719+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=337707365}) 2024-09-16T05:26:59.718 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.719+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2646954339}) 2024-09-16T05:26:59.718 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.719+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.718 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.719+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=337707365}) 2024-09-16T05:26:59.718 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.719+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3485191487}) 2024-09-16T05:26:59.719 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.723+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.720 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.723+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638065667}) 2024-09-16T05:26:59.720 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.723+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2643960667}) 2024-09-16T05:26:59.720 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.723+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.720 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.723+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638065667}) 2024-09-16T05:26:59.720 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.723+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1650471066}) 2024-09-16T05:26:59.721 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.723+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.722 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.723+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667628798}) 2024-09-16T05:26:59.722 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.723+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1645370056}) 2024-09-16T05:26:59.722 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.723+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.722 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.723+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667628798}) 2024-09-16T05:26:59.722 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.723+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1266056302}) 2024-09-16T05:26:59.723 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.727+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.723 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.727+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344266290}) 2024-09-16T05:26:59.724 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.727+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125406280}) 2024-09-16T05:26:59.724 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.727+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.724 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.727+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344266290}) 2024-09-16T05:26:59.724 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.727+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=517167576}) 2024-09-16T05:26:59.725 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.727+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.725 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.727+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2877878815}) 2024-09-16T05:26:59.725 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.727+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3688858735}) 2024-09-16T05:26:59.726 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.727+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.726 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.727+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2877878815}) 2024-09-16T05:26:59.726 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.727+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=383558045}) 2024-09-16T05:26:59.727 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.731+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.727 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.731+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2043723786}) 2024-09-16T05:26:59.727 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.731+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2538189950}) 2024-09-16T05:26:59.728 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.731+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.728 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.731+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2043723786}) 2024-09-16T05:26:59.728 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.731+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=649070107}) 2024-09-16T05:26:59.729 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.731+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.729 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.731+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834556124}) 2024-09-16T05:26:59.729 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.731+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4252230220}) 2024-09-16T05:26:59.729 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.731+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.729 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.731+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834556124}) 2024-09-16T05:26:59.730 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.731+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3583211568}) 2024-09-16T05:26:59.731 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.735+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.731 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.735+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1753547549}) 2024-09-16T05:26:59.731 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.735+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3441134697}) 2024-09-16T05:26:59.731 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.735+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.731 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.735+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1753547549}) 2024-09-16T05:26:59.732 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.735+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3443179553}) 2024-09-16T05:26:59.733 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.735+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.733 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.735+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799826314}) 2024-09-16T05:26:59.733 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.735+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2706621099}) 2024-09-16T05:26:59.733 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.735+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.733 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.735+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799826314}) 2024-09-16T05:26:59.733 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.735+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2649416670}) 2024-09-16T05:26:59.735 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.739+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.735 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.739+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870596721}) 2024-09-16T05:26:59.735 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.739+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4059956833}) 2024-09-16T05:26:59.735 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.739+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.735 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.739+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870596721}) 2024-09-16T05:26:59.735 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.739+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3356122576}) 2024-09-16T05:26:59.737 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.739+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.737 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.739+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365816982}) 2024-09-16T05:26:59.737 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.739+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1934843438}) 2024-09-16T05:26:59.737 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.739+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.737 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.739+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365816982}) 2024-09-16T05:26:59.737 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.739+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3357751781}) 2024-09-16T05:26:59.739 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.743+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.739 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.743+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4233839233}) 2024-09-16T05:26:59.739 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.743+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3602845717}) 2024-09-16T05:26:59.739 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.743+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.739 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.743+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4233839233}) 2024-09-16T05:26:59.739 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.743+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3545287729}) 2024-09-16T05:26:59.741 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.743+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.741 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.743+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058989229}) 2024-09-16T05:26:59.741 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.743+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=758422429}) 2024-09-16T05:26:59.741 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.743+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.741 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.743+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058989229}) 2024-09-16T05:26:59.741 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.743+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2066869057}) 2024-09-16T05:26:59.742 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.743+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.742 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1260643948}) 2024-09-16T05:26:59.743 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2025681216}) 2024-09-16T05:26:59.743 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.743 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1260643948}) 2024-09-16T05:26:59.743 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2421240569}) 2024-09-16T05:26:59.744 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.744 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1524537499}) 2024-09-16T05:26:59.745 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816917045}) 2024-09-16T05:26:59.745 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.745 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1524537499}) 2024-09-16T05:26:59.745 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=558596417}) 2024-09-16T05:26:59.746 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.746 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2328376762}) 2024-09-16T05:26:59.746 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=888911239}) 2024-09-16T05:26:59.746 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.747 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2328376762}) 2024-09-16T05:26:59.747 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.747+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2218649265}) 2024-09-16T05:26:59.748 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.751+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.748 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.751+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234182735}) 2024-09-16T05:26:59.748 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.751+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=159323361}) 2024-09-16T05:26:59.748 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.751+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.748 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.751+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234182735}) 2024-09-16T05:26:59.748 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.751+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=448366240}) 2024-09-16T05:26:59.750 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.751+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.750 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.751+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494129569}) 2024-09-16T05:26:59.750 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.751+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3027146970}) 2024-09-16T05:26:59.750 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.751+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.750 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.751+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494129569}) 2024-09-16T05:26:59.750 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.751+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3689803420}) 2024-09-16T05:26:59.752 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.755+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.752 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.755+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689799971}) 2024-09-16T05:26:59.752 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.755+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2255813221}) 2024-09-16T05:26:59.752 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.755+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.752 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.755+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689799971}) 2024-09-16T05:26:59.753 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.755+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1533635498}) 2024-09-16T05:26:59.754 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.755+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.754 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.755+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770728898}) 2024-09-16T05:26:59.754 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.755+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=287515643}) 2024-09-16T05:26:59.754 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.755+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.754 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.755+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770728898}) 2024-09-16T05:26:59.755 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.755+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1053732806}) 2024-09-16T05:26:59.756 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.759+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.756 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.759+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422601696}) 2024-09-16T05:26:59.756 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.759+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2462756004}) 2024-09-16T05:26:59.757 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.759+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.757 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.759+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422601696}) 2024-09-16T05:26:59.757 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.759+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=112004647}) 2024-09-16T05:26:59.758 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.759+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.758 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.759+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972281965}) 2024-09-16T05:26:59.758 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.759+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2480607381}) 2024-09-16T05:26:59.759 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.759+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.759 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.759+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972281965}) 2024-09-16T05:26:59.759 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.763+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3136653841}) 2024-09-16T05:26:59.760 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.763+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.760 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.763+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306711026}) 2024-09-16T05:26:59.760 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.763+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2212778993}) 2024-09-16T05:26:59.761 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.763+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.761 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.763+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306711026}) 2024-09-16T05:26:59.761 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.763+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=80511307}) 2024-09-16T05:26:59.762 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.763+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.763 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.763+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2968563379}) 2024-09-16T05:26:59.763 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.763+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1133483639}) 2024-09-16T05:26:59.763 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.763+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.763 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.767+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2968563379}) 2024-09-16T05:26:59.763 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.767+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2789397022}) 2024-09-16T05:26:59.765 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.767+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.765 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.767+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539074141}) 2024-09-16T05:26:59.765 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.767+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2404143318}) 2024-09-16T05:26:59.765 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.767+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.765 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.767+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539074141}) 2024-09-16T05:26:59.765 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.767+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3801759480}) 2024-09-16T05:26:59.766 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.767+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.766 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.767+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=776172686}) 2024-09-16T05:26:59.767 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.767+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1841174019}) 2024-09-16T05:26:59.767 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.767+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.767 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.767+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=776172686}) 2024-09-16T05:26:59.767 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.767+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3261309540}) 2024-09-16T05:26:59.769 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.771+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.769 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.771+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242031549}) 2024-09-16T05:26:59.769 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.771+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1958561399}) 2024-09-16T05:26:59.769 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.771+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.769 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.771+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242031549}) 2024-09-16T05:26:59.769 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.771+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=877982435}) 2024-09-16T05:26:59.771 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.771+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.771 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.771+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939605500}) 2024-09-16T05:26:59.771 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.771+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2744856593}) 2024-09-16T05:26:59.771 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.771+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.771 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.771+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939605500}) 2024-09-16T05:26:59.771 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.771+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1695097951}) 2024-09-16T05:26:59.772 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.775+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.773 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.775+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1740062165}) 2024-09-16T05:26:59.773 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.775+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1705414913}) 2024-09-16T05:26:59.773 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.775+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.773 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.775+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1740062165}) 2024-09-16T05:26:59.773 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.775+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3665094640}) 2024-09-16T05:26:59.775 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.775+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.775 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.775+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895400419}) 2024-09-16T05:26:59.775 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.775+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=895945052}) 2024-09-16T05:26:59.775 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.779+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.775 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.779+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895400419}) 2024-09-16T05:26:59.775 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.779+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1170331833}) 2024-09-16T05:26:59.777 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.779+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.777 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.779+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1334457636}) 2024-09-16T05:26:59.777 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.779+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1193631765}) 2024-09-16T05:26:59.777 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.779+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.777 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.779+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1334457636}) 2024-09-16T05:26:59.777 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.779+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2945896186}) 2024-09-16T05:26:59.779 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.783+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.779 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.783+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249525773}) 2024-09-16T05:26:59.779 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.783+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3155979906}) 2024-09-16T05:26:59.779 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.783+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.779 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.783+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249525773}) 2024-09-16T05:26:59.779 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.783+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1205237291}) 2024-09-16T05:26:59.781 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.783+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.782 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.783+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882457281}) 2024-09-16T05:26:59.782 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.783+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153672731}) 2024-09-16T05:26:59.782 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.783+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.782 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.783+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882457281}) 2024-09-16T05:26:59.782 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.783+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2827633114}) 2024-09-16T05:26:59.783 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.787+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.783 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.787+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545461846}) 2024-09-16T05:26:59.783 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.787+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1026001793}) 2024-09-16T05:26:59.783 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.787+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.783 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.787+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545461846}) 2024-09-16T05:26:59.784 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.787+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3747580762}) 2024-09-16T05:26:59.785 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.787+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.785 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.787+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283677050}) 2024-09-16T05:26:59.785 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.787+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2409277786}) 2024-09-16T05:26:59.785 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.787+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.786 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.787+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283677050}) 2024-09-16T05:26:59.786 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.787+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1267228463}) 2024-09-16T05:26:59.787 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.791+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.787 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.791+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2491430595}) 2024-09-16T05:26:59.787 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.791+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=44207984}) 2024-09-16T05:26:59.787 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.791+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.787 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.791+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2491430595}) 2024-09-16T05:26:59.787 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.791+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3913446029}) 2024-09-16T05:26:59.789 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.791+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.789 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.791+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983085536}) 2024-09-16T05:26:59.790 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.791+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4097628715}) 2024-09-16T05:26:59.790 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.791+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.790 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.791+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983085536}) 2024-09-16T05:26:59.790 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.791+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2373446631}) 2024-09-16T05:26:59.791 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.795+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.791 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.795+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1723168359}) 2024-09-16T05:26:59.791 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.795+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1368539695}) 2024-09-16T05:26:59.791 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.795+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.791 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.795+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1723168359}) 2024-09-16T05:26:59.791 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.795+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2084385691}) 2024-09-16T05:26:59.793 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.795+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.793 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.795+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3681611547}) 2024-09-16T05:26:59.793 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.795+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2003505343}) 2024-09-16T05:26:59.793 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.795+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.793 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.795+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3681611547}) 2024-09-16T05:26:59.794 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.795+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1849770878}) 2024-09-16T05:26:59.795 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.799+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.795 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.799+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1428121091}) 2024-09-16T05:26:59.795 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.799+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2025333821}) 2024-09-16T05:26:59.795 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.799+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.795 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.799+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1428121091}) 2024-09-16T05:26:59.795 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.799+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3190061071}) 2024-09-16T05:26:59.797 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.799+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.797 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.799+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279606658}) 2024-09-16T05:26:59.797 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.799+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3581121700}) 2024-09-16T05:26:59.797 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.799+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.797 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.799+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279606658}) 2024-09-16T05:26:59.797 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.799+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1313553045}) 2024-09-16T05:26:59.799 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.803+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.799 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.803+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594714972}) 2024-09-16T05:26:59.799 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.803+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3310251355}) 2024-09-16T05:26:59.799 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.803+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.799 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.803+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594714972}) 2024-09-16T05:26:59.799 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.803+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2967131347}) 2024-09-16T05:26:59.801 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.803+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.801 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.803+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2843311658}) 2024-09-16T05:26:59.801 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.803+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=196610500}) 2024-09-16T05:26:59.801 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.803+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.801 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.803+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2843311658}) 2024-09-16T05:26:59.801 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.803+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=198552534}) 2024-09-16T05:26:59.803 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.807+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.803 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.807+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3811816032}) 2024-09-16T05:26:59.803 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.807+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307771345}) 2024-09-16T05:26:59.803 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.807+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.803 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.807+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3811816032}) 2024-09-16T05:26:59.803 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.807+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3118607692}) 2024-09-16T05:26:59.805 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.807+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.805 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.807+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3695785847}) 2024-09-16T05:26:59.805 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.807+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3027901569}) 2024-09-16T05:26:59.805 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.807+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.805 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.807+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3695785847}) 2024-09-16T05:26:59.805 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.807+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651228825}) 2024-09-16T05:26:59.807 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.811+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.807 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.811+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243574557}) 2024-09-16T05:26:59.807 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.811+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=90993617}) 2024-09-16T05:26:59.807 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.811+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.807 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.811+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243574557}) 2024-09-16T05:26:59.807 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.811+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1677455736}) 2024-09-16T05:26:59.809 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.811+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.809 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.811+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451475814}) 2024-09-16T05:26:59.809 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.811+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3405084201}) 2024-09-16T05:26:59.809 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.811+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.809 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.811+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451475814}) 2024-09-16T05:26:59.809 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.811+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4258792999}) 2024-09-16T05:26:59.811 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.815+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.811 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.815+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570281851}) 2024-09-16T05:26:59.811 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.815+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1993503609}) 2024-09-16T05:26:59.811 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.815+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.811 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.815+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570281851}) 2024-09-16T05:26:59.811 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.815+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3164874119}) 2024-09-16T05:26:59.813 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.815+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.813 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.815+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479376021}) 2024-09-16T05:26:59.813 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.815+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=518269503}) 2024-09-16T05:26:59.813 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.815+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.813 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.815+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479376021}) 2024-09-16T05:26:59.813 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.815+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=105985342}) 2024-09-16T05:26:59.815 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.815 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076421479}) 2024-09-16T05:26:59.815 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2139636879}) 2024-09-16T05:26:59.815 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.815 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076421479}) 2024-09-16T05:26:59.815 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3812879385}) 2024-09-16T05:26:59.817 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.817 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2806732669}) 2024-09-16T05:26:59.817 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=807634657}) 2024-09-16T05:26:59.817 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.817 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2806732669}) 2024-09-16T05:26:59.817 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3024961035}) 2024-09-16T05:26:59.818 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.819 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000018583}) 2024-09-16T05:26:59.819 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4058563799}) 2024-09-16T05:26:59.819 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.819+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.819 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.823+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000018583}) 2024-09-16T05:26:59.819 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.823+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1663409404}) 2024-09-16T05:26:59.820 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.823+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.820 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.823+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070532095}) 2024-09-16T05:26:59.821 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.823+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1950441285}) 2024-09-16T05:26:59.821 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.823+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.821 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.823+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070532095}) 2024-09-16T05:26:59.821 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.823+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1700582352}) 2024-09-16T05:26:59.822 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.823+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.822 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.823+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=815120497}) 2024-09-16T05:26:59.823 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.823+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=749598639}) 2024-09-16T05:26:59.823 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.823+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.823 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.823+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=815120497}) 2024-09-16T05:26:59.823 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.823+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3330037530}) 2024-09-16T05:26:59.824 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.827+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.824 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.827+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799028711}) 2024-09-16T05:26:59.825 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.827+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=419931920}) 2024-09-16T05:26:59.825 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.827+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.825 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.827+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799028711}) 2024-09-16T05:26:59.825 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.827+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1317214122}) 2024-09-16T05:26:59.826 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.827+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.826 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.827+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3500156217}) 2024-09-16T05:26:59.826 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.827+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=871216982}) 2024-09-16T05:26:59.827 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.827+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.827 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.827+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3500156217}) 2024-09-16T05:26:59.827 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.827+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2694484510}) 2024-09-16T05:26:59.828 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.831+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.828 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.831+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955364219}) 2024-09-16T05:26:59.828 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.831+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2817517246}) 2024-09-16T05:26:59.828 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.831+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.829 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.831+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955364219}) 2024-09-16T05:26:59.829 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.831+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=780994738}) 2024-09-16T05:26:59.834 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.831+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.835 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.831+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824797387}) 2024-09-16T05:26:59.835 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.831+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874867491}) 2024-09-16T05:26:59.835 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.831+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.835 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.831+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824797387}) 2024-09-16T05:26:59.835 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.831+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2641815924}) 2024-09-16T05:26:59.835 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.835+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.835 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.835+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983315017}) 2024-09-16T05:26:59.835 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.835+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=555552958}) 2024-09-16T05:26:59.835 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.835+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.835 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.835+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983315017}) 2024-09-16T05:26:59.835 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.835+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679793220}) 2024-09-16T05:26:59.835 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.835+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.836 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.835+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=63459929}) 2024-09-16T05:26:59.836 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.835+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2571272579}) 2024-09-16T05:26:59.836 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.835+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.836 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.835+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=63459929}) 2024-09-16T05:26:59.836 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.835+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1742169941}) 2024-09-16T05:26:59.837 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.839+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.837 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.839+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4248516267}) 2024-09-16T05:26:59.837 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.839+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2713284244}) 2024-09-16T05:26:59.837 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.839+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.837 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.839+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4248516267}) 2024-09-16T05:26:59.837 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.839+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1064982145}) 2024-09-16T05:26:59.838 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.839+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.838 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.839+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=62,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17} crc {logm=422508018,mdsmap=2485929294,mgr=3496963760,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=2689991764,mon_config_key=1589051853}) 2024-09-16T05:26:59.839 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.839+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=63,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14} crc {logm=1001254229,mdsmap=1713391792,mgr=3253566819,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=199179091,mon_config_key=1819754182}) 2024-09-16T05:26:59.839 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.839+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.839 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.839+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=62,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17} crc {logm=422508018,mdsmap=2485929294,mgr=3496963760,mgr_command_descs=2107673600,mgr_metadata=1462959313,mgrstat=2689991764,mon_config_key=1589051853}) 2024-09-16T05:26:59.839 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.843+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=56,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=2,osd_metadata=3} crc {logm=735995950,mdsmap=2485929294,mgr=3253566819,mgr_metadata=1462959313,mgrstat=199179091,mon_config_key=1589051853,monmap=3371862038,osd_metadata=718901854}) 2024-09-16T05:26:59.841 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.843+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.841 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.843+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=39,osdmap=51} crc {monmap=3045472550,osd_metadata=487805116,osd_pg_creating=2737448099,osd_snap=619776518,osdmap=2915092975}) 2024-09-16T05:26:59.841 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.843+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=50} crc {monmap=3045472550,osd_metadata=487805116,osd_pg_creating=2737448099,osd_snap=3443869400,osdmap=2373261889}) 2024-09-16T05:26:59.841 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.843+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.841 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.843+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=39,osdmap=51} crc {monmap=3045472550,osd_metadata=487805116,osd_pg_creating=2737448099,osd_snap=619776518,osdmap=2915092975}) 2024-09-16T05:26:59.841 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.843+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=39,osdmap=52} crc {monmap=3371862038,osd_metadata=487805116,osd_pg_creating=2737448099,osd_snap=2561399862,osdmap=2338258063}) 2024-09-16T05:26:59.843 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.847+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.843 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.847+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1365134785}) 2024-09-16T05:26:59.843 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.847+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1122012907}) 2024-09-16T05:26:59.843 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.847+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.843 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.847+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1365134785}) 2024-09-16T05:26:59.844 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.847+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1717400470}) 2024-09-16T05:26:59.846 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.847+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.846 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.847+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2827867724}) 2024-09-16T05:26:59.846 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.847+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3606382759}) 2024-09-16T05:26:59.846 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.847+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.846 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.847+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2827867724}) 2024-09-16T05:26:59.846 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.847+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1843871799}) 2024-09-16T05:26:59.848 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.851+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.848 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.851+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2731624128}) 2024-09-16T05:26:59.848 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.851+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4092311770}) 2024-09-16T05:26:59.848 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.851+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.848 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.851+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2731624128}) 2024-09-16T05:26:59.848 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.851+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=953515356}) 2024-09-16T05:26:59.850 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.851+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.850 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.851+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=47024312}) 2024-09-16T05:26:59.850 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.851+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2760727429}) 2024-09-16T05:26:59.850 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.851+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.851 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.855+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=47024312}) 2024-09-16T05:26:59.851 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.855+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2534046470}) 2024-09-16T05:26:59.853 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.855+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.853 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.855+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2688095326}) 2024-09-16T05:26:59.853 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.855+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=441248753}) 2024-09-16T05:26:59.853 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.855+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.853 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.855+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2688095326}) 2024-09-16T05:26:59.854 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.855+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=409501538}) 2024-09-16T05:26:59.855 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.859+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.855 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.859+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3544547701}) 2024-09-16T05:26:59.855 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.859+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3313332112}) 2024-09-16T05:26:59.856 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.859+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.856 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.859+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3544547701}) 2024-09-16T05:26:59.856 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.859+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3494439665}) 2024-09-16T05:26:59.859 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.863+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.859 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.863+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=988950735}) 2024-09-16T05:26:59.859 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.863+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=123091119}) 2024-09-16T05:26:59.860 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.863+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.860 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.863+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=988950735}) 2024-09-16T05:26:59.860 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.863+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2646091335}) 2024-09-16T05:26:59.863 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.867+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.864 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.867+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2619391345}) 2024-09-16T05:26:59.864 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.867+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3776460582}) 2024-09-16T05:26:59.864 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.867+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.864 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.867+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2619391345}) 2024-09-16T05:26:59.864 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.867+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4094312639}) 2024-09-16T05:26:59.868 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.871+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.868 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.871+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1319765694}) 2024-09-16T05:26:59.868 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.871+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1933151706}) 2024-09-16T05:26:59.868 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.871+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.868 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.871+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1319765694}) 2024-09-16T05:26:59.868 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.871+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2613635818}) 2024-09-16T05:26:59.870 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.871+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.870 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.871+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=50} crc {osdmap=66899355}) 2024-09-16T05:26:59.870 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.871+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=49} crc {osdmap=1684615816}) 2024-09-16T05:26:59.870 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.871+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:26:59.870 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.871+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=50} crc {osdmap=66899355}) 2024-09-16T05:26:59.870 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-09-16T05:26:59.871+0000 7fa8557e9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=48} crc {osdmap=3847197363}) 2024-09-16T05:27:00.703 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:27:00.703 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:27:00.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 10 monclient: get_monmap_and_config 2024-09-16T05:27:00.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 10 monclient: build_initial_monmap 2024-09-16T05:27:00.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 10 monclient: monmap: 2024-09-16T05:27:00.833 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:00.833 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:00.833 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:00.838026+0000 2024-09-16T05:27:00.833 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:00.838026+0000 2024-09-16T05:27:00.833 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:00.833 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:00.833 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:00.833 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:00.833 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:00.833 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:00.833 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:00.833 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:00.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 1 Processor -- start 2024-09-16T05:27:00.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 1 -- start start 2024-09-16T05:27:00.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 10 monclient: init 2024-09-16T05:27:00.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:00.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-16T05:27:00.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8fb0071da0 addr v1:172.21.15.149:6789/0 2024-09-16T05:27:00.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8fb00a1ab0 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:00.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8fb0099cc0 0x7f8fb0072890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:00.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 10 monclient(hunting): picked mon.noname-a con 0x7f8fb0099cc0 addr v2:172.21.15.114:3300/0 2024-09-16T05:27:00.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb0099910 con 0x7f8fb00a1ab0 2024-09-16T05:27:00.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb00a1870 con 0x7f8fb0071da0 2024-09-16T05:27:00.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:00.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f8fb0073e20 con 0x7f8fb0099cc0 2024-09-16T05:27:00.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:00.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb83b1640 10 monclient(hunting): authenticate will time out at 2105.467700s 2024-09-16T05:27:00.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb694e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8fb0099cc0 0x7f8fb0072890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:00.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb694e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8fb0099cc0 0x7f8fb0072890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35088/0 (socket says 172.21.15.114:35088) 2024-09-16T05:27:00.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb594c640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8fb00a1ab0 0x7f8fb013c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37590/0 (socket says 172.21.15.114:37590) 2024-09-16T05:27:00.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb694e640 1 -- 172.21.15.114:0/4009169734 learned_addr learned my addr 172.21.15.114:0/4009169734 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:00.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb694e640 10 monclient(hunting): get_auth_request con 0x7f8fb0099cc0 auth_method 0 2024-09-16T05:27:00.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb694e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:00.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb694e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:00.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.835+0000 7f8fb694e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:00.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb694e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:00.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb694e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:00.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb694e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:00.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb694e640 10 monclient(hunting): handle_auth_done global_id 134151 payload 274 2024-09-16T05:27:00.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb694e640 1 -- 172.21.15.114:0/4009169734 >> v1:172.21.15.149:6789/0 conn(0x7f8fb0071da0 legacy=0x7f8fb0072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:00.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb694e640 1 -- 172.21.15.114:0/4009169734 >> v1:172.21.15.114:6789/0 conn(0x7f8fb00a1ab0 legacy=0x7f8fb013c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb694e640 10 monclient: _finish_hunting 0 2024-09-16T05:27:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb694e640 1 monclient: found mon.noname-a 2024-09-16T05:27:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb694e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb694e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb694e640 1 -- 172.21.15.114:0/4009169734 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb00a1120 con 0x7f8fb0099cc0 2024-09-16T05:27:00.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb694e640 1 --2- 172.21.15.114:0/4009169734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fb0099cc0 0x7f8fb0072890 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8fb009b790 tx=0x7f8fa801c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4fec1134e7a0aa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:00.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb514b640 1 -- 172.21.15.114:0/4009169734 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa80a5750 con 0x7f8fb0099cc0 2024-09-16T05:27:00.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb514b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:00.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb514b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:00.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb514b640 10 monclient: dump: 2024-09-16T05:27:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:00.844 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb514b640 10 monclient: _finish_auth 0 2024-09-16T05:27:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb514b640 10 monclient: _check_auth_tickets 2024-09-16T05:27:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb514b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:00.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 5 monclient: authenticate success, global_id 134151 2024-09-16T05:27:00.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb514b640 1 -- 172.21.15.114:0/4009169734 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fa80a58f0 con 0x7f8fb0099cc0 2024-09-16T05:27:00.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:00.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb514b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:00.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:00.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 4 set_mon_vals no callback set 2024-09-16T05:27:00.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb514b640 1 -- 172.21.15.114:0/4009169734 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa80a5cc0 con 0x7f8fb0099cc0 2024-09-16T05:27:00.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb514b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:00.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb514b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:00.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb514b640 10 monclient: dump: 2024-09-16T05:27:00.846 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:00.846 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:00.846 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:00.846 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:00.846 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:00.846 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:00.846 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:00.846 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:00.846 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:00.846 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:00.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 10 monclient: shutdown 2024-09-16T05:27:00.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:00.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 1 -- 172.21.15.114:0/4009169734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fb0099cc0 msgr2=0x7f8fb0072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:00.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 1 --2- 172.21.15.114:0/4009169734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fb0099cc0 0x7f8fb0072890 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8fb009b790 tx=0x7f8fa801c870 comp rx=0 tx=0).stop 2024-09-16T05:27:00.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 1 -- 172.21.15.114:0/4009169734 shutdown_connections 2024-09-16T05:27:00.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 1 --2- 172.21.15.114:0/4009169734 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fb0099cc0 0x7f8fb0072890 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:00.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 1 -- 172.21.15.114:0/4009169734 >> 172.21.15.114:0/4009169734 conn(0x7f8fb00a1fb0 msgr2=0x7f8fb00a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:00.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 1 -- 172.21.15.114:0/4009169734 shutdown_connections 2024-09-16T05:27:00.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 1 -- 172.21.15.114:0/4009169734 wait complete. 2024-09-16T05:27:00.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 10 monclient: build_initial_monmap 2024-09-16T05:27:00.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 10 monclient: monmap: 2024-09-16T05:27:00.847 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:00.847 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:00.847 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:00.847 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:00.848 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:00.848 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:00.848 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:00.848 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:00.848 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:00.848 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:00.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.839+0000 7f8fb83b1640 1 librados: starting msgr at 2024-09-16T05:27:00.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 1 librados: starting objecter 2024-09-16T05:27:00.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 1 Processor -- start 2024-09-16T05:27:00.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 1 -- start start 2024-09-16T05:27:00.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 1 librados: setting wanted keys 2024-09-16T05:27:00.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 1 librados: calling monclient init 2024-09-16T05:27:00.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient: init 2024-09-16T05:27:00.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:00.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:27:00.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fb0071da0 0x7f8fb0098550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:00.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient(hunting): picked mon.noname-c con 0x7f8fb0071da0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:00.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fb0099cc0 0x7f8fb0098a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:00.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient(hunting): picked mon.noname-a con 0x7f8fb0099cc0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:00.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8fb00a1ab0 0x7f8fb008d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:00.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient(hunting): picked mon.noname-b con 0x7f8fb00a1ab0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:00.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:00.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb0142b60 con 0x7f8fb0099cc0 2024-09-16T05:27:00.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:00.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb01429e0 con 0x7f8fb00a1ab0 2024-09-16T05:27:00.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:00.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8fb0142ce0 con 0x7f8fb0071da0 2024-09-16T05:27:00.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:00.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient(hunting): authenticate will time out at 2105.472706s 2024-09-16T05:27:00.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fb0099cc0 0x7f8fb0098a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:00.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fb0099cc0 0x7f8fb0098a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35096/0 (socket says 172.21.15.114:35096) 2024-09-16T05:27:00.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb614d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fb0071da0 0x7f8fb0098550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:00.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 1 -- 172.21.15.114:0/1091249747 learned_addr learned my addr 172.21.15.114:0/1091249747 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:00.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 10 monclient(hunting): get_auth_request con 0x7f8fb0099cc0 auth_method 0 2024-09-16T05:27:00.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:00.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:00.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:00.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb614d640 10 monclient(hunting): get_auth_request con 0x7f8fb0071da0 auth_method 0 2024-09-16T05:27:00.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb614d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:00.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb614d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:00.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb614d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:00.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:00.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:00.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:00.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb614d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:00.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb614d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:00.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb614d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:00.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 10 monclient(hunting): handle_auth_done global_id 134157 payload 824 2024-09-16T05:27:00.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 1 -- 172.21.15.114:0/1091249747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fb0071da0 msgr2=0x7f8fb0098550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:00.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 1 --2- 172.21.15.114:0/1091249747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fb0071da0 0x7f8fb0098550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:00.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 1 -- 172.21.15.114:0/1091249747 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8fb00a1ab0 msgr2=0x7f8fb008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:00.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 1 --2- 172.21.15.114:0/1091249747 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8fb00a1ab0 0x7f8fb008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:00.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 10 monclient: _finish_hunting 0 2024-09-16T05:27:00.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 1 monclient: found mon.noname-a 2024-09-16T05:27:00.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:00.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:00.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 1 -- 172.21.15.114:0/1091249747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb008ffb0 con 0x7f8fb0099cc0 2024-09-16T05:27:00.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb594c640 1 --2- 172.21.15.114:0/1091249747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fb0099cc0 0x7f8fb0098a90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8fa000c400 tx=0x7f8fa000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:00.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 10 client.?.objecter ms_handle_connect 0x7f8fb0099cc0 2024-09-16T05:27:00.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:00.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 1 -- 172.21.15.114:0/1091249747 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0017070 con 0x7f8fb0099cc0 2024-09-16T05:27:00.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:00.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:00.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 10 monclient: dump: 2024-09-16T05:27:00.853 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:00.853 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:00.853 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:00.853 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:00.853 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:00.853 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:00.853 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:00.854 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:00.854 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:00.854 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:00.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:27:00.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:27:00.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:00.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 1 -- 172.21.15.114:0/1091249747 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fa00074e0 con 0x7f8fb0099cc0 2024-09-16T05:27:00.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 5 monclient: authenticate success, global_id 134157 2024-09-16T05:27:00.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient: _renew_subs 2024-09-16T05:27:00.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:00.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 1 -- 172.21.15.114:0/1091249747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fb013fbf0 con 0x7f8fb0099cc0 2024-09-16T05:27:00.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:00.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 1 -- 172.21.15.114:0/1091249747 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa00059d0 con 0x7f8fb0099cc0 2024-09-16T05:27:00.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:00.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:00.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 10 monclient: dump: 2024-09-16T05:27:00.855 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:00.855 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:00.855 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:00.855 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:00.855 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:00.855 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:00.855 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:00.855 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:00.855 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:00.855 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:00.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb494a640 4 set_mon_vals no callback set 2024-09-16T05:27:00.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 client.134157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:00.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient: _renew_subs 2024-09-16T05:27:00.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:00.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 1 -- 172.21.15.114:0/1091249747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fb008fcd0 con 0x7f8fb0099cc0 2024-09-16T05:27:00.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 1 librados: init done 2024-09-16T05:27:00.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 1 -- 172.21.15.114:0/1091249747 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8fa001a070 con 0x7f8fb0099cc0 2024-09-16T05:27:00.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:00.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:00.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:00.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb83b1640 1 -- 172.21.15.114:0/1091249747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fb0072180 con 0x7f8fb0099cc0 2024-09-16T05:27:00.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 1 --2- 172.21.15.114:0/1091249747 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8f94037840 0x7f8f94039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:00.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 1 -- 172.21.15.114:0/1091249747 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7f8fa0047fb0 con 0x7f8fb0099cc0 2024-09-16T05:27:00.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 10 client.134157.objecter ms_dispatch 0x7f8fb0000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:00.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 3 client.134157.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:00.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 3 client.134157.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:00.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 20 client.134157.objecter dump_active .. 0 homeless 2024-09-16T05:27:00.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb614d640 1 --2- 172.21.15.114:0/1091249747 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8f94037840 0x7f8f94039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:00.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb614d640 10 monclient: get_auth_request con 0x7f8f94037840 auth_method 0 2024-09-16T05:27:00.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fa6ffd640 10 client.134157.objecter ms_handle_connect 0x7f8f94037840 2024-09-16T05:27:00.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.843+0000 7f8fb614d640 1 --2- 172.21.15.114:0/1091249747 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8f94037840 0x7f8f94039d00 secure :-1 s=READY pgs=3628 cs=0 l=1 rev1=1 crypto rx=0x7f8fac007fd0 tx=0x7f8fac007e40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:00.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.847+0000 7f8fa6ffd640 1 -- 172.21.15.114:0/1091249747 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f8fa0010950 con 0x7f8fb0099cc0 2024-09-16T05:27:00.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.847+0000 7f8fa6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:00.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:00.847+0000 7f8fa6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:27:01.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:27:01.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:27:01.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:01.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 1 -- 172.21.15.114:0/1091249747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8fb00a4220 con 0x7f8fb0099cc0 2024-09-16T05:27:01.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fa6ffd640 1 -- 172.21.15.114:0/1091249747 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f8fa0038630 con 0x7f8fb0099cc0 2024-09-16T05:27:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fa6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:27:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fa6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:27:01.058 INFO:teuthology.orchestra.run.smithi114.stdout:{"election_epoch":84,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","modified":"2024-09-16T05:06:54.422314Z","created":"2024-09-16T05:06:54.422314Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:27:01.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 10 librados: watch_flush enter 2024-09-16T05:27:01.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 10 librados: watch_flush exit 2024-09-16T05:27:01.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 20 client.134157.objecter shutdown clearing up homeless session... 2024-09-16T05:27:01.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 10 client.134157.objecter successfully canceled tick 2024-09-16T05:27:01.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 1 -- 172.21.15.114:0/1091249747 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8f94037840 msgr2=0x7f8f94039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 1 --2- 172.21.15.114:0/1091249747 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8f94037840 0x7f8f94039d00 secure :-1 s=READY pgs=3628 cs=0 l=1 rev1=1 crypto rx=0x7f8fac007fd0 tx=0x7f8fac007e40 comp rx=0 tx=0).stop 2024-09-16T05:27:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 10 monclient: shutdown 2024-09-16T05:27:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 1 -- 172.21.15.114:0/1091249747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fb0099cc0 msgr2=0x7f8fb0098a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 1 --2- 172.21.15.114:0/1091249747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fb0099cc0 0x7f8fb0098a90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8fa000c400 tx=0x7f8fa000c8c0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 1 -- 172.21.15.114:0/1091249747 shutdown_connections 2024-09-16T05:27:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 1 --2- 172.21.15.114:0/1091249747 >> v2:172.21.15.149:6812/1185531135 conn(0x7f8f94037840 0x7f8f94039d00 unknown :-1 s=CLOSED pgs=3628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 1 --2- 172.21.15.114:0/1091249747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fb0099cc0 0x7f8fb0098a90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 1 --2- 172.21.15.114:0/1091249747 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8fb00a1ab0 0x7f8fb008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 1 --2- 172.21.15.114:0/1091249747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fb0071da0 0x7f8fb0098550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 1 -- 172.21.15.114:0/1091249747 >> 172.21.15.114:0/1091249747 conn(0x7f8fb00a1fb0 msgr2=0x7f8fb009fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:01.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 1 -- 172.21.15.114:0/1091249747 shutdown_connections 2024-09-16T05:27:01.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 1 -- 172.21.15.114:0/1091249747 wait complete. 2024-09-16T05:27:01.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.059+0000 7f8fb83b1640 1 librados: shutdown 2024-09-16T05:27:01.073 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:27:01.073 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T05:27:01.082 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T05:27:01.082 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:27:01.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.187+0000 7fd7e2d5b640 10 monclient: get_monmap_and_config 2024-09-16T05:27:01.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.187+0000 7fd7e2d5b640 10 monclient: build_initial_monmap 2024-09-16T05:27:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.187+0000 7fd7e2d5b640 10 monclient: monmap: 2024-09-16T05:27:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:01.192216+0000 2024-09-16T05:27:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:01.192216+0000 2024-09-16T05:27:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:01.187 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 1 Processor -- start 2024-09-16T05:27:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 1 -- start start 2024-09-16T05:27:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 10 monclient: init 2024-09-16T05:27:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T05:27:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7dc137850 0x7fd7dc137c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 10 monclient(hunting): picked mon.noname-b con 0x7fd7dc137850 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd7dc072830 0x7fd7dc0a7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 10 monclient(hunting): picked mon.noname-c con 0x7fd7dc072830 addr v2:172.21.15.149:3300/0 2024-09-16T05:27:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7dc0a7c50 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7dc0a94d0 con 0x7fd7dc0a7c50 2024-09-16T05:27:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fd7dc09bf60 con 0x7fd7dc072830 2024-09-16T05:27:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd7dc09c240 con 0x7fd7dc137850 2024-09-16T05:27:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e2d5b640 10 monclient(hunting): authenticate will time out at 2105.821398s 2024-09-16T05:27:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7dc137850 0x7fd7dc137c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7dc137850 0x7fd7dc137c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51686/0 (socket says 172.21.15.114:51686) 2024-09-16T05:27:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 1 -- 172.21.15.114:0/2235043330 learned_addr learned my addr 172.21.15.114:0/2235043330 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 10 monclient(hunting): get_auth_request con 0x7fd7dc137850 auth_method 0 2024-09-16T05:27:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e12f8640 1 --1- 172.21.15.114:0/2235043330 >> v1:172.21.15.114:6789/0 conn(0x7fd7dc0a7c50 0x7fd7dc13e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37594/0 (socket says 172.21.15.114:37594) 2024-09-16T05:27:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 10 monclient(hunting): handle_auth_done global_id 104396 payload 274 2024-09-16T05:27:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7dbfff640 1 --2- 172.21.15.114:0/2235043330 >> v2:172.21.15.149:3300/0 conn(0x7fd7dc072830 0x7fd7dc0a7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 1 -- 172.21.15.114:0/2235043330 >> v2:172.21.15.149:3300/0 conn(0x7fd7dc072830 msgr2=0x7fd7dc0a7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7db7fe640 1 -- 172.21.15.114:0/2235043330 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1510923536 0 0) 0x7fd7dc0a94d0 con 0x7fd7dc0a7c50 2024-09-16T05:27:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 1 --2- 172.21.15.114:0/2235043330 >> v2:172.21.15.149:3300/0 conn(0x7fd7dc072830 0x7fd7dc0a7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 1 -- 172.21.15.114:0/2235043330 >> v1:172.21.15.114:6789/0 conn(0x7fd7dc0a7c50 legacy=0x7fd7dc13e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 10 monclient: _finish_hunting 0 2024-09-16T05:27:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 1 monclient: found mon.noname-b 2024-09-16T05:27:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:27:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 1 -- 172.21.15.114:0/2235043330 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7dc09c520 con 0x7fd7dc137850 2024-09-16T05:27:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7dbfff640 10 monclient: get_auth_request con 0x7fd7dc072830 auth_method 0 2024-09-16T05:27:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7dbfff640 1 --2- 172.21.15.114:0/2235043330 >> v2:172.21.15.149:3300/0 conn(0x7fd7dc072830 0x7fd7dc0a7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:27:01.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7db7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:27:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.191+0000 7fd7e0af7640 1 --2- 172.21.15.114:0/2235043330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7dc137850 0x7fd7dc137c30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009870 tx=0x7fd7d401c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a187c3e82a810bf1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7db7fe640 1 -- 172.21.15.114:0/2235043330 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d40b0070 con 0x7fd7dc137850 2024-09-16T05:27:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7db7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:27:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7db7fe640 10 monclient: dump: 2024-09-16T05:27:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:01.200 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.200 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:01.200 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:01.200 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:01.200 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7db7fe640 10 monclient: _finish_auth 0 2024-09-16T05:27:01.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7db7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:27:01.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:01.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 5 monclient: authenticate success, global_id 104396 2024-09-16T05:27:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7db7fe640 1 -- 172.21.15.114:0/2235043330 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7d40ab600 con 0x7fd7dc137850 2024-09-16T05:27:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7db7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7db7fe640 1 -- 172.21.15.114:0/2235043330 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d40ab9d0 con 0x7fd7dc137850 2024-09-16T05:27:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 4 set_mon_vals no callback set 2024-09-16T05:27:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7db7fe640 10 monclient: dump: 2024-09-16T05:27:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:01.202 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient: shutdown 2024-09-16T05:27:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 -- 172.21.15.114:0/2235043330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7dc137850 msgr2=0x7fd7dc137c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 --2- 172.21.15.114:0/2235043330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7dc137850 0x7fd7dc137c30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009870 tx=0x7fd7d401c4b0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 -- 172.21.15.114:0/2235043330 shutdown_connections 2024-09-16T05:27:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 --2- 172.21.15.114:0/2235043330 >> v2:172.21.15.149:3300/0 conn(0x7fd7dc072830 0x7fd7dc0a7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 --2- 172.21.15.114:0/2235043330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7dc137850 0x7fd7dc137c30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 -- 172.21.15.114:0/2235043330 >> 172.21.15.114:0/2235043330 conn(0x7fd7dc0a9d10 msgr2=0x7fd7dc0aa120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 -- 172.21.15.114:0/2235043330 shutdown_connections 2024-09-16T05:27:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 -- 172.21.15.114:0/2235043330 wait complete. 2024-09-16T05:27:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient: build_initial_monmap 2024-09-16T05:27:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient: monmap: 2024-09-16T05:27:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:01.203 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:01.203 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 librados: starting msgr at 2024-09-16T05:27:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 librados: starting objecter 2024-09-16T05:27:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 Processor -- start 2024-09-16T05:27:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 -- start start 2024-09-16T05:27:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 librados: setting wanted keys 2024-09-16T05:27:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 librados: calling monclient init 2024-09-16T05:27:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient: init 2024-09-16T05:27:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:27:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7dc072830 0x7fd7dc1340a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient(hunting): picked mon.noname-a con 0x7fd7dc072830 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7dc0a7c50 0x7fd7dc1345e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient(hunting): picked mon.noname-b con 0x7fd7dc0a7c50 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:01.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7dc137850 0x7fd7dc134b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient(hunting): picked mon.noname-c con 0x7fd7dc137850 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:01.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc1424c0 con 0x7fd7dc072830 2024-09-16T05:27:01.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc142340 con 0x7fd7dc0a7c50 2024-09-16T05:27:01.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd7dc142640 con 0x7fd7dc137850 2024-09-16T05:27:01.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:01.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient(hunting): authenticate will time out at 2105.826086s 2024-09-16T05:27:01.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7dc072830 0x7fd7dc1340a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7dc072830 0x7fd7dc1340a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35108/0 (socket says 172.21.15.114:35108) 2024-09-16T05:27:01.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 1 -- 172.21.15.114:0/2518679271 learned_addr learned my addr 172.21.15.114:0/2518679271 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:01.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 10 monclient(hunting): get_auth_request con 0x7fd7dc072830 auth_method 0 2024-09-16T05:27:01.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:01.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:01.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:01.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:01.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:01.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e12f8640 1 --2- 172.21.15.114:0/2518679271 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7dc137850 0x7fd7dc134b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:01.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e12f8640 10 monclient(hunting): get_auth_request con 0x7fd7dc137850 auth_method 0 2024-09-16T05:27:01.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e12f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:01.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e12f8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:01.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e12f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:01.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e12f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:01.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e12f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:01.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e12f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:01.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7dbfff640 1 --2- 172.21.15.114:0/2518679271 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7dc0a7c50 0x7fd7dc1345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 10 monclient(hunting): handle_auth_done global_id 134163 payload 824 2024-09-16T05:27:01.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 1 -- 172.21.15.114:0/2518679271 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7dc137850 msgr2=0x7fd7dc134b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 1 --2- 172.21.15.114:0/2518679271 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7dc137850 0x7fd7dc134b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 1 -- 172.21.15.114:0/2518679271 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7dc0a7c50 msgr2=0x7fd7dc1345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 1 --2- 172.21.15.114:0/2518679271 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7dc0a7c50 0x7fd7dc1345e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 10 monclient: _finish_hunting 0 2024-09-16T05:27:01.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 1 monclient: found mon.noname-a 2024-09-16T05:27:01.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:01.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:01.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 1 -- 172.21.15.114:0/2518679271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7dc073e20 con 0x7fd7dc072830 2024-09-16T05:27:01.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7dbfff640 10 monclient: get_auth_request con 0x7fd7dc0a7c50 auth_method 0 2024-09-16T05:27:01.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7dbfff640 1 --2- 172.21.15.114:0/2518679271 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7dc0a7c50 0x7fd7dc1345e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:27:01.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e0af7640 1 --2- 172.21.15.114:0/2518679271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7dc072830 0x7fd7dc1340a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd7d401a0e0 tx=0x7fd7d4019250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:01.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 10 client.?.objecter ms_handle_connect 0x7fd7dc072830 2024-09-16T05:27:01.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:01.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 1 -- 172.21.15.114:0/2518679271 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d40b0050 con 0x7fd7dc072830 2024-09-16T05:27:01.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:01.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:01.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 10 monclient: dump: 2024-09-16T05:27:01.208 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:01.208 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:01.208 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.208 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.208 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:01.209 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.209 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:01.209 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:01.209 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:01.209 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 10 monclient: _finish_auth 0 2024-09-16T05:27:01.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 10 monclient: _check_auth_tickets 2024-09-16T05:27:01.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:01.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 5 monclient: authenticate success, global_id 134163 2024-09-16T05:27:01.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 1 -- 172.21.15.114:0/2518679271 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7d40b1040 con 0x7fd7dc072830 2024-09-16T05:27:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient: _renew_subs 2024-09-16T05:27:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 -- 172.21.15.114:0/2518679271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7dc13bb10 con 0x7fd7dc072830 2024-09-16T05:27:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 1 -- 172.21.15.114:0/2518679271 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d40b5850 con 0x7fd7dc072830 2024-09-16T05:27:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 10 monclient: dump: 2024-09-16T05:27:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.211 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:01.211 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:01.211 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:01.211 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7daffd640 4 set_mon_vals no callback set 2024-09-16T05:27:01.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 client.134163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:01.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient: _renew_subs 2024-09-16T05:27:01.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:01.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 -- 172.21.15.114:0/2518679271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7dc13de00 con 0x7fd7dc072830 2024-09-16T05:27:01.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 librados: init done 2024-09-16T05:27:01.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:01.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:01.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:01.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 1 -- 172.21.15.114:0/2518679271 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd7d40be020 con 0x7fd7dc072830 2024-09-16T05:27:01.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7e2d5b640 1 -- 172.21.15.114:0/2518679271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7dc0a7710 con 0x7fd7dc072830 2024-09-16T05:27:01.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.195+0000 7fd7d97fa640 1 --2- 172.21.15.114:0/2518679271 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd7bc037840 0x7fd7bc039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.199+0000 7fd7d97fa640 1 -- 172.21.15.114:0/2518679271 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7fd7d40e2a80 con 0x7fd7dc072830 2024-09-16T05:27:01.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.199+0000 7fd7d97fa640 10 client.134163.objecter ms_dispatch 0x7fd7dc000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:01.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.199+0000 7fd7d97fa640 3 client.134163.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:01.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.199+0000 7fd7d97fa640 3 client.134163.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:01.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.199+0000 7fd7d97fa640 20 client.134163.objecter dump_active .. 0 homeless 2024-09-16T05:27:01.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.199+0000 7fd7dbfff640 1 --2- 172.21.15.114:0/2518679271 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd7bc037840 0x7fd7bc039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.199+0000 7fd7dbfff640 10 monclient: get_auth_request con 0x7fd7bc037840 auth_method 0 2024-09-16T05:27:01.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.199+0000 7fd7dbfff640 1 --2- 172.21.15.114:0/2518679271 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd7bc037840 0x7fd7bc039d00 secure :-1 s=READY pgs=3629 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc13b710 tx=0x7fd7c4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:01.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.199+0000 7fd7d97fa640 10 client.134163.objecter ms_handle_connect 0x7fd7bc037840 2024-09-16T05:27:01.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.203+0000 7fd7d97fa640 1 -- 172.21.15.114:0/2518679271 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd7d4016130 con 0x7fd7dc072830 2024-09-16T05:27:01.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.203+0000 7fd7d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:01.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.203+0000 7fd7d97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:27:01.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.351+0000 7fd7e2d5b640 1 -- 172.21.15.114:0/2518679271 --> v2:172.21.15.149:6812/1185531135 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd7dc0a5670 con 0x7fd7bc037840 2024-09-16T05:27:01.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.359+0000 7fd7d97fa640 1 -- 172.21.15.114:0/2518679271 <== mgr.4108 v2:172.21.15.149:6812/1185531135 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114079 (secure 0 0 0) 0x7fd7dc0a5670 con 0x7fd7bc037840 2024-09-16T05:27:01.357 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:27:01.358 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-16T05:27:01.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.359+0000 7fd7e2d5b640 10 librados: watch_flush enter 2024-09-16T05:27:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.359+0000 7fd7e2d5b640 10 librados: watch_flush exit 2024-09-16T05:27:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.359+0000 7fd7e2d5b640 20 client.134163.objecter shutdown clearing up homeless session... 2024-09-16T05:27:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.359+0000 7fd7e2d5b640 10 client.134163.objecter successfully canceled tick 2024-09-16T05:27:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.359+0000 7fd7e2d5b640 1 -- 172.21.15.114:0/2518679271 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd7bc037840 msgr2=0x7fd7bc039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.359+0000 7fd7e2d5b640 1 --2- 172.21.15.114:0/2518679271 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd7bc037840 0x7fd7bc039d00 secure :-1 s=READY pgs=3629 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc13b710 tx=0x7fd7c4008040 comp rx=0 tx=0).stop 2024-09-16T05:27:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.359+0000 7fd7e2d5b640 10 monclient: shutdown 2024-09-16T05:27:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.359+0000 7fd7e2d5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.359+0000 7fd7e2d5b640 1 -- 172.21.15.114:0/2518679271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7dc072830 msgr2=0x7fd7dc1340a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.359+0000 7fd7e2d5b640 1 --2- 172.21.15.114:0/2518679271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7dc072830 0x7fd7dc1340a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd7d401a0e0 tx=0x7fd7d4019250 comp rx=0 tx=0).stop 2024-09-16T05:27:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.363+0000 7fd7e2d5b640 1 -- 172.21.15.114:0/2518679271 shutdown_connections 2024-09-16T05:27:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.363+0000 7fd7e2d5b640 1 --2- 172.21.15.114:0/2518679271 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd7bc037840 0x7fd7bc039d00 unknown :-1 s=CLOSED pgs=3629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.363+0000 7fd7e2d5b640 1 --2- 172.21.15.114:0/2518679271 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7dc137850 0x7fd7dc134b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.363+0000 7fd7e2d5b640 1 --2- 172.21.15.114:0/2518679271 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7dc0a7c50 0x7fd7dc1345e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.363+0000 7fd7e2d5b640 1 --2- 172.21.15.114:0/2518679271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7dc072830 0x7fd7dc1340a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.363+0000 7fd7e2d5b640 1 -- 172.21.15.114:0/2518679271 >> 172.21.15.114:0/2518679271 conn(0x7fd7dc0a9d10 msgr2=0x7fd7dc0a4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.363+0000 7fd7e2d5b640 1 -- 172.21.15.114:0/2518679271 shutdown_connections 2024-09-16T05:27:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.363+0000 7fd7e2d5b640 1 -- 172.21.15.114:0/2518679271 wait complete. 2024-09-16T05:27:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.363+0000 7fd7e2d5b640 1 librados: shutdown 2024-09-16T05:27:01.373 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1266,"stamp":"2024-09-16T05:27:00.512220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":464520,"kb_used_data":60696,"kb_used_omap":9,"kb_used_meta":403574,"kb_avail":565766520,"statfs":{"total":579820584960,"available":579344916480,"internally_reserved":0,"allocated":62152704,"data_stored":32273216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":413260449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003811"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021853+0000","last_change":"2024-09-16T05:08:51.596823+0000","last_active":"2024-09-16T05:25:06.021853+0000","last_peered":"2024-09-16T05:25:06.021853+0000","last_clean":"2024-09-16T05:25:06.021853+0000","last_became_active":"2024-09-16T05:08:51.571632+0000","last_became_peered":"2024-09-16T05:08:51.571632+0000","last_unstale":"2024-09-16T05:25:06.021853+0000","last_undegraded":"2024-09-16T05:25:06.021853+0000","last_fullsized":"2024-09-16T05:25:06.021853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:16.107411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1324,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411274+0000","last_change":"2024-09-16T05:08:51.611318+0000","last_active":"2024-09-16T05:25:05.411274+0000","last_peered":"2024-09-16T05:25:05.411274+0000","last_clean":"2024-09-16T05:25:05.411274+0000","last_became_active":"2024-09-16T05:08:51.585360+0000","last_became_peered":"2024-09-16T05:08:51.585360+0000","last_unstale":"2024-09-16T05:25:05.411274+0000","last_undegraded":"2024-09-16T05:25:05.411274+0000","last_fullsized":"2024-09-16T05:25:05.411274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:19:58.506693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021814+0000","last_change":"2024-09-16T05:08:51.532106+0000","last_active":"2024-09-16T05:25:06.021814+0000","last_peered":"2024-09-16T05:25:06.021814+0000","last_clean":"2024-09-16T05:25:06.021814+0000","last_became_active":"2024-09-16T05:08:51.519558+0000","last_became_peered":"2024-09-16T05:08:51.519558+0000","last_unstale":"2024-09-16T05:25:06.021814+0000","last_undegraded":"2024-09-16T05:25:06.021814+0000","last_fullsized":"2024-09-16T05:25:06.021814+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:52:54.764452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1316,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.441285+0000","last_change":"2024-09-16T05:08:51.705649+0000","last_active":"2024-09-16T05:25:05.441285+0000","last_peered":"2024-09-16T05:25:05.441285+0000","last_clean":"2024-09-16T05:25:05.441285+0000","last_became_active":"2024-09-16T05:08:51.689931+0000","last_became_peered":"2024-09-16T05:08:51.689931+0000","last_unstale":"2024-09-16T05:25:05.441285+0000","last_undegraded":"2024-09-16T05:25:05.441285+0000","last_fullsized":"2024-09-16T05:25:05.441285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:31.971156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522829+0000","last_change":"2024-09-16T05:08:51.564493+0000","last_active":"2024-09-16T05:25:05.522829+0000","last_peered":"2024-09-16T05:25:05.522829+0000","last_clean":"2024-09-16T05:25:05.522829+0000","last_became_active":"2024-09-16T05:08:51.546851+0000","last_became_peered":"2024-09-16T05:08:51.546851+0000","last_unstale":"2024-09-16T05:25:05.522829+0000","last_undegraded":"2024-09-16T05:25:05.522829+0000","last_fullsized":"2024-09-16T05:25:05.522829+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:23.731218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.463123+0000","last_change":"2024-09-16T05:08:51.617848+0000","last_active":"2024-09-16T05:25:04.463123+0000","last_peered":"2024-09-16T05:25:04.463123+0000","last_clean":"2024-09-16T05:25:04.463123+0000","last_became_active":"2024-09-16T05:08:51.605843+0000","last_became_peered":"2024-09-16T05:08:51.605843+0000","last_unstale":"2024-09-16T05:25:04.463123+0000","last_undegraded":"2024-09-16T05:25:04.463123+0000","last_fullsized":"2024-09-16T05:25:04.463123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:24:14.376183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1317,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.441115+0000","last_change":"2024-09-16T05:08:51.615740+0000","last_active":"2024-09-16T05:25:05.441115+0000","last_peered":"2024-09-16T05:25:05.441115+0000","last_clean":"2024-09-16T05:25:05.441115+0000","last_became_active":"2024-09-16T05:08:51.609820+0000","last_became_peered":"2024-09-16T05:08:51.609820+0000","last_unstale":"2024-09-16T05:25:05.441115+0000","last_undegraded":"2024-09-16T05:25:05.441115+0000","last_fullsized":"2024-09-16T05:25:05.441115+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:35:22.111340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1322,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021670+0000","last_change":"2024-09-16T05:08:51.603805+0000","last_active":"2024-09-16T05:25:06.021670+0000","last_peered":"2024-09-16T05:25:06.021670+0000","last_clean":"2024-09-16T05:25:06.021670+0000","last_became_active":"2024-09-16T05:08:51.567268+0000","last_became_peered":"2024-09-16T05:08:51.567268+0000","last_unstale":"2024-09-16T05:25:06.021670+0000","last_undegraded":"2024-09-16T05:25:06.021670+0000","last_fullsized":"2024-09-16T05:25:06.021670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:43:35.723700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1325,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411521+0000","last_change":"2024-09-16T05:08:51.640189+0000","last_active":"2024-09-16T05:25:05.411521+0000","last_peered":"2024-09-16T05:25:05.411521+0000","last_clean":"2024-09-16T05:25:05.411521+0000","last_became_active":"2024-09-16T05:08:51.634768+0000","last_became_peered":"2024-09-16T05:08:51.634768+0000","last_unstale":"2024-09-16T05:25:05.411521+0000","last_undegraded":"2024-09-16T05:25:05.411521+0000","last_fullsized":"2024-09-16T05:25:05.411521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:52:08.194663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1323,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.735797+0000","last_change":"2024-09-16T05:08:51.589983+0000","last_active":"2024-09-16T05:25:04.735797+0000","last_peered":"2024-09-16T05:25:04.735797+0000","last_clean":"2024-09-16T05:25:04.735797+0000","last_became_active":"2024-09-16T05:08:51.589818+0000","last_became_peered":"2024-09-16T05:08:51.589818+0000","last_unstale":"2024-09-16T05:25:04.735797+0000","last_undegraded":"2024-09-16T05:25:04.735797+0000","last_fullsized":"2024-09-16T05:25:04.735797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:55:59.967412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.465088+0000","last_change":"2024-09-16T05:08:51.609659+0000","last_active":"2024-09-16T05:25:04.465088+0000","last_peered":"2024-09-16T05:25:04.465088+0000","last_clean":"2024-09-16T05:25:04.465088+0000","last_became_active":"2024-09-16T05:08:51.605719+0000","last_became_peered":"2024-09-16T05:08:51.605719+0000","last_unstale":"2024-09-16T05:25:04.465088+0000","last_undegraded":"2024-09-16T05:25:04.465088+0000","last_fullsized":"2024-09-16T05:25:04.465088+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:43:56.008078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1325,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411040+0000","last_change":"2024-09-16T05:08:51.630514+0000","last_active":"2024-09-16T05:25:05.411040+0000","last_peered":"2024-09-16T05:25:05.411040+0000","last_clean":"2024-09-16T05:25:05.411040+0000","last_became_active":"2024-09-16T05:08:51.611801+0000","last_became_peered":"2024-09-16T05:08:51.611801+0000","last_unstale":"2024-09-16T05:25:05.411040+0000","last_undegraded":"2024-09-16T05:25:05.411040+0000","last_fullsized":"2024-09-16T05:25:05.411040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:46.255904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522975+0000","last_change":"2024-09-16T05:08:51.605439+0000","last_active":"2024-09-16T05:25:05.522975+0000","last_peered":"2024-09-16T05:25:05.522975+0000","last_clean":"2024-09-16T05:25:05.522975+0000","last_became_active":"2024-09-16T05:08:51.586611+0000","last_became_peered":"2024-09-16T05:08:51.586611+0000","last_unstale":"2024-09-16T05:25:05.522975+0000","last_undegraded":"2024-09-16T05:25:05.522975+0000","last_fullsized":"2024-09-16T05:25:05.522975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:39:46.639778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":1492,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.410709+0000","last_change":"2024-09-16T05:07:39.580660+0000","last_active":"2024-09-16T05:25:05.410709+0000","last_peered":"2024-09-16T05:25:05.410709+0000","last_clean":"2024-09-16T05:25:05.410709+0000","last_became_active":"2024-09-16T05:07:39.576756+0000","last_became_peered":"2024-09-16T05:07:39.576756+0000","last_unstale":"2024-09-16T05:25:05.410709+0000","last_undegraded":"2024-09-16T05:25:05.410709+0000","last_fullsized":"2024-09-16T05:25:05.410709+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_clean_scrub_stamp":"2024-09-16T05:07:38.167879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:57.060906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1342,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411039+0000","last_change":"2024-09-16T05:07:48.267201+0000","last_active":"2024-09-16T05:25:05.411039+0000","last_peered":"2024-09-16T05:25:05.411039+0000","last_clean":"2024-09-16T05:25:05.411039+0000","last_became_active":"2024-09-16T05:07:46.246409+0000","last_became_peered":"2024-09-16T05:07:46.246409+0000","last_unstale":"2024-09-16T05:25:05.411039+0000","last_undegraded":"2024-09-16T05:25:05.411039+0000","last_fullsized":"2024-09-16T05:25:05.411039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:13.808827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010413149999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1323,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.735867+0000","last_change":"2024-09-16T05:08:51.582564+0000","last_active":"2024-09-16T05:25:04.735867+0000","last_peered":"2024-09-16T05:25:04.735867+0000","last_clean":"2024-09-16T05:25:04.735867+0000","last_became_active":"2024-09-16T05:08:51.582424+0000","last_became_peered":"2024-09-16T05:08:51.582424+0000","last_unstale":"2024-09-16T05:25:04.735867+0000","last_undegraded":"2024-09-16T05:25:04.735867+0000","last_fullsized":"2024-09-16T05:25:04.735867+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:43.882076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1333,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.441020+0000","last_change":"2024-09-16T05:07:48.266504+0000","last_active":"2024-09-16T05:25:05.441020+0000","last_peered":"2024-09-16T05:25:05.441020+0000","last_clean":"2024-09-16T05:25:05.441020+0000","last_became_active":"2024-09-16T05:07:46.254322+0000","last_became_peered":"2024-09-16T05:07:46.254322+0000","last_unstale":"2024-09-16T05:25:05.441020+0000","last_undegraded":"2024-09-16T05:25:05.441020+0000","last_fullsized":"2024-09-16T05:25:05.441020+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:00.175590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093680599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522665+0000","last_change":"2024-09-16T05:08:51.549772+0000","last_active":"2024-09-16T05:25:05.522665+0000","last_peered":"2024-09-16T05:25:05.522665+0000","last_clean":"2024-09-16T05:25:05.522665+0000","last_became_active":"2024-09-16T05:08:51.539573+0000","last_became_peered":"2024-09-16T05:08:51.539573+0000","last_unstale":"2024-09-16T05:25:05.522665+0000","last_undegraded":"2024-09-16T05:25:05.522665+0000","last_fullsized":"2024-09-16T05:25:05.522665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:10:54.876819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1362,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.462909+0000","last_change":"2024-09-16T05:07:48.263513+0000","last_active":"2024-09-16T05:25:04.462909+0000","last_peered":"2024-09-16T05:25:04.462909+0000","last_clean":"2024-09-16T05:25:04.462909+0000","last_became_active":"2024-09-16T05:07:46.252254+0000","last_became_peered":"2024-09-16T05:07:46.252254+0000","last_unstale":"2024-09-16T05:25:04.462909+0000","last_undegraded":"2024-09-16T05:25:04.462909+0000","last_fullsized":"2024-09-16T05:25:04.462909+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:33.736560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081560699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.465371+0000","last_change":"2024-09-16T05:08:51.542756+0000","last_active":"2024-09-16T05:25:04.465371+0000","last_peered":"2024-09-16T05:25:04.465371+0000","last_clean":"2024-09-16T05:25:04.465371+0000","last_became_active":"2024-09-16T05:08:51.541976+0000","last_became_peered":"2024-09-16T05:08:51.541976+0000","last_unstale":"2024-09-16T05:25:04.465371+0000","last_undegraded":"2024-09-16T05:25:04.465371+0000","last_fullsized":"2024-09-16T05:25:04.465371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:48:46.043655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":1347,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522619+0000","last_change":"2024-09-16T05:07:48.274329+0000","last_active":"2024-09-16T05:25:05.522619+0000","last_peered":"2024-09-16T05:25:05.522619+0000","last_clean":"2024-09-16T05:25:05.522619+0000","last_became_active":"2024-09-16T05:07:46.257118+0000","last_became_peered":"2024-09-16T05:07:46.257118+0000","last_unstale":"2024-09-16T05:25:05.522619+0000","last_undegraded":"2024-09-16T05:25:05.522619+0000","last_fullsized":"2024-09-16T05:25:05.522619+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:35.932772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020216650000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1323,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021532+0000","last_change":"2024-09-16T05:08:51.506142+0000","last_active":"2024-09-16T05:25:06.021532+0000","last_peered":"2024-09-16T05:25:06.021532+0000","last_clean":"2024-09-16T05:25:06.021532+0000","last_became_active":"2024-09-16T05:08:51.491760+0000","last_became_peered":"2024-09-16T05:08:51.491760+0000","last_unstale":"2024-09-16T05:25:06.021532+0000","last_undegraded":"2024-09-16T05:25:06.021532+0000","last_fullsized":"2024-09-16T05:25:06.021532+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:04:43.526278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1339,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.735843+0000","last_change":"2024-09-16T05:07:48.352028+0000","last_active":"2024-09-16T05:25:04.735843+0000","last_peered":"2024-09-16T05:25:04.735843+0000","last_clean":"2024-09-16T05:25:04.735843+0000","last_became_active":"2024-09-16T05:07:46.255633+0000","last_became_peered":"2024-09-16T05:07:46.255633+0000","last_unstale":"2024-09-16T05:25:04.735843+0000","last_undegraded":"2024-09-16T05:25:04.735843+0000","last_fullsized":"2024-09-16T05:25:04.735843+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:07.079219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000883379,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"21'7","reported_seq":1359,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.462678+0000","last_change":"2024-09-16T05:08:51.617951+0000","last_active":"2024-09-16T05:25:04.462678+0000","last_peered":"2024-09-16T05:25:04.462678+0000","last_clean":"2024-09-16T05:25:04.462678+0000","last_became_active":"2024-09-16T05:08:51.614284+0000","last_became_peered":"2024-09-16T05:08:51.614284+0000","last_unstale":"2024-09-16T05:25:04.462678+0000","last_undegraded":"2024-09-16T05:25:04.462678+0000","last_fullsized":"2024-09-16T05:25:04.462678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:18.960138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1341,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411163+0000","last_change":"2024-09-16T05:07:48.267372+0000","last_active":"2024-09-16T05:25:05.411163+0000","last_peered":"2024-09-16T05:25:05.411163+0000","last_clean":"2024-09-16T05:25:05.411163+0000","last_became_active":"2024-09-16T05:07:46.243891+0000","last_became_peered":"2024-09-16T05:07:46.243891+0000","last_unstale":"2024-09-16T05:25:05.411163+0000","last_undegraded":"2024-09-16T05:25:05.411163+0000","last_fullsized":"2024-09-16T05:25:05.411163+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:48:25.061723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011829709999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1325,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.410924+0000","last_change":"2024-09-16T05:08:51.634776+0000","last_active":"2024-09-16T05:25:05.410924+0000","last_peered":"2024-09-16T05:25:05.410924+0000","last_clean":"2024-09-16T05:25:05.410924+0000","last_became_active":"2024-09-16T05:08:51.632213+0000","last_became_peered":"2024-09-16T05:08:51.632213+0000","last_unstale":"2024-09-16T05:25:05.410924+0000","last_undegraded":"2024-09-16T05:25:05.410924+0000","last_fullsized":"2024-09-16T05:25:05.410924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:47:43.409907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1337,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021856+0000","last_change":"2024-09-16T05:07:48.275545+0000","last_active":"2024-09-16T05:25:06.021856+0000","last_peered":"2024-09-16T05:25:06.021856+0000","last_clean":"2024-09-16T05:25:06.021856+0000","last_became_active":"2024-09-16T05:07:46.252207+0000","last_became_peered":"2024-09-16T05:07:46.252207+0000","last_unstale":"2024-09-16T05:25:06.021856+0000","last_undegraded":"2024-09-16T05:25:06.021856+0000","last_fullsized":"2024-09-16T05:25:06.021856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:36:08.378166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073333700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522617+0000","last_change":"2024-09-16T05:08:51.499333+0000","last_active":"2024-09-16T05:25:05.522617+0000","last_peered":"2024-09-16T05:25:05.522617+0000","last_clean":"2024-09-16T05:25:05.522617+0000","last_became_active":"2024-09-16T05:08:51.486807+0000","last_became_peered":"2024-09-16T05:08:51.486807+0000","last_unstale":"2024-09-16T05:25:05.522617+0000","last_undegraded":"2024-09-16T05:25:05.522617+0000","last_fullsized":"2024-09-16T05:25:05.522617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:43.084461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1340,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411030+0000","last_change":"2024-09-16T05:07:48.267914+0000","last_active":"2024-09-16T05:25:05.411030+0000","last_peered":"2024-09-16T05:25:05.411030+0000","last_clean":"2024-09-16T05:25:05.411030+0000","last_became_active":"2024-09-16T05:07:46.254034+0000","last_became_peered":"2024-09-16T05:07:46.254034+0000","last_unstale":"2024-09-16T05:25:05.411030+0000","last_undegraded":"2024-09-16T05:25:05.411030+0000","last_fullsized":"2024-09-16T05:25:05.411030+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:57:37.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014490219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021562+0000","last_change":"2024-09-16T05:08:51.548570+0000","last_active":"2024-09-16T05:25:06.021562+0000","last_peered":"2024-09-16T05:25:06.021562+0000","last_clean":"2024-09-16T05:25:06.021562+0000","last_became_active":"2024-09-16T05:08:51.532685+0000","last_became_peered":"2024-09-16T05:08:51.532685+0000","last_unstale":"2024-09-16T05:25:06.021562+0000","last_undegraded":"2024-09-16T05:25:06.021562+0000","last_fullsized":"2024-09-16T05:25:06.021562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:14:25.488128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021522+0000","last_change":"2024-09-16T05:08:51.604786+0000","last_active":"2024-09-16T05:25:06.021522+0000","last_peered":"2024-09-16T05:25:06.021522+0000","last_clean":"2024-09-16T05:25:06.021522+0000","last_became_active":"2024-09-16T05:08:51.567861+0000","last_became_peered":"2024-09-16T05:08:51.567861+0000","last_unstale":"2024-09-16T05:25:06.021522+0000","last_undegraded":"2024-09-16T05:25:06.021522+0000","last_fullsized":"2024-09-16T05:25:06.021522+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:48:14.930571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1345,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.463346+0000","last_change":"2024-09-16T05:08:51.563783+0000","last_active":"2024-09-16T05:25:04.463346+0000","last_peered":"2024-09-16T05:25:04.463346+0000","last_clean":"2024-09-16T05:25:04.463346+0000","last_became_active":"2024-09-16T05:08:51.563420+0000","last_became_peered":"2024-09-16T05:08:51.563420+0000","last_unstale":"2024-09-16T05:25:04.463346+0000","last_undegraded":"2024-09-16T05:25:04.463346+0000","last_fullsized":"2024-09-16T05:25:04.463346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:52.527569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1328,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522885+0000","last_change":"2024-09-16T05:08:51.513817+0000","last_active":"2024-09-16T05:25:05.522885+0000","last_peered":"2024-09-16T05:25:05.522885+0000","last_clean":"2024-09-16T05:25:05.522885+0000","last_became_active":"2024-09-16T05:08:51.503720+0000","last_became_peered":"2024-09-16T05:08:51.503720+0000","last_unstale":"2024-09-16T05:25:05.522885+0000","last_undegraded":"2024-09-16T05:25:05.522885+0000","last_fullsized":"2024-09-16T05:25:05.522885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:31.884629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021857+0000","last_change":"2024-09-16T05:08:51.505598+0000","last_active":"2024-09-16T05:25:06.021857+0000","last_peered":"2024-09-16T05:25:06.021857+0000","last_clean":"2024-09-16T05:25:06.021857+0000","last_became_active":"2024-09-16T05:08:51.494038+0000","last_became_peered":"2024-09-16T05:08:51.494038+0000","last_unstale":"2024-09-16T05:25:06.021857+0000","last_undegraded":"2024-09-16T05:25:06.021857+0000","last_fullsized":"2024-09-16T05:25:06.021857+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:39:28.303932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1324,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411448+0000","last_change":"2024-09-16T05:08:51.680865+0000","last_active":"2024-09-16T05:25:05.411448+0000","last_peered":"2024-09-16T05:25:05.411448+0000","last_clean":"2024-09-16T05:25:05.411448+0000","last_became_active":"2024-09-16T05:08:51.680588+0000","last_became_peered":"2024-09-16T05:08:51.680588+0000","last_unstale":"2024-09-16T05:25:05.411448+0000","last_undegraded":"2024-09-16T05:25:05.411448+0000","last_fullsized":"2024-09-16T05:25:05.411448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:52.176771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.463756+0000","last_change":"2024-09-16T05:08:51.490785+0000","last_active":"2024-09-16T05:25:04.463756+0000","last_peered":"2024-09-16T05:25:04.463756+0000","last_clean":"2024-09-16T05:25:04.463756+0000","last_became_active":"2024-09-16T05:08:51.480748+0000","last_became_peered":"2024-09-16T05:08:51.480748+0000","last_unstale":"2024-09-16T05:25:04.463756+0000","last_undegraded":"2024-09-16T05:25:04.463756+0000","last_fullsized":"2024-09-16T05:25:04.463756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:21:24.554888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1325,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411671+0000","last_change":"2024-09-16T05:08:51.623309+0000","last_active":"2024-09-16T05:25:05.411671+0000","last_peered":"2024-09-16T05:25:05.411671+0000","last_clean":"2024-09-16T05:25:05.411671+0000","last_became_active":"2024-09-16T05:08:51.622941+0000","last_became_peered":"2024-09-16T05:08:51.622941+0000","last_unstale":"2024-09-16T05:25:05.411671+0000","last_undegraded":"2024-09-16T05:25:05.411671+0000","last_fullsized":"2024-09-16T05:25:05.411671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:00:12.586886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1328,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522520+0000","last_change":"2024-09-16T05:08:51.493220+0000","last_active":"2024-09-16T05:25:05.522520+0000","last_peered":"2024-09-16T05:25:05.522520+0000","last_clean":"2024-09-16T05:25:05.522520+0000","last_became_active":"2024-09-16T05:08:51.486048+0000","last_became_peered":"2024-09-16T05:08:51.486048+0000","last_unstale":"2024-09-16T05:25:05.522520+0000","last_undegraded":"2024-09-16T05:25:05.522520+0000","last_fullsized":"2024-09-16T05:25:05.522520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:52:13.473044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021819+0000","last_change":"2024-09-16T05:08:51.472111+0000","last_active":"2024-09-16T05:25:06.021819+0000","last_peered":"2024-09-16T05:25:06.021819+0000","last_clean":"2024-09-16T05:25:06.021819+0000","last_became_active":"2024-09-16T05:08:51.469556+0000","last_became_peered":"2024-09-16T05:08:51.469556+0000","last_unstale":"2024-09-16T05:25:06.021819+0000","last_undegraded":"2024-09-16T05:25:06.021819+0000","last_fullsized":"2024-09-16T05:25:06.021819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:14.956562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1345,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.462705+0000","last_change":"2024-09-16T05:08:51.533742+0000","last_active":"2024-09-16T05:25:04.462705+0000","last_peered":"2024-09-16T05:25:04.462705+0000","last_clean":"2024-09-16T05:25:04.462705+0000","last_became_active":"2024-09-16T05:08:51.533473+0000","last_became_peered":"2024-09-16T05:08:51.533473+0000","last_unstale":"2024-09-16T05:25:04.462705+0000","last_undegraded":"2024-09-16T05:25:04.462705+0000","last_fullsized":"2024-09-16T05:25:04.462705+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:21.233490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1328,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522976+0000","last_change":"2024-09-16T05:08:51.509275+0000","last_active":"2024-09-16T05:25:05.522976+0000","last_peered":"2024-09-16T05:25:05.522976+0000","last_clean":"2024-09-16T05:25:05.522976+0000","last_became_active":"2024-09-16T05:08:51.493446+0000","last_became_peered":"2024-09-16T05:08:51.493446+0000","last_unstale":"2024-09-16T05:25:05.522976+0000","last_undegraded":"2024-09-16T05:25:05.522976+0000","last_fullsized":"2024-09-16T05:25:05.522976+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:19.908189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834575083,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73084,"kb_used_data":9940,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298756,"statfs":{"total":96636764160,"available":96561926144,"internally_reserved":0,"allocated":10178560,"data_stored":5202552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575084,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77260,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294580,"statfs":{"total":96636764160,"available":96557649920,"internally_reserved":0,"allocated":10182656,"data_stored":5202571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607787,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73604,"kb_used_data":10460,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298236,"statfs":{"total":96636764160,"available":96561393664,"internally_reserved":0,"allocated":10711040,"data_stored":5731480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607787,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73608,"kb_used_data":10464,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298232,"statfs":{"total":96636764160,"available":96561389568,"internally_reserved":0,"allocated":10715136,"data_stored":5731485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640492,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77248,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294592,"statfs":{"total":96636764160,"available":96557662208,"internally_reserved":0,"allocated":10182656,"data_stored":5202571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640492,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89716,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94282124,"statfs":{"total":96636764160,"available":96544894976,"internally_reserved":0,"allocated":10182656,"data_stored":5202557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81656263},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:27:01.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:27:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.491+0000 7fc132e49640 10 monclient: get_monmap_and_config 2024-09-16T05:27:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.491+0000 7fc132e49640 10 monclient: build_initial_monmap 2024-09-16T05:27:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.491+0000 7fc132e49640 10 monclient: monmap: 2024-09-16T05:27:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:01.489 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:01.494814+0000 2024-09-16T05:27:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:01.494814+0000 2024-09-16T05:27:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:01.490 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.491+0000 7fc132e49640 1 Processor -- start 2024-09-16T05:27:01.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.491+0000 7fc132e49640 1 -- start start 2024-09-16T05:27:01.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.491+0000 7fc132e49640 10 monclient: init 2024-09-16T05:27:01.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.491+0000 7fc132e49640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:01.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.491+0000 7fc132e49640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-16T05:27:01.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.491+0000 7fc132e49640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc12c137730 0x7fc12c137b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 10 monclient(hunting): picked mon.noname-c con 0x7fc12c137730 addr v2:172.21.15.149:3300/0 2024-09-16T05:27:01.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc12c0a5d00 0x7fc12c0a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 10 monclient(hunting): picked mon.noname-a con 0x7fc12c0a5d00 addr v2:172.21.15.114:3300/0 2024-09-16T05:27:01.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc12c0a47b0 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:01.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc12c0a55e0 con 0x7fc12c0a47b0 2024-09-16T05:27:01.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fc12c09c450 con 0x7fc12c0a5d00 2024-09-16T05:27:01.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fc12c0a3710 con 0x7fc12c137730 2024-09-16T05:27:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 10 monclient(hunting): authenticate will time out at 2106.124072s 2024-09-16T05:27:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc12c0a5d00 0x7fc12c0a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc12c0a5d00 0x7fc12c0a4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35118/0 (socket says 172.21.15.114:35118) 2024-09-16T05:27:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 1 -- 172.21.15.114:0/2203190868 learned_addr learned my addr 172.21.15.114:0/2203190868 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:01.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 10 monclient(hunting): get_auth_request con 0x7fc12c0a5d00 auth_method 0 2024-09-16T05:27:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:01.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc130be5640 1 --2- 172.21.15.114:0/2203190868 >> v2:172.21.15.149:3300/0 conn(0x7fc12c137730 0x7fc12c137b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 10 monclient(hunting): handle_auth_done global_id 134166 payload 274 2024-09-16T05:27:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 1 -- 172.21.15.114:0/2203190868 >> v2:172.21.15.149:3300/0 conn(0x7fc12c137730 msgr2=0x7fc12c137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 1 --2- 172.21.15.114:0/2203190868 >> v2:172.21.15.149:3300/0 conn(0x7fc12c137730 0x7fc12c137b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 1 -- 172.21.15.114:0/2203190868 >> v1:172.21.15.114:6789/0 conn(0x7fc12c0a47b0 legacy=0x7fc12c13e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 10 monclient: _finish_hunting 0 2024-09-16T05:27:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 1 monclient: found mon.noname-a 2024-09-16T05:27:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 1 -- 172.21.15.114:0/2203190868 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc12c09be00 con 0x7fc12c0a5d00 2024-09-16T05:27:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc130be5640 10 monclient: get_auth_request con 0x7fc12c137730 auth_method 0 2024-09-16T05:27:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc130be5640 1 --2- 172.21.15.114:0/2203190868 >> v2:172.21.15.149:3300/0 conn(0x7fc12c137730 0x7fc12c137b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:27:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc123fff640 1 --2- 172.21.15.114:0/2203190868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12c0a5d00 0x7fc12c0a4270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0a73e0 tx=0x7fc11c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c909297dccfdf8b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:01.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc1237fe640 1 -- 172.21.15.114:0/2203190868 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c0b0070 con 0x7fc12c0a5d00 2024-09-16T05:27:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc1237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc1237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc1237fe640 10 monclient: dump: 2024-09-16T05:27:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.502 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:01.503 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc1237fe640 10 monclient: _finish_auth 0 2024-09-16T05:27:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc1237fe640 10 monclient: _check_auth_tickets 2024-09-16T05:27:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc1237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 5 monclient: authenticate success, global_id 134166 2024-09-16T05:27:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc1237fe640 1 -- 172.21.15.114:0/2203190868 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc11c0ab600 con 0x7fc12c0a5d00 2024-09-16T05:27:01.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc1237fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 4 set_mon_vals no callback set 2024-09-16T05:27:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc1237fe640 1 -- 172.21.15.114:0/2203190868 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c0ab9d0 con 0x7fc12c0a5d00 2024-09-16T05:27:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc1237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc1237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc1237fe640 10 monclient: dump: 2024-09-16T05:27:01.505 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:01.506 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 10 monclient: shutdown 2024-09-16T05:27:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 1 -- 172.21.15.114:0/2203190868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12c0a5d00 msgr2=0x7fc12c0a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 1 --2- 172.21.15.114:0/2203190868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12c0a5d00 0x7fc12c0a4270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0a73e0 tx=0x7fc11c01c440 comp rx=0 tx=0).stop 2024-09-16T05:27:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 1 -- 172.21.15.114:0/2203190868 shutdown_connections 2024-09-16T05:27:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 1 --2- 172.21.15.114:0/2203190868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12c0a5d00 0x7fc12c0a4270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 1 --2- 172.21.15.114:0/2203190868 >> v2:172.21.15.149:3300/0 conn(0x7fc12c137730 0x7fc12c137b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 1 -- 172.21.15.114:0/2203190868 >> 172.21.15.114:0/2203190868 conn(0x7fc12c0a9c70 msgr2=0x7fc12c0aa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 1 -- 172.21.15.114:0/2203190868 shutdown_connections 2024-09-16T05:27:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.495+0000 7fc132e49640 1 -- 172.21.15.114:0/2203190868 wait complete. 2024-09-16T05:27:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient: build_initial_monmap 2024-09-16T05:27:01.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient: monmap: 2024-09-16T05:27:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:01.508 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 librados: starting msgr at 2024-09-16T05:27:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 librados: starting objecter 2024-09-16T05:27:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 Processor -- start 2024-09-16T05:27:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 -- start start 2024-09-16T05:27:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 librados: setting wanted keys 2024-09-16T05:27:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 librados: calling monclient init 2024-09-16T05:27:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient: init 2024-09-16T05:27:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:27:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12c0a47b0 0x7fc12c1cf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient(hunting): picked mon.noname-a con 0x7fc12c0a47b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc12c0a5d00 0x7fc12c13e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient(hunting): picked mon.noname-b con 0x7fc12c0a5d00 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12c137730 0x7fc12c13eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient(hunting): picked mon.noname-c con 0x7fc12c137730 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c072a20 con 0x7fc12c0a47b0 2024-09-16T05:27:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c0728a0 con 0x7fc12c0a5d00 2024-09-16T05:27:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc12c1371a0 con 0x7fc12c137730 2024-09-16T05:27:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient(hunting): authenticate will time out at 2106.129127s 2024-09-16T05:27:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12c0a47b0 0x7fc12c1cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12c0a47b0 0x7fc12c1cf280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35132/0 (socket says 172.21.15.114:35132) 2024-09-16T05:27:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 1 -- 172.21.15.114:0/870759426 learned_addr learned my addr 172.21.15.114:0/870759426 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 10 monclient(hunting): get_auth_request con 0x7fc12c0a47b0 auth_method 0 2024-09-16T05:27:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1313e6640 1 --2- 172.21.15.114:0/870759426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12c137730 0x7fc12c13eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1313e6640 10 monclient(hunting): get_auth_request con 0x7fc12c137730 auth_method 0 2024-09-16T05:27:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1313e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1313e6640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1313e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1313e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1313e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1313e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc123fff640 1 --2- 172.21.15.114:0/870759426 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc12c0a5d00 0x7fc12c13e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 10 monclient(hunting): handle_auth_done global_id 134172 payload 824 2024-09-16T05:27:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 1 -- 172.21.15.114:0/870759426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12c137730 msgr2=0x7fc12c13eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 1 --2- 172.21.15.114:0/870759426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12c137730 0x7fc12c13eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 1 -- 172.21.15.114:0/870759426 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc12c0a5d00 msgr2=0x7fc12c13e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 1 --2- 172.21.15.114:0/870759426 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc12c0a5d00 0x7fc12c13e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 10 monclient: _finish_hunting 0 2024-09-16T05:27:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 1 monclient: found mon.noname-a 2024-09-16T05:27:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 1 -- 172.21.15.114:0/870759426 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc12c13cdc0 con 0x7fc12c0a47b0 2024-09-16T05:27:01.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc123fff640 10 monclient: get_auth_request con 0x7fc12c0a5d00 auth_method 0 2024-09-16T05:27:01.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc123fff640 1 --2- 172.21.15.114:0/870759426 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc12c0a5d00 0x7fc12c13e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:27:01.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc130be5640 1 --2- 172.21.15.114:0/870759426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12c0a47b0 0x7fc12c1cf280 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc12800c7c0 tx=0x7fc12800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:01.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 10 client.?.objecter ms_handle_connect 0x7fc12c0a47b0 2024-09-16T05:27:01.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:01.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 1 -- 172.21.15.114:0/870759426 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc128007a30 con 0x7fc12c0a47b0 2024-09-16T05:27:01.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:01.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:01.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 10 monclient: dump: 2024-09-16T05:27:01.518 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:01.518 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:01.519 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.519 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.519 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:01.519 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.519 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:01.519 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:01.519 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:01.519 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 10 monclient: _finish_auth 0 2024-09-16T05:27:01.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 10 monclient: _check_auth_tickets 2024-09-16T05:27:01.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:01.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 1 -- 172.21.15.114:0/870759426 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc128007bd0 con 0x7fc12c0a47b0 2024-09-16T05:27:01.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:01.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 1 -- 172.21.15.114:0/870759426 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1280113f0 con 0x7fc12c0a47b0 2024-09-16T05:27:01.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 5 monclient: authenticate success, global_id 134172 2024-09-16T05:27:01.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient: _renew_subs 2024-09-16T05:27:01.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:01.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 -- 172.21.15.114:0/870759426 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc12c1cf7c0 con 0x7fc12c0a47b0 2024-09-16T05:27:01.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc122ffd640 4 set_mon_vals no callback set 2024-09-16T05:27:01.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:01.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:01.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 10 monclient: dump: 2024-09-16T05:27:01.520 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:01.521 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:01.521 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.521 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.521 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:01.521 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.521 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:01.521 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:01.521 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:01.521 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 client.134172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:01.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient: _renew_subs 2024-09-16T05:27:01.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:01.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 -- 172.21.15.114:0/870759426 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc12c13f8d0 con 0x7fc12c0a47b0 2024-09-16T05:27:01.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 librados: init done 2024-09-16T05:27:01.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 1 -- 172.21.15.114:0/870759426 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc128011590 con 0x7fc12c0a47b0 2024-09-16T05:27:01.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:01.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:01.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:01.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc132e49640 1 -- 172.21.15.114:0/870759426 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc12c0a4270 con 0x7fc12c0a47b0 2024-09-16T05:27:01.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 1 --2- 172.21.15.114:0/870759426 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc118037810 0x7fc118039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 1 -- 172.21.15.114:0/870759426 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7fc128048d20 con 0x7fc12c0a47b0 2024-09-16T05:27:01.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 10 client.134172.objecter ms_dispatch 0x7fc12c000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:01.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 3 client.134172.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:01.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 3 client.134172.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:01.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc1217fa640 20 client.134172.objecter dump_active .. 0 homeless 2024-09-16T05:27:01.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc123fff640 1 --2- 172.21.15.114:0/870759426 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc118037810 0x7fc118039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.499+0000 7fc123fff640 10 monclient: get_auth_request con 0x7fc118037810 auth_method 0 2024-09-16T05:27:01.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.503+0000 7fc1217fa640 10 client.134172.objecter ms_handle_connect 0x7fc118037810 2024-09-16T05:27:01.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.503+0000 7fc123fff640 1 --2- 172.21.15.114:0/870759426 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc118037810 0x7fc118039cd0 secure :-1 s=READY pgs=3630 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0ad000 tx=0x7fc11c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:01.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.503+0000 7fc1217fa640 1 -- 172.21.15.114:0/870759426 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc12804d070 con 0x7fc12c0a47b0 2024-09-16T05:27:01.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.503+0000 7fc1217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:01.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.503+0000 7fc1217fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:27:01.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.663+0000 7fc132e49640 1 -- 172.21.15.114:0/870759426 --> v2:172.21.15.149:6812/1185531135 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc12c09ceb0 con 0x7fc118037810 2024-09-16T05:27:01.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc1217fa640 1 -- 172.21.15.114:0/870759426 <== mgr.4108 v2:172.21.15.149:6812/1185531135 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114079 (secure 0 0 0) 0x7fc12c09ceb0 con 0x7fc118037810 2024-09-16T05:27:01.672 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:27:01.673 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-16T05:27:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 10 librados: watch_flush enter 2024-09-16T05:27:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 10 librados: watch_flush exit 2024-09-16T05:27:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 20 client.134172.objecter shutdown clearing up homeless session... 2024-09-16T05:27:01.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 10 client.134172.objecter successfully canceled tick 2024-09-16T05:27:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 1 -- 172.21.15.114:0/870759426 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc118037810 msgr2=0x7fc118039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 1 --2- 172.21.15.114:0/870759426 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc118037810 0x7fc118039cd0 secure :-1 s=READY pgs=3630 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0ad000 tx=0x7fc11c008040 comp rx=0 tx=0).stop 2024-09-16T05:27:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 10 monclient: shutdown 2024-09-16T05:27:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 1 -- 172.21.15.114:0/870759426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12c0a47b0 msgr2=0x7fc12c1cf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 1 --2- 172.21.15.114:0/870759426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12c0a47b0 0x7fc12c1cf280 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc12800c7c0 tx=0x7fc12800cc80 comp rx=0 tx=0).stop 2024-09-16T05:27:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 1 -- 172.21.15.114:0/870759426 shutdown_connections 2024-09-16T05:27:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 1 --2- 172.21.15.114:0/870759426 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc118037810 0x7fc118039cd0 unknown :-1 s=CLOSED pgs=3630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 1 --2- 172.21.15.114:0/870759426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc12c137730 0x7fc12c13eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 1 --2- 172.21.15.114:0/870759426 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc12c0a5d00 0x7fc12c13e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 1 --2- 172.21.15.114:0/870759426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc12c0a47b0 0x7fc12c1cf280 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 1 -- 172.21.15.114:0/870759426 >> 172.21.15.114:0/870759426 conn(0x7fc12c0a9c70 msgr2=0x7fc12c136220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 1 -- 172.21.15.114:0/870759426 shutdown_connections 2024-09-16T05:27:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.675+0000 7fc132e49640 1 -- 172.21.15.114:0/870759426 wait complete. 2024-09-16T05:27:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.679+0000 7fc132e49640 1 librados: shutdown 2024-09-16T05:27:01.691 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1266,"stamp":"2024-09-16T05:27:00.512220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":464520,"kb_used_data":60696,"kb_used_omap":9,"kb_used_meta":403574,"kb_avail":565766520,"statfs":{"total":579820584960,"available":579344916480,"internally_reserved":0,"allocated":62152704,"data_stored":32273216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":413260449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003811"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021853+0000","last_change":"2024-09-16T05:08:51.596823+0000","last_active":"2024-09-16T05:25:06.021853+0000","last_peered":"2024-09-16T05:25:06.021853+0000","last_clean":"2024-09-16T05:25:06.021853+0000","last_became_active":"2024-09-16T05:08:51.571632+0000","last_became_peered":"2024-09-16T05:08:51.571632+0000","last_unstale":"2024-09-16T05:25:06.021853+0000","last_undegraded":"2024-09-16T05:25:06.021853+0000","last_fullsized":"2024-09-16T05:25:06.021853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:16.107411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1324,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411274+0000","last_change":"2024-09-16T05:08:51.611318+0000","last_active":"2024-09-16T05:25:05.411274+0000","last_peered":"2024-09-16T05:25:05.411274+0000","last_clean":"2024-09-16T05:25:05.411274+0000","last_became_active":"2024-09-16T05:08:51.585360+0000","last_became_peered":"2024-09-16T05:08:51.585360+0000","last_unstale":"2024-09-16T05:25:05.411274+0000","last_undegraded":"2024-09-16T05:25:05.411274+0000","last_fullsized":"2024-09-16T05:25:05.411274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:19:58.506693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021814+0000","last_change":"2024-09-16T05:08:51.532106+0000","last_active":"2024-09-16T05:25:06.021814+0000","last_peered":"2024-09-16T05:25:06.021814+0000","last_clean":"2024-09-16T05:25:06.021814+0000","last_became_active":"2024-09-16T05:08:51.519558+0000","last_became_peered":"2024-09-16T05:08:51.519558+0000","last_unstale":"2024-09-16T05:25:06.021814+0000","last_undegraded":"2024-09-16T05:25:06.021814+0000","last_fullsized":"2024-09-16T05:25:06.021814+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:52:54.764452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1316,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.441285+0000","last_change":"2024-09-16T05:08:51.705649+0000","last_active":"2024-09-16T05:25:05.441285+0000","last_peered":"2024-09-16T05:25:05.441285+0000","last_clean":"2024-09-16T05:25:05.441285+0000","last_became_active":"2024-09-16T05:08:51.689931+0000","last_became_peered":"2024-09-16T05:08:51.689931+0000","last_unstale":"2024-09-16T05:25:05.441285+0000","last_undegraded":"2024-09-16T05:25:05.441285+0000","last_fullsized":"2024-09-16T05:25:05.441285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:31.971156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522829+0000","last_change":"2024-09-16T05:08:51.564493+0000","last_active":"2024-09-16T05:25:05.522829+0000","last_peered":"2024-09-16T05:25:05.522829+0000","last_clean":"2024-09-16T05:25:05.522829+0000","last_became_active":"2024-09-16T05:08:51.546851+0000","last_became_peered":"2024-09-16T05:08:51.546851+0000","last_unstale":"2024-09-16T05:25:05.522829+0000","last_undegraded":"2024-09-16T05:25:05.522829+0000","last_fullsized":"2024-09-16T05:25:05.522829+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:23.731218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.463123+0000","last_change":"2024-09-16T05:08:51.617848+0000","last_active":"2024-09-16T05:25:04.463123+0000","last_peered":"2024-09-16T05:25:04.463123+0000","last_clean":"2024-09-16T05:25:04.463123+0000","last_became_active":"2024-09-16T05:08:51.605843+0000","last_became_peered":"2024-09-16T05:08:51.605843+0000","last_unstale":"2024-09-16T05:25:04.463123+0000","last_undegraded":"2024-09-16T05:25:04.463123+0000","last_fullsized":"2024-09-16T05:25:04.463123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:24:14.376183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1317,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.441115+0000","last_change":"2024-09-16T05:08:51.615740+0000","last_active":"2024-09-16T05:25:05.441115+0000","last_peered":"2024-09-16T05:25:05.441115+0000","last_clean":"2024-09-16T05:25:05.441115+0000","last_became_active":"2024-09-16T05:08:51.609820+0000","last_became_peered":"2024-09-16T05:08:51.609820+0000","last_unstale":"2024-09-16T05:25:05.441115+0000","last_undegraded":"2024-09-16T05:25:05.441115+0000","last_fullsized":"2024-09-16T05:25:05.441115+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:35:22.111340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1322,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021670+0000","last_change":"2024-09-16T05:08:51.603805+0000","last_active":"2024-09-16T05:25:06.021670+0000","last_peered":"2024-09-16T05:25:06.021670+0000","last_clean":"2024-09-16T05:25:06.021670+0000","last_became_active":"2024-09-16T05:08:51.567268+0000","last_became_peered":"2024-09-16T05:08:51.567268+0000","last_unstale":"2024-09-16T05:25:06.021670+0000","last_undegraded":"2024-09-16T05:25:06.021670+0000","last_fullsized":"2024-09-16T05:25:06.021670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:43:35.723700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1325,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411521+0000","last_change":"2024-09-16T05:08:51.640189+0000","last_active":"2024-09-16T05:25:05.411521+0000","last_peered":"2024-09-16T05:25:05.411521+0000","last_clean":"2024-09-16T05:25:05.411521+0000","last_became_active":"2024-09-16T05:08:51.634768+0000","last_became_peered":"2024-09-16T05:08:51.634768+0000","last_unstale":"2024-09-16T05:25:05.411521+0000","last_undegraded":"2024-09-16T05:25:05.411521+0000","last_fullsized":"2024-09-16T05:25:05.411521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:52:08.194663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1323,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.735797+0000","last_change":"2024-09-16T05:08:51.589983+0000","last_active":"2024-09-16T05:25:04.735797+0000","last_peered":"2024-09-16T05:25:04.735797+0000","last_clean":"2024-09-16T05:25:04.735797+0000","last_became_active":"2024-09-16T05:08:51.589818+0000","last_became_peered":"2024-09-16T05:08:51.589818+0000","last_unstale":"2024-09-16T05:25:04.735797+0000","last_undegraded":"2024-09-16T05:25:04.735797+0000","last_fullsized":"2024-09-16T05:25:04.735797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:55:59.967412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.465088+0000","last_change":"2024-09-16T05:08:51.609659+0000","last_active":"2024-09-16T05:25:04.465088+0000","last_peered":"2024-09-16T05:25:04.465088+0000","last_clean":"2024-09-16T05:25:04.465088+0000","last_became_active":"2024-09-16T05:08:51.605719+0000","last_became_peered":"2024-09-16T05:08:51.605719+0000","last_unstale":"2024-09-16T05:25:04.465088+0000","last_undegraded":"2024-09-16T05:25:04.465088+0000","last_fullsized":"2024-09-16T05:25:04.465088+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:43:56.008078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1325,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411040+0000","last_change":"2024-09-16T05:08:51.630514+0000","last_active":"2024-09-16T05:25:05.411040+0000","last_peered":"2024-09-16T05:25:05.411040+0000","last_clean":"2024-09-16T05:25:05.411040+0000","last_became_active":"2024-09-16T05:08:51.611801+0000","last_became_peered":"2024-09-16T05:08:51.611801+0000","last_unstale":"2024-09-16T05:25:05.411040+0000","last_undegraded":"2024-09-16T05:25:05.411040+0000","last_fullsized":"2024-09-16T05:25:05.411040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:46.255904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522975+0000","last_change":"2024-09-16T05:08:51.605439+0000","last_active":"2024-09-16T05:25:05.522975+0000","last_peered":"2024-09-16T05:25:05.522975+0000","last_clean":"2024-09-16T05:25:05.522975+0000","last_became_active":"2024-09-16T05:08:51.586611+0000","last_became_peered":"2024-09-16T05:08:51.586611+0000","last_unstale":"2024-09-16T05:25:05.522975+0000","last_undegraded":"2024-09-16T05:25:05.522975+0000","last_fullsized":"2024-09-16T05:25:05.522975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:39:46.639778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":1492,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.410709+0000","last_change":"2024-09-16T05:07:39.580660+0000","last_active":"2024-09-16T05:25:05.410709+0000","last_peered":"2024-09-16T05:25:05.410709+0000","last_clean":"2024-09-16T05:25:05.410709+0000","last_became_active":"2024-09-16T05:07:39.576756+0000","last_became_peered":"2024-09-16T05:07:39.576756+0000","last_unstale":"2024-09-16T05:25:05.410709+0000","last_undegraded":"2024-09-16T05:25:05.410709+0000","last_fullsized":"2024-09-16T05:25:05.410709+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_clean_scrub_stamp":"2024-09-16T05:07:38.167879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:57.060906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1342,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411039+0000","last_change":"2024-09-16T05:07:48.267201+0000","last_active":"2024-09-16T05:25:05.411039+0000","last_peered":"2024-09-16T05:25:05.411039+0000","last_clean":"2024-09-16T05:25:05.411039+0000","last_became_active":"2024-09-16T05:07:46.246409+0000","last_became_peered":"2024-09-16T05:07:46.246409+0000","last_unstale":"2024-09-16T05:25:05.411039+0000","last_undegraded":"2024-09-16T05:25:05.411039+0000","last_fullsized":"2024-09-16T05:25:05.411039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:13.808827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010413149999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1323,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.735867+0000","last_change":"2024-09-16T05:08:51.582564+0000","last_active":"2024-09-16T05:25:04.735867+0000","last_peered":"2024-09-16T05:25:04.735867+0000","last_clean":"2024-09-16T05:25:04.735867+0000","last_became_active":"2024-09-16T05:08:51.582424+0000","last_became_peered":"2024-09-16T05:08:51.582424+0000","last_unstale":"2024-09-16T05:25:04.735867+0000","last_undegraded":"2024-09-16T05:25:04.735867+0000","last_fullsized":"2024-09-16T05:25:04.735867+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:43.882076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1333,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.441020+0000","last_change":"2024-09-16T05:07:48.266504+0000","last_active":"2024-09-16T05:25:05.441020+0000","last_peered":"2024-09-16T05:25:05.441020+0000","last_clean":"2024-09-16T05:25:05.441020+0000","last_became_active":"2024-09-16T05:07:46.254322+0000","last_became_peered":"2024-09-16T05:07:46.254322+0000","last_unstale":"2024-09-16T05:25:05.441020+0000","last_undegraded":"2024-09-16T05:25:05.441020+0000","last_fullsized":"2024-09-16T05:25:05.441020+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:00.175590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093680599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522665+0000","last_change":"2024-09-16T05:08:51.549772+0000","last_active":"2024-09-16T05:25:05.522665+0000","last_peered":"2024-09-16T05:25:05.522665+0000","last_clean":"2024-09-16T05:25:05.522665+0000","last_became_active":"2024-09-16T05:08:51.539573+0000","last_became_peered":"2024-09-16T05:08:51.539573+0000","last_unstale":"2024-09-16T05:25:05.522665+0000","last_undegraded":"2024-09-16T05:25:05.522665+0000","last_fullsized":"2024-09-16T05:25:05.522665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:10:54.876819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1362,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.462909+0000","last_change":"2024-09-16T05:07:48.263513+0000","last_active":"2024-09-16T05:25:04.462909+0000","last_peered":"2024-09-16T05:25:04.462909+0000","last_clean":"2024-09-16T05:25:04.462909+0000","last_became_active":"2024-09-16T05:07:46.252254+0000","last_became_peered":"2024-09-16T05:07:46.252254+0000","last_unstale":"2024-09-16T05:25:04.462909+0000","last_undegraded":"2024-09-16T05:25:04.462909+0000","last_fullsized":"2024-09-16T05:25:04.462909+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:33.736560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081560699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.465371+0000","last_change":"2024-09-16T05:08:51.542756+0000","last_active":"2024-09-16T05:25:04.465371+0000","last_peered":"2024-09-16T05:25:04.465371+0000","last_clean":"2024-09-16T05:25:04.465371+0000","last_became_active":"2024-09-16T05:08:51.541976+0000","last_became_peered":"2024-09-16T05:08:51.541976+0000","last_unstale":"2024-09-16T05:25:04.465371+0000","last_undegraded":"2024-09-16T05:25:04.465371+0000","last_fullsized":"2024-09-16T05:25:04.465371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:48:46.043655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":1347,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522619+0000","last_change":"2024-09-16T05:07:48.274329+0000","last_active":"2024-09-16T05:25:05.522619+0000","last_peered":"2024-09-16T05:25:05.522619+0000","last_clean":"2024-09-16T05:25:05.522619+0000","last_became_active":"2024-09-16T05:07:46.257118+0000","last_became_peered":"2024-09-16T05:07:46.257118+0000","last_unstale":"2024-09-16T05:25:05.522619+0000","last_undegraded":"2024-09-16T05:25:05.522619+0000","last_fullsized":"2024-09-16T05:25:05.522619+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:35.932772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020216650000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1323,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021532+0000","last_change":"2024-09-16T05:08:51.506142+0000","last_active":"2024-09-16T05:25:06.021532+0000","last_peered":"2024-09-16T05:25:06.021532+0000","last_clean":"2024-09-16T05:25:06.021532+0000","last_became_active":"2024-09-16T05:08:51.491760+0000","last_became_peered":"2024-09-16T05:08:51.491760+0000","last_unstale":"2024-09-16T05:25:06.021532+0000","last_undegraded":"2024-09-16T05:25:06.021532+0000","last_fullsized":"2024-09-16T05:25:06.021532+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:04:43.526278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1339,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.735843+0000","last_change":"2024-09-16T05:07:48.352028+0000","last_active":"2024-09-16T05:25:04.735843+0000","last_peered":"2024-09-16T05:25:04.735843+0000","last_clean":"2024-09-16T05:25:04.735843+0000","last_became_active":"2024-09-16T05:07:46.255633+0000","last_became_peered":"2024-09-16T05:07:46.255633+0000","last_unstale":"2024-09-16T05:25:04.735843+0000","last_undegraded":"2024-09-16T05:25:04.735843+0000","last_fullsized":"2024-09-16T05:25:04.735843+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:07.079219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000883379,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"21'7","reported_seq":1359,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.462678+0000","last_change":"2024-09-16T05:08:51.617951+0000","last_active":"2024-09-16T05:25:04.462678+0000","last_peered":"2024-09-16T05:25:04.462678+0000","last_clean":"2024-09-16T05:25:04.462678+0000","last_became_active":"2024-09-16T05:08:51.614284+0000","last_became_peered":"2024-09-16T05:08:51.614284+0000","last_unstale":"2024-09-16T05:25:04.462678+0000","last_undegraded":"2024-09-16T05:25:04.462678+0000","last_fullsized":"2024-09-16T05:25:04.462678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:18.960138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1341,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411163+0000","last_change":"2024-09-16T05:07:48.267372+0000","last_active":"2024-09-16T05:25:05.411163+0000","last_peered":"2024-09-16T05:25:05.411163+0000","last_clean":"2024-09-16T05:25:05.411163+0000","last_became_active":"2024-09-16T05:07:46.243891+0000","last_became_peered":"2024-09-16T05:07:46.243891+0000","last_unstale":"2024-09-16T05:25:05.411163+0000","last_undegraded":"2024-09-16T05:25:05.411163+0000","last_fullsized":"2024-09-16T05:25:05.411163+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:48:25.061723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011829709999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1325,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.410924+0000","last_change":"2024-09-16T05:08:51.634776+0000","last_active":"2024-09-16T05:25:05.410924+0000","last_peered":"2024-09-16T05:25:05.410924+0000","last_clean":"2024-09-16T05:25:05.410924+0000","last_became_active":"2024-09-16T05:08:51.632213+0000","last_became_peered":"2024-09-16T05:08:51.632213+0000","last_unstale":"2024-09-16T05:25:05.410924+0000","last_undegraded":"2024-09-16T05:25:05.410924+0000","last_fullsized":"2024-09-16T05:25:05.410924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:47:43.409907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1337,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021856+0000","last_change":"2024-09-16T05:07:48.275545+0000","last_active":"2024-09-16T05:25:06.021856+0000","last_peered":"2024-09-16T05:25:06.021856+0000","last_clean":"2024-09-16T05:25:06.021856+0000","last_became_active":"2024-09-16T05:07:46.252207+0000","last_became_peered":"2024-09-16T05:07:46.252207+0000","last_unstale":"2024-09-16T05:25:06.021856+0000","last_undegraded":"2024-09-16T05:25:06.021856+0000","last_fullsized":"2024-09-16T05:25:06.021856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:36:08.378166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073333700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522617+0000","last_change":"2024-09-16T05:08:51.499333+0000","last_active":"2024-09-16T05:25:05.522617+0000","last_peered":"2024-09-16T05:25:05.522617+0000","last_clean":"2024-09-16T05:25:05.522617+0000","last_became_active":"2024-09-16T05:08:51.486807+0000","last_became_peered":"2024-09-16T05:08:51.486807+0000","last_unstale":"2024-09-16T05:25:05.522617+0000","last_undegraded":"2024-09-16T05:25:05.522617+0000","last_fullsized":"2024-09-16T05:25:05.522617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:43.084461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1340,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411030+0000","last_change":"2024-09-16T05:07:48.267914+0000","last_active":"2024-09-16T05:25:05.411030+0000","last_peered":"2024-09-16T05:25:05.411030+0000","last_clean":"2024-09-16T05:25:05.411030+0000","last_became_active":"2024-09-16T05:07:46.254034+0000","last_became_peered":"2024-09-16T05:07:46.254034+0000","last_unstale":"2024-09-16T05:25:05.411030+0000","last_undegraded":"2024-09-16T05:25:05.411030+0000","last_fullsized":"2024-09-16T05:25:05.411030+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:57:37.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014490219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021562+0000","last_change":"2024-09-16T05:08:51.548570+0000","last_active":"2024-09-16T05:25:06.021562+0000","last_peered":"2024-09-16T05:25:06.021562+0000","last_clean":"2024-09-16T05:25:06.021562+0000","last_became_active":"2024-09-16T05:08:51.532685+0000","last_became_peered":"2024-09-16T05:08:51.532685+0000","last_unstale":"2024-09-16T05:25:06.021562+0000","last_undegraded":"2024-09-16T05:25:06.021562+0000","last_fullsized":"2024-09-16T05:25:06.021562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:14:25.488128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021522+0000","last_change":"2024-09-16T05:08:51.604786+0000","last_active":"2024-09-16T05:25:06.021522+0000","last_peered":"2024-09-16T05:25:06.021522+0000","last_clean":"2024-09-16T05:25:06.021522+0000","last_became_active":"2024-09-16T05:08:51.567861+0000","last_became_peered":"2024-09-16T05:08:51.567861+0000","last_unstale":"2024-09-16T05:25:06.021522+0000","last_undegraded":"2024-09-16T05:25:06.021522+0000","last_fullsized":"2024-09-16T05:25:06.021522+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:48:14.930571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1345,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.463346+0000","last_change":"2024-09-16T05:08:51.563783+0000","last_active":"2024-09-16T05:25:04.463346+0000","last_peered":"2024-09-16T05:25:04.463346+0000","last_clean":"2024-09-16T05:25:04.463346+0000","last_became_active":"2024-09-16T05:08:51.563420+0000","last_became_peered":"2024-09-16T05:08:51.563420+0000","last_unstale":"2024-09-16T05:25:04.463346+0000","last_undegraded":"2024-09-16T05:25:04.463346+0000","last_fullsized":"2024-09-16T05:25:04.463346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:52.527569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1328,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522885+0000","last_change":"2024-09-16T05:08:51.513817+0000","last_active":"2024-09-16T05:25:05.522885+0000","last_peered":"2024-09-16T05:25:05.522885+0000","last_clean":"2024-09-16T05:25:05.522885+0000","last_became_active":"2024-09-16T05:08:51.503720+0000","last_became_peered":"2024-09-16T05:08:51.503720+0000","last_unstale":"2024-09-16T05:25:05.522885+0000","last_undegraded":"2024-09-16T05:25:05.522885+0000","last_fullsized":"2024-09-16T05:25:05.522885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:31.884629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021857+0000","last_change":"2024-09-16T05:08:51.505598+0000","last_active":"2024-09-16T05:25:06.021857+0000","last_peered":"2024-09-16T05:25:06.021857+0000","last_clean":"2024-09-16T05:25:06.021857+0000","last_became_active":"2024-09-16T05:08:51.494038+0000","last_became_peered":"2024-09-16T05:08:51.494038+0000","last_unstale":"2024-09-16T05:25:06.021857+0000","last_undegraded":"2024-09-16T05:25:06.021857+0000","last_fullsized":"2024-09-16T05:25:06.021857+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:39:28.303932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1324,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411448+0000","last_change":"2024-09-16T05:08:51.680865+0000","last_active":"2024-09-16T05:25:05.411448+0000","last_peered":"2024-09-16T05:25:05.411448+0000","last_clean":"2024-09-16T05:25:05.411448+0000","last_became_active":"2024-09-16T05:08:51.680588+0000","last_became_peered":"2024-09-16T05:08:51.680588+0000","last_unstale":"2024-09-16T05:25:05.411448+0000","last_undegraded":"2024-09-16T05:25:05.411448+0000","last_fullsized":"2024-09-16T05:25:05.411448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:52.176771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.463756+0000","last_change":"2024-09-16T05:08:51.490785+0000","last_active":"2024-09-16T05:25:04.463756+0000","last_peered":"2024-09-16T05:25:04.463756+0000","last_clean":"2024-09-16T05:25:04.463756+0000","last_became_active":"2024-09-16T05:08:51.480748+0000","last_became_peered":"2024-09-16T05:08:51.480748+0000","last_unstale":"2024-09-16T05:25:04.463756+0000","last_undegraded":"2024-09-16T05:25:04.463756+0000","last_fullsized":"2024-09-16T05:25:04.463756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:21:24.554888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1325,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411671+0000","last_change":"2024-09-16T05:08:51.623309+0000","last_active":"2024-09-16T05:25:05.411671+0000","last_peered":"2024-09-16T05:25:05.411671+0000","last_clean":"2024-09-16T05:25:05.411671+0000","last_became_active":"2024-09-16T05:08:51.622941+0000","last_became_peered":"2024-09-16T05:08:51.622941+0000","last_unstale":"2024-09-16T05:25:05.411671+0000","last_undegraded":"2024-09-16T05:25:05.411671+0000","last_fullsized":"2024-09-16T05:25:05.411671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:00:12.586886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1328,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522520+0000","last_change":"2024-09-16T05:08:51.493220+0000","last_active":"2024-09-16T05:25:05.522520+0000","last_peered":"2024-09-16T05:25:05.522520+0000","last_clean":"2024-09-16T05:25:05.522520+0000","last_became_active":"2024-09-16T05:08:51.486048+0000","last_became_peered":"2024-09-16T05:08:51.486048+0000","last_unstale":"2024-09-16T05:25:05.522520+0000","last_undegraded":"2024-09-16T05:25:05.522520+0000","last_fullsized":"2024-09-16T05:25:05.522520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:52:13.473044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021819+0000","last_change":"2024-09-16T05:08:51.472111+0000","last_active":"2024-09-16T05:25:06.021819+0000","last_peered":"2024-09-16T05:25:06.021819+0000","last_clean":"2024-09-16T05:25:06.021819+0000","last_became_active":"2024-09-16T05:08:51.469556+0000","last_became_peered":"2024-09-16T05:08:51.469556+0000","last_unstale":"2024-09-16T05:25:06.021819+0000","last_undegraded":"2024-09-16T05:25:06.021819+0000","last_fullsized":"2024-09-16T05:25:06.021819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:14.956562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1345,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.462705+0000","last_change":"2024-09-16T05:08:51.533742+0000","last_active":"2024-09-16T05:25:04.462705+0000","last_peered":"2024-09-16T05:25:04.462705+0000","last_clean":"2024-09-16T05:25:04.462705+0000","last_became_active":"2024-09-16T05:08:51.533473+0000","last_became_peered":"2024-09-16T05:08:51.533473+0000","last_unstale":"2024-09-16T05:25:04.462705+0000","last_undegraded":"2024-09-16T05:25:04.462705+0000","last_fullsized":"2024-09-16T05:25:04.462705+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:21.233490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1328,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522976+0000","last_change":"2024-09-16T05:08:51.509275+0000","last_active":"2024-09-16T05:25:05.522976+0000","last_peered":"2024-09-16T05:25:05.522976+0000","last_clean":"2024-09-16T05:25:05.522976+0000","last_became_active":"2024-09-16T05:08:51.493446+0000","last_became_peered":"2024-09-16T05:08:51.493446+0000","last_unstale":"2024-09-16T05:25:05.522976+0000","last_undegraded":"2024-09-16T05:25:05.522976+0000","last_fullsized":"2024-09-16T05:25:05.522976+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:19.908189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834575083,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73084,"kb_used_data":9940,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298756,"statfs":{"total":96636764160,"available":96561926144,"internally_reserved":0,"allocated":10178560,"data_stored":5202552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575084,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77260,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294580,"statfs":{"total":96636764160,"available":96557649920,"internally_reserved":0,"allocated":10182656,"data_stored":5202571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607787,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73604,"kb_used_data":10460,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298236,"statfs":{"total":96636764160,"available":96561393664,"internally_reserved":0,"allocated":10711040,"data_stored":5731480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607787,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73608,"kb_used_data":10464,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298232,"statfs":{"total":96636764160,"available":96561389568,"internally_reserved":0,"allocated":10715136,"data_stored":5731485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640492,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77248,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294592,"statfs":{"total":96636764160,"available":96557662208,"internally_reserved":0,"allocated":10182656,"data_stored":5202571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640492,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89716,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94282124,"statfs":{"total":96636764160,"available":96544894976,"internally_reserved":0,"allocated":10182656,"data_stored":5202557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81656263},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:27:01.693 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T05:27:01.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:27:01.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 10 monclient: get_monmap_and_config 2024-09-16T05:27:01.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 10 monclient: build_initial_monmap 2024-09-16T05:27:01.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 10 monclient: monmap: 2024-09-16T05:27:01.805 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:01.805 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:01.806 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:01.810806+0000 2024-09-16T05:27:01.806 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:01.810806+0000 2024-09-16T05:27:01.806 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:01.806 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.806 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:01.806 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:01.806 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:01.806 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:01.806 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:01.806 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 1 Processor -- start 2024-09-16T05:27:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 1 -- start start 2024-09-16T05:27:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 10 monclient: init 2024-09-16T05:27:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-16T05:27:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f982c09b6c0 0x7f982c0a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 10 monclient(hunting): picked mon.noname-a con 0x7f982c09b6c0 addr v2:172.21.15.114:3300/0 2024-09-16T05:27:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f982c0a1400 0x7f982c0a1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 10 monclient(hunting): picked mon.noname-c con 0x7f982c0a1400 addr v2:172.21.15.149:3300/0 2024-09-16T05:27:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c139640 0x7f982c13ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 10 monclient(hunting): picked mon.noname-b con 0x7f982c139640 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f982c09e390 con 0x7f982c09b6c0 2024-09-16T05:27:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f982c0a1d50 con 0x7f982c0a1400 2024-09-16T05:27:01.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f982c09ece0 con 0x7f982c139640 2024-09-16T05:27:01.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:01.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.807+0000 7f9832341640 10 monclient(hunting): authenticate will time out at 2106.439925s 2024-09-16T05:27:01.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982bfff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f982c09b6c0 0x7f982c0a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c139640 0x7f982c13ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982bfff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f982c09b6c0 0x7f982c0a0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35138/0 (socket says 172.21.15.114:35138) 2024-09-16T05:27:01.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982bfff640 1 -- 172.21.15.114:0/3519259457 learned_addr learned my addr 172.21.15.114:0/3519259457 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 10 monclient(hunting): get_auth_request con 0x7f982c139640 auth_method 0 2024-09-16T05:27:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982bfff640 10 monclient(hunting): get_auth_request con 0x7f982c09b6c0 auth_method 0 2024-09-16T05:27:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982b7fe640 1 --2- 172.21.15.114:0/3519259457 >> v2:172.21.15.149:3300/0 conn(0x7f982c0a1400 0x7f982c0a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 10 monclient(hunting): handle_auth_done global_id 104405 payload 274 2024-09-16T05:27:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 1 -- 172.21.15.114:0/3519259457 >> v2:172.21.15.149:3300/0 conn(0x7f982c0a1400 msgr2=0x7f982c0a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 1 --2- 172.21.15.114:0/3519259457 >> v2:172.21.15.149:3300/0 conn(0x7f982c0a1400 0x7f982c0a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 1 -- 172.21.15.114:0/3519259457 >> v2:172.21.15.114:3300/0 conn(0x7f982c09b6c0 msgr2=0x7f982c0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 1 --2- 172.21.15.114:0/3519259457 >> v2:172.21.15.114:3300/0 conn(0x7f982c09b6c0 0x7f982c0a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 10 monclient: _finish_hunting 0 2024-09-16T05:27:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 1 monclient: found mon.noname-b 2024-09-16T05:27:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:27:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 1 -- 172.21.15.114:0/3519259457 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982c09d4e0 con 0x7f982c139640 2024-09-16T05:27:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982b7fe640 10 monclient: get_auth_request con 0x7f982c0a1400 auth_method 0 2024-09-16T05:27:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982bfff640 1 --2- 172.21.15.114:0/3519259457 >> v2:172.21.15.114:3300/0 conn(0x7f982c09b6c0 0x7f982c0a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:27:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982b7fe640 1 --2- 172.21.15.114:0/3519259457 >> v2:172.21.15.149:3300/0 conn(0x7f982c0a1400 0x7f982c0a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:27:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f98308de640 1 --2- 172.21.15.114:0/3519259457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c139640 0x7f982c13ba30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f982c09c630 tx=0x7f981c01c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27fe5a06b79805b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982affd640 1 -- 172.21.15.114:0/3519259457 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981c0b0070 con 0x7f982c139640 2024-09-16T05:27:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:27:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982affd640 10 monclient: dump: 2024-09-16T05:27:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:01.816 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.816 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:01.816 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:01.816 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:01.816 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982affd640 10 monclient: _finish_auth 0 2024-09-16T05:27:01.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982affd640 10 monclient: _check_auth_tickets 2024-09-16T05:27:01.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982affd640 1 -- 172.21.15.114:0/3519259457 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f981c0ab600 con 0x7f982c139640 2024-09-16T05:27:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 5 monclient: authenticate success, global_id 104405 2024-09-16T05:27:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 4 set_mon_vals no callback set 2024-09-16T05:27:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982affd640 1 -- 172.21.15.114:0/3519259457 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981c0ab9d0 con 0x7f982c139640 2024-09-16T05:27:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f982affd640 10 monclient: dump: 2024-09-16T05:27:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:01.818 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 10 monclient: shutdown 2024-09-16T05:27:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 1 -- 172.21.15.114:0/3519259457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c139640 msgr2=0x7f982c13ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 1 --2- 172.21.15.114:0/3519259457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c139640 0x7f982c13ba30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f982c09c630 tx=0x7f981c01c320 comp rx=0 tx=0).stop 2024-09-16T05:27:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 1 -- 172.21.15.114:0/3519259457 shutdown_connections 2024-09-16T05:27:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 1 --2- 172.21.15.114:0/3519259457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c139640 0x7f982c13ba30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 1 --2- 172.21.15.114:0/3519259457 >> v2:172.21.15.149:3300/0 conn(0x7f982c0a1400 0x7f982c0a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 1 --2- 172.21.15.114:0/3519259457 >> v2:172.21.15.114:3300/0 conn(0x7f982c09b6c0 0x7f982c0a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 1 -- 172.21.15.114:0/3519259457 >> 172.21.15.114:0/3519259457 conn(0x7f982c0a1fb0 msgr2=0x7f982c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 1 -- 172.21.15.114:0/3519259457 shutdown_connections 2024-09-16T05:27:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 1 -- 172.21.15.114:0/3519259457 wait complete. 2024-09-16T05:27:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 10 monclient: build_initial_monmap 2024-09-16T05:27:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.811+0000 7f9832341640 10 monclient: monmap: 2024-09-16T05:27:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:01.819 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 librados: starting msgr at 2024-09-16T05:27:01.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 librados: starting objecter 2024-09-16T05:27:01.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 Processor -- start 2024-09-16T05:27:01.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 -- start start 2024-09-16T05:27:01.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 librados: setting wanted keys 2024-09-16T05:27:01.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 librados: calling monclient init 2024-09-16T05:27:01.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient: init 2024-09-16T05:27:01.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:01.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:27:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f982c09b6c0 0x7f982c0994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient(hunting): picked mon.noname-b con 0x7f982c09b6c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0a1400 0x7f982c099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient(hunting): picked mon.noname-c con 0x7f982c0a1400 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f982c139640 0x7f982c099f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient(hunting): picked mon.noname-a con 0x7f982c139640 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f982c13e790 con 0x7f982c139640 2024-09-16T05:27:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f982c09cad0 con 0x7f982c09b6c0 2024-09-16T05:27:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f982c13e910 con 0x7f982c0a1400 2024-09-16T05:27:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient(hunting): authenticate will time out at 2106.444804s 2024-09-16T05:27:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f982c139640 0x7f982c099f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982b7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0a1400 0x7f982c099a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f982c139640 0x7f982c099f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35154/0 (socket says 172.21.15.114:35154) 2024-09-16T05:27:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 1 -- 172.21.15.114:0/3398917159 learned_addr learned my addr 172.21.15.114:0/3398917159 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 10 monclient(hunting): get_auth_request con 0x7f982c139640 auth_method 0 2024-09-16T05:27:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982b7fe640 10 monclient(hunting): get_auth_request con 0x7f982c0a1400 auth_method 0 2024-09-16T05:27:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982bfff640 1 --2- 172.21.15.114:0/3398917159 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f982c09b6c0 0x7f982c0994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982bfff640 10 monclient(hunting): get_auth_request con 0x7f982c09b6c0 auth_method 0 2024-09-16T05:27:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 10 monclient(hunting): handle_auth_done global_id 134178 payload 824 2024-09-16T05:27:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 1 -- 172.21.15.114:0/3398917159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0a1400 msgr2=0x7f982c099a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 1 --2- 172.21.15.114:0/3398917159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0a1400 0x7f982c099a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 1 -- 172.21.15.114:0/3398917159 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f982c09b6c0 msgr2=0x7f982c0994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 1 --2- 172.21.15.114:0/3398917159 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f982c09b6c0 0x7f982c0994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 10 monclient: _finish_hunting 0 2024-09-16T05:27:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 1 monclient: found mon.noname-a 2024-09-16T05:27:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 1 -- 172.21.15.114:0/3398917159 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982c0942e0 con 0x7f982c139640 2024-09-16T05:27:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982b7fe640 1 --2- 172.21.15.114:0/3398917159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0a1400 0x7f982c099a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:27:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f98308de640 1 --2- 172.21.15.114:0/3398917159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f982c139640 0x7f982c099f60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f981c01a9d0 tx=0x7f981c0190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 10 client.?.objecter ms_handle_connect 0x7f982c139640 2024-09-16T05:27:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 1 -- 172.21.15.114:0/3398917159 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981c0abe60 con 0x7f982c139640 2024-09-16T05:27:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 10 monclient: dump: 2024-09-16T05:27:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:01.829 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 10 monclient: _finish_auth 0 2024-09-16T05:27:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:27:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 5 monclient: authenticate success, global_id 134178 2024-09-16T05:27:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 1 -- 172.21.15.114:0/3398917159 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f981c0b9070 con 0x7f982c139640 2024-09-16T05:27:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient: _renew_subs 2024-09-16T05:27:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 -- 172.21.15.114:0/3398917159 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f982c13bc20 con 0x7f982c139640 2024-09-16T05:27:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 1 -- 172.21.15.114:0/3398917159 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981c0b0040 con 0x7f982c139640 2024-09-16T05:27:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 10 monclient: dump: 2024-09-16T05:27:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:01.831 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982a7fc640 4 set_mon_vals no callback set 2024-09-16T05:27:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 client.134178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient: _renew_subs 2024-09-16T05:27:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 -- 172.21.15.114:0/3398917159 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f982c093d20 con 0x7f982c139640 2024-09-16T05:27:01.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 librados: init done 2024-09-16T05:27:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 1 -- 172.21.15.114:0/3398917159 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f981c0b4650 con 0x7f982c139640 2024-09-16T05:27:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9832341640 1 -- 172.21.15.114:0/3398917159 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f982c093b20 con 0x7f982c139640 2024-09-16T05:27:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 1 --2- 172.21.15.114:0/3398917159 >> v2:172.21.15.149:6812/1185531135 conn(0x7f980c037810 0x7f980c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 1 -- 172.21.15.114:0/3398917159 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7f981c0e2800 con 0x7f982c139640 2024-09-16T05:27:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 10 client.134178.objecter ms_dispatch 0x7f982c000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 3 client.134178.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 3 client.134178.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f9828ff9640 20 client.134178.objecter dump_active .. 0 homeless 2024-09-16T05:27:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982bfff640 1 --2- 172.21.15.114:0/3398917159 >> v2:172.21.15.149:6812/1185531135 conn(0x7f980c037810 0x7f980c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:01.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.815+0000 7f982bfff640 10 monclient: get_auth_request con 0x7f980c037810 auth_method 0 2024-09-16T05:27:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.819+0000 7f9828ff9640 10 client.134178.objecter ms_handle_connect 0x7f980c037810 2024-09-16T05:27:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.819+0000 7f982bfff640 1 --2- 172.21.15.114:0/3398917159 >> v2:172.21.15.149:6812/1185531135 conn(0x7f980c037810 0x7f980c039cd0 secure :-1 s=READY pgs=3631 cs=0 l=1 rev1=1 crypto rx=0x7f9824009ce0 tx=0x7f982400b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.819+0000 7f9828ff9640 1 -- 172.21.15.114:0/3398917159 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f981c016030 con 0x7f982c139640 2024-09-16T05:27:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.819+0000 7f9828ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:01.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.819+0000 7f9828ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:27:01.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.979+0000 7f9832341640 1 -- 172.21.15.114:0/3398917159 --> v2:172.21.15.149:6812/1185531135 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f982c0a0ec0 con 0x7f980c037810 2024-09-16T05:27:01.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.987+0000 7f9828ff9640 1 -- 172.21.15.114:0/3398917159 <== mgr.4108 v2:172.21.15.149:6812/1185531135 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114079 (secure 0 0 0) 0x7f982c0a0ec0 con 0x7f980c037810 2024-09-16T05:27:01.987 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:27:01.987 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-16T05:27:01.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 10 librados: watch_flush enter 2024-09-16T05:27:01.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 10 librados: watch_flush exit 2024-09-16T05:27:01.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 20 client.134178.objecter shutdown clearing up homeless session... 2024-09-16T05:27:01.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 10 client.134178.objecter successfully canceled tick 2024-09-16T05:27:01.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 1 -- 172.21.15.114:0/3398917159 >> v2:172.21.15.149:6812/1185531135 conn(0x7f980c037810 msgr2=0x7f980c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 1 --2- 172.21.15.114:0/3398917159 >> v2:172.21.15.149:6812/1185531135 conn(0x7f980c037810 0x7f980c039cd0 secure :-1 s=READY pgs=3631 cs=0 l=1 rev1=1 crypto rx=0x7f9824009ce0 tx=0x7f982400b040 comp rx=0 tx=0).stop 2024-09-16T05:27:01.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 10 monclient: shutdown 2024-09-16T05:27:01.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:01.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 1 -- 172.21.15.114:0/3398917159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f982c139640 msgr2=0x7f982c099f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:01.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 1 --2- 172.21.15.114:0/3398917159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f982c139640 0x7f982c099f60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f981c01a9d0 tx=0x7f981c0190c0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 1 -- 172.21.15.114:0/3398917159 shutdown_connections 2024-09-16T05:27:01.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 1 --2- 172.21.15.114:0/3398917159 >> v2:172.21.15.149:6812/1185531135 conn(0x7f980c037810 0x7f980c039cd0 unknown :-1 s=CLOSED pgs=3631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 1 --2- 172.21.15.114:0/3398917159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f982c139640 0x7f982c099f60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 1 --2- 172.21.15.114:0/3398917159 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0a1400 0x7f982c099a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 1 --2- 172.21.15.114:0/3398917159 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f982c09b6c0 0x7f982c0994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:01.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 1 -- 172.21.15.114:0/3398917159 >> 172.21.15.114:0/3398917159 conn(0x7f982c0a1fb0 msgr2=0x7f982c138a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:01.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 1 -- 172.21.15.114:0/3398917159 shutdown_connections 2024-09-16T05:27:01.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 1 -- 172.21.15.114:0/3398917159 wait complete. 2024-09-16T05:27:01.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:01.991+0000 7f9832341640 1 librados: shutdown 2024-09-16T05:27:02.005 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1266,"stamp":"2024-09-16T05:27:00.512220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":464520,"kb_used_data":60696,"kb_used_omap":9,"kb_used_meta":403574,"kb_avail":565766520,"statfs":{"total":579820584960,"available":579344916480,"internally_reserved":0,"allocated":62152704,"data_stored":32273216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":413260449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003811"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021853+0000","last_change":"2024-09-16T05:08:51.596823+0000","last_active":"2024-09-16T05:25:06.021853+0000","last_peered":"2024-09-16T05:25:06.021853+0000","last_clean":"2024-09-16T05:25:06.021853+0000","last_became_active":"2024-09-16T05:08:51.571632+0000","last_became_peered":"2024-09-16T05:08:51.571632+0000","last_unstale":"2024-09-16T05:25:06.021853+0000","last_undegraded":"2024-09-16T05:25:06.021853+0000","last_fullsized":"2024-09-16T05:25:06.021853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:16.107411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1324,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411274+0000","last_change":"2024-09-16T05:08:51.611318+0000","last_active":"2024-09-16T05:25:05.411274+0000","last_peered":"2024-09-16T05:25:05.411274+0000","last_clean":"2024-09-16T05:25:05.411274+0000","last_became_active":"2024-09-16T05:08:51.585360+0000","last_became_peered":"2024-09-16T05:08:51.585360+0000","last_unstale":"2024-09-16T05:25:05.411274+0000","last_undegraded":"2024-09-16T05:25:05.411274+0000","last_fullsized":"2024-09-16T05:25:05.411274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:19:58.506693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021814+0000","last_change":"2024-09-16T05:08:51.532106+0000","last_active":"2024-09-16T05:25:06.021814+0000","last_peered":"2024-09-16T05:25:06.021814+0000","last_clean":"2024-09-16T05:25:06.021814+0000","last_became_active":"2024-09-16T05:08:51.519558+0000","last_became_peered":"2024-09-16T05:08:51.519558+0000","last_unstale":"2024-09-16T05:25:06.021814+0000","last_undegraded":"2024-09-16T05:25:06.021814+0000","last_fullsized":"2024-09-16T05:25:06.021814+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:52:54.764452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1316,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.441285+0000","last_change":"2024-09-16T05:08:51.705649+0000","last_active":"2024-09-16T05:25:05.441285+0000","last_peered":"2024-09-16T05:25:05.441285+0000","last_clean":"2024-09-16T05:25:05.441285+0000","last_became_active":"2024-09-16T05:08:51.689931+0000","last_became_peered":"2024-09-16T05:08:51.689931+0000","last_unstale":"2024-09-16T05:25:05.441285+0000","last_undegraded":"2024-09-16T05:25:05.441285+0000","last_fullsized":"2024-09-16T05:25:05.441285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:31.971156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522829+0000","last_change":"2024-09-16T05:08:51.564493+0000","last_active":"2024-09-16T05:25:05.522829+0000","last_peered":"2024-09-16T05:25:05.522829+0000","last_clean":"2024-09-16T05:25:05.522829+0000","last_became_active":"2024-09-16T05:08:51.546851+0000","last_became_peered":"2024-09-16T05:08:51.546851+0000","last_unstale":"2024-09-16T05:25:05.522829+0000","last_undegraded":"2024-09-16T05:25:05.522829+0000","last_fullsized":"2024-09-16T05:25:05.522829+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:23.731218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.463123+0000","last_change":"2024-09-16T05:08:51.617848+0000","last_active":"2024-09-16T05:25:04.463123+0000","last_peered":"2024-09-16T05:25:04.463123+0000","last_clean":"2024-09-16T05:25:04.463123+0000","last_became_active":"2024-09-16T05:08:51.605843+0000","last_became_peered":"2024-09-16T05:08:51.605843+0000","last_unstale":"2024-09-16T05:25:04.463123+0000","last_undegraded":"2024-09-16T05:25:04.463123+0000","last_fullsized":"2024-09-16T05:25:04.463123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:24:14.376183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1317,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.441115+0000","last_change":"2024-09-16T05:08:51.615740+0000","last_active":"2024-09-16T05:25:05.441115+0000","last_peered":"2024-09-16T05:25:05.441115+0000","last_clean":"2024-09-16T05:25:05.441115+0000","last_became_active":"2024-09-16T05:08:51.609820+0000","last_became_peered":"2024-09-16T05:08:51.609820+0000","last_unstale":"2024-09-16T05:25:05.441115+0000","last_undegraded":"2024-09-16T05:25:05.441115+0000","last_fullsized":"2024-09-16T05:25:05.441115+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:35:22.111340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1322,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021670+0000","last_change":"2024-09-16T05:08:51.603805+0000","last_active":"2024-09-16T05:25:06.021670+0000","last_peered":"2024-09-16T05:25:06.021670+0000","last_clean":"2024-09-16T05:25:06.021670+0000","last_became_active":"2024-09-16T05:08:51.567268+0000","last_became_peered":"2024-09-16T05:08:51.567268+0000","last_unstale":"2024-09-16T05:25:06.021670+0000","last_undegraded":"2024-09-16T05:25:06.021670+0000","last_fullsized":"2024-09-16T05:25:06.021670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:43:35.723700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1325,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411521+0000","last_change":"2024-09-16T05:08:51.640189+0000","last_active":"2024-09-16T05:25:05.411521+0000","last_peered":"2024-09-16T05:25:05.411521+0000","last_clean":"2024-09-16T05:25:05.411521+0000","last_became_active":"2024-09-16T05:08:51.634768+0000","last_became_peered":"2024-09-16T05:08:51.634768+0000","last_unstale":"2024-09-16T05:25:05.411521+0000","last_undegraded":"2024-09-16T05:25:05.411521+0000","last_fullsized":"2024-09-16T05:25:05.411521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:52:08.194663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1323,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.735797+0000","last_change":"2024-09-16T05:08:51.589983+0000","last_active":"2024-09-16T05:25:04.735797+0000","last_peered":"2024-09-16T05:25:04.735797+0000","last_clean":"2024-09-16T05:25:04.735797+0000","last_became_active":"2024-09-16T05:08:51.589818+0000","last_became_peered":"2024-09-16T05:08:51.589818+0000","last_unstale":"2024-09-16T05:25:04.735797+0000","last_undegraded":"2024-09-16T05:25:04.735797+0000","last_fullsized":"2024-09-16T05:25:04.735797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:55:59.967412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.465088+0000","last_change":"2024-09-16T05:08:51.609659+0000","last_active":"2024-09-16T05:25:04.465088+0000","last_peered":"2024-09-16T05:25:04.465088+0000","last_clean":"2024-09-16T05:25:04.465088+0000","last_became_active":"2024-09-16T05:08:51.605719+0000","last_became_peered":"2024-09-16T05:08:51.605719+0000","last_unstale":"2024-09-16T05:25:04.465088+0000","last_undegraded":"2024-09-16T05:25:04.465088+0000","last_fullsized":"2024-09-16T05:25:04.465088+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:43:56.008078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1325,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411040+0000","last_change":"2024-09-16T05:08:51.630514+0000","last_active":"2024-09-16T05:25:05.411040+0000","last_peered":"2024-09-16T05:25:05.411040+0000","last_clean":"2024-09-16T05:25:05.411040+0000","last_became_active":"2024-09-16T05:08:51.611801+0000","last_became_peered":"2024-09-16T05:08:51.611801+0000","last_unstale":"2024-09-16T05:25:05.411040+0000","last_undegraded":"2024-09-16T05:25:05.411040+0000","last_fullsized":"2024-09-16T05:25:05.411040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:46.255904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522975+0000","last_change":"2024-09-16T05:08:51.605439+0000","last_active":"2024-09-16T05:25:05.522975+0000","last_peered":"2024-09-16T05:25:05.522975+0000","last_clean":"2024-09-16T05:25:05.522975+0000","last_became_active":"2024-09-16T05:08:51.586611+0000","last_became_peered":"2024-09-16T05:08:51.586611+0000","last_unstale":"2024-09-16T05:25:05.522975+0000","last_undegraded":"2024-09-16T05:25:05.522975+0000","last_fullsized":"2024-09-16T05:25:05.522975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:39:46.639778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":1492,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.410709+0000","last_change":"2024-09-16T05:07:39.580660+0000","last_active":"2024-09-16T05:25:05.410709+0000","last_peered":"2024-09-16T05:25:05.410709+0000","last_clean":"2024-09-16T05:25:05.410709+0000","last_became_active":"2024-09-16T05:07:39.576756+0000","last_became_peered":"2024-09-16T05:07:39.576756+0000","last_unstale":"2024-09-16T05:25:05.410709+0000","last_undegraded":"2024-09-16T05:25:05.410709+0000","last_fullsized":"2024-09-16T05:25:05.410709+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_clean_scrub_stamp":"2024-09-16T05:07:38.167879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:57.060906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1342,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411039+0000","last_change":"2024-09-16T05:07:48.267201+0000","last_active":"2024-09-16T05:25:05.411039+0000","last_peered":"2024-09-16T05:25:05.411039+0000","last_clean":"2024-09-16T05:25:05.411039+0000","last_became_active":"2024-09-16T05:07:46.246409+0000","last_became_peered":"2024-09-16T05:07:46.246409+0000","last_unstale":"2024-09-16T05:25:05.411039+0000","last_undegraded":"2024-09-16T05:25:05.411039+0000","last_fullsized":"2024-09-16T05:25:05.411039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:13.808827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010413149999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1323,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.735867+0000","last_change":"2024-09-16T05:08:51.582564+0000","last_active":"2024-09-16T05:25:04.735867+0000","last_peered":"2024-09-16T05:25:04.735867+0000","last_clean":"2024-09-16T05:25:04.735867+0000","last_became_active":"2024-09-16T05:08:51.582424+0000","last_became_peered":"2024-09-16T05:08:51.582424+0000","last_unstale":"2024-09-16T05:25:04.735867+0000","last_undegraded":"2024-09-16T05:25:04.735867+0000","last_fullsized":"2024-09-16T05:25:04.735867+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:43.882076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1333,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.441020+0000","last_change":"2024-09-16T05:07:48.266504+0000","last_active":"2024-09-16T05:25:05.441020+0000","last_peered":"2024-09-16T05:25:05.441020+0000","last_clean":"2024-09-16T05:25:05.441020+0000","last_became_active":"2024-09-16T05:07:46.254322+0000","last_became_peered":"2024-09-16T05:07:46.254322+0000","last_unstale":"2024-09-16T05:25:05.441020+0000","last_undegraded":"2024-09-16T05:25:05.441020+0000","last_fullsized":"2024-09-16T05:25:05.441020+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:00.175590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093680599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522665+0000","last_change":"2024-09-16T05:08:51.549772+0000","last_active":"2024-09-16T05:25:05.522665+0000","last_peered":"2024-09-16T05:25:05.522665+0000","last_clean":"2024-09-16T05:25:05.522665+0000","last_became_active":"2024-09-16T05:08:51.539573+0000","last_became_peered":"2024-09-16T05:08:51.539573+0000","last_unstale":"2024-09-16T05:25:05.522665+0000","last_undegraded":"2024-09-16T05:25:05.522665+0000","last_fullsized":"2024-09-16T05:25:05.522665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:10:54.876819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1362,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.462909+0000","last_change":"2024-09-16T05:07:48.263513+0000","last_active":"2024-09-16T05:25:04.462909+0000","last_peered":"2024-09-16T05:25:04.462909+0000","last_clean":"2024-09-16T05:25:04.462909+0000","last_became_active":"2024-09-16T05:07:46.252254+0000","last_became_peered":"2024-09-16T05:07:46.252254+0000","last_unstale":"2024-09-16T05:25:04.462909+0000","last_undegraded":"2024-09-16T05:25:04.462909+0000","last_fullsized":"2024-09-16T05:25:04.462909+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:33.736560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081560699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.465371+0000","last_change":"2024-09-16T05:08:51.542756+0000","last_active":"2024-09-16T05:25:04.465371+0000","last_peered":"2024-09-16T05:25:04.465371+0000","last_clean":"2024-09-16T05:25:04.465371+0000","last_became_active":"2024-09-16T05:08:51.541976+0000","last_became_peered":"2024-09-16T05:08:51.541976+0000","last_unstale":"2024-09-16T05:25:04.465371+0000","last_undegraded":"2024-09-16T05:25:04.465371+0000","last_fullsized":"2024-09-16T05:25:04.465371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:48:46.043655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":1347,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522619+0000","last_change":"2024-09-16T05:07:48.274329+0000","last_active":"2024-09-16T05:25:05.522619+0000","last_peered":"2024-09-16T05:25:05.522619+0000","last_clean":"2024-09-16T05:25:05.522619+0000","last_became_active":"2024-09-16T05:07:46.257118+0000","last_became_peered":"2024-09-16T05:07:46.257118+0000","last_unstale":"2024-09-16T05:25:05.522619+0000","last_undegraded":"2024-09-16T05:25:05.522619+0000","last_fullsized":"2024-09-16T05:25:05.522619+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:35.932772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020216650000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1323,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021532+0000","last_change":"2024-09-16T05:08:51.506142+0000","last_active":"2024-09-16T05:25:06.021532+0000","last_peered":"2024-09-16T05:25:06.021532+0000","last_clean":"2024-09-16T05:25:06.021532+0000","last_became_active":"2024-09-16T05:08:51.491760+0000","last_became_peered":"2024-09-16T05:08:51.491760+0000","last_unstale":"2024-09-16T05:25:06.021532+0000","last_undegraded":"2024-09-16T05:25:06.021532+0000","last_fullsized":"2024-09-16T05:25:06.021532+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:04:43.526278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1339,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.735843+0000","last_change":"2024-09-16T05:07:48.352028+0000","last_active":"2024-09-16T05:25:04.735843+0000","last_peered":"2024-09-16T05:25:04.735843+0000","last_clean":"2024-09-16T05:25:04.735843+0000","last_became_active":"2024-09-16T05:07:46.255633+0000","last_became_peered":"2024-09-16T05:07:46.255633+0000","last_unstale":"2024-09-16T05:25:04.735843+0000","last_undegraded":"2024-09-16T05:25:04.735843+0000","last_fullsized":"2024-09-16T05:25:04.735843+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:07.079219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000883379,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"21'7","reported_seq":1359,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.462678+0000","last_change":"2024-09-16T05:08:51.617951+0000","last_active":"2024-09-16T05:25:04.462678+0000","last_peered":"2024-09-16T05:25:04.462678+0000","last_clean":"2024-09-16T05:25:04.462678+0000","last_became_active":"2024-09-16T05:08:51.614284+0000","last_became_peered":"2024-09-16T05:08:51.614284+0000","last_unstale":"2024-09-16T05:25:04.462678+0000","last_undegraded":"2024-09-16T05:25:04.462678+0000","last_fullsized":"2024-09-16T05:25:04.462678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:18.960138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1341,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411163+0000","last_change":"2024-09-16T05:07:48.267372+0000","last_active":"2024-09-16T05:25:05.411163+0000","last_peered":"2024-09-16T05:25:05.411163+0000","last_clean":"2024-09-16T05:25:05.411163+0000","last_became_active":"2024-09-16T05:07:46.243891+0000","last_became_peered":"2024-09-16T05:07:46.243891+0000","last_unstale":"2024-09-16T05:25:05.411163+0000","last_undegraded":"2024-09-16T05:25:05.411163+0000","last_fullsized":"2024-09-16T05:25:05.411163+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:48:25.061723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011829709999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1325,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.410924+0000","last_change":"2024-09-16T05:08:51.634776+0000","last_active":"2024-09-16T05:25:05.410924+0000","last_peered":"2024-09-16T05:25:05.410924+0000","last_clean":"2024-09-16T05:25:05.410924+0000","last_became_active":"2024-09-16T05:08:51.632213+0000","last_became_peered":"2024-09-16T05:08:51.632213+0000","last_unstale":"2024-09-16T05:25:05.410924+0000","last_undegraded":"2024-09-16T05:25:05.410924+0000","last_fullsized":"2024-09-16T05:25:05.410924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:47:43.409907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1337,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021856+0000","last_change":"2024-09-16T05:07:48.275545+0000","last_active":"2024-09-16T05:25:06.021856+0000","last_peered":"2024-09-16T05:25:06.021856+0000","last_clean":"2024-09-16T05:25:06.021856+0000","last_became_active":"2024-09-16T05:07:46.252207+0000","last_became_peered":"2024-09-16T05:07:46.252207+0000","last_unstale":"2024-09-16T05:25:06.021856+0000","last_undegraded":"2024-09-16T05:25:06.021856+0000","last_fullsized":"2024-09-16T05:25:06.021856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:36:08.378166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073333700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522617+0000","last_change":"2024-09-16T05:08:51.499333+0000","last_active":"2024-09-16T05:25:05.522617+0000","last_peered":"2024-09-16T05:25:05.522617+0000","last_clean":"2024-09-16T05:25:05.522617+0000","last_became_active":"2024-09-16T05:08:51.486807+0000","last_became_peered":"2024-09-16T05:08:51.486807+0000","last_unstale":"2024-09-16T05:25:05.522617+0000","last_undegraded":"2024-09-16T05:25:05.522617+0000","last_fullsized":"2024-09-16T05:25:05.522617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:43.084461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1340,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411030+0000","last_change":"2024-09-16T05:07:48.267914+0000","last_active":"2024-09-16T05:25:05.411030+0000","last_peered":"2024-09-16T05:25:05.411030+0000","last_clean":"2024-09-16T05:25:05.411030+0000","last_became_active":"2024-09-16T05:07:46.254034+0000","last_became_peered":"2024-09-16T05:07:46.254034+0000","last_unstale":"2024-09-16T05:25:05.411030+0000","last_undegraded":"2024-09-16T05:25:05.411030+0000","last_fullsized":"2024-09-16T05:25:05.411030+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:57:37.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014490219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021562+0000","last_change":"2024-09-16T05:08:51.548570+0000","last_active":"2024-09-16T05:25:06.021562+0000","last_peered":"2024-09-16T05:25:06.021562+0000","last_clean":"2024-09-16T05:25:06.021562+0000","last_became_active":"2024-09-16T05:08:51.532685+0000","last_became_peered":"2024-09-16T05:08:51.532685+0000","last_unstale":"2024-09-16T05:25:06.021562+0000","last_undegraded":"2024-09-16T05:25:06.021562+0000","last_fullsized":"2024-09-16T05:25:06.021562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:14:25.488128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021522+0000","last_change":"2024-09-16T05:08:51.604786+0000","last_active":"2024-09-16T05:25:06.021522+0000","last_peered":"2024-09-16T05:25:06.021522+0000","last_clean":"2024-09-16T05:25:06.021522+0000","last_became_active":"2024-09-16T05:08:51.567861+0000","last_became_peered":"2024-09-16T05:08:51.567861+0000","last_unstale":"2024-09-16T05:25:06.021522+0000","last_undegraded":"2024-09-16T05:25:06.021522+0000","last_fullsized":"2024-09-16T05:25:06.021522+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:48:14.930571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1345,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.463346+0000","last_change":"2024-09-16T05:08:51.563783+0000","last_active":"2024-09-16T05:25:04.463346+0000","last_peered":"2024-09-16T05:25:04.463346+0000","last_clean":"2024-09-16T05:25:04.463346+0000","last_became_active":"2024-09-16T05:08:51.563420+0000","last_became_peered":"2024-09-16T05:08:51.563420+0000","last_unstale":"2024-09-16T05:25:04.463346+0000","last_undegraded":"2024-09-16T05:25:04.463346+0000","last_fullsized":"2024-09-16T05:25:04.463346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:52.527569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1328,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522885+0000","last_change":"2024-09-16T05:08:51.513817+0000","last_active":"2024-09-16T05:25:05.522885+0000","last_peered":"2024-09-16T05:25:05.522885+0000","last_clean":"2024-09-16T05:25:05.522885+0000","last_became_active":"2024-09-16T05:08:51.503720+0000","last_became_peered":"2024-09-16T05:08:51.503720+0000","last_unstale":"2024-09-16T05:25:05.522885+0000","last_undegraded":"2024-09-16T05:25:05.522885+0000","last_fullsized":"2024-09-16T05:25:05.522885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:31.884629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021857+0000","last_change":"2024-09-16T05:08:51.505598+0000","last_active":"2024-09-16T05:25:06.021857+0000","last_peered":"2024-09-16T05:25:06.021857+0000","last_clean":"2024-09-16T05:25:06.021857+0000","last_became_active":"2024-09-16T05:08:51.494038+0000","last_became_peered":"2024-09-16T05:08:51.494038+0000","last_unstale":"2024-09-16T05:25:06.021857+0000","last_undegraded":"2024-09-16T05:25:06.021857+0000","last_fullsized":"2024-09-16T05:25:06.021857+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:39:28.303932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1324,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411448+0000","last_change":"2024-09-16T05:08:51.680865+0000","last_active":"2024-09-16T05:25:05.411448+0000","last_peered":"2024-09-16T05:25:05.411448+0000","last_clean":"2024-09-16T05:25:05.411448+0000","last_became_active":"2024-09-16T05:08:51.680588+0000","last_became_peered":"2024-09-16T05:08:51.680588+0000","last_unstale":"2024-09-16T05:25:05.411448+0000","last_undegraded":"2024-09-16T05:25:05.411448+0000","last_fullsized":"2024-09-16T05:25:05.411448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:52.176771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.463756+0000","last_change":"2024-09-16T05:08:51.490785+0000","last_active":"2024-09-16T05:25:04.463756+0000","last_peered":"2024-09-16T05:25:04.463756+0000","last_clean":"2024-09-16T05:25:04.463756+0000","last_became_active":"2024-09-16T05:08:51.480748+0000","last_became_peered":"2024-09-16T05:08:51.480748+0000","last_unstale":"2024-09-16T05:25:04.463756+0000","last_undegraded":"2024-09-16T05:25:04.463756+0000","last_fullsized":"2024-09-16T05:25:04.463756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:21:24.554888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1325,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.411671+0000","last_change":"2024-09-16T05:08:51.623309+0000","last_active":"2024-09-16T05:25:05.411671+0000","last_peered":"2024-09-16T05:25:05.411671+0000","last_clean":"2024-09-16T05:25:05.411671+0000","last_became_active":"2024-09-16T05:08:51.622941+0000","last_became_peered":"2024-09-16T05:08:51.622941+0000","last_unstale":"2024-09-16T05:25:05.411671+0000","last_undegraded":"2024-09-16T05:25:05.411671+0000","last_fullsized":"2024-09-16T05:25:05.411671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:00:12.586886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1328,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522520+0000","last_change":"2024-09-16T05:08:51.493220+0000","last_active":"2024-09-16T05:25:05.522520+0000","last_peered":"2024-09-16T05:25:05.522520+0000","last_clean":"2024-09-16T05:25:05.522520+0000","last_became_active":"2024-09-16T05:08:51.486048+0000","last_became_peered":"2024-09-16T05:08:51.486048+0000","last_unstale":"2024-09-16T05:25:05.522520+0000","last_undegraded":"2024-09-16T05:25:05.522520+0000","last_fullsized":"2024-09-16T05:25:05.522520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:52:13.473044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021819+0000","last_change":"2024-09-16T05:08:51.472111+0000","last_active":"2024-09-16T05:25:06.021819+0000","last_peered":"2024-09-16T05:25:06.021819+0000","last_clean":"2024-09-16T05:25:06.021819+0000","last_became_active":"2024-09-16T05:08:51.469556+0000","last_became_peered":"2024-09-16T05:08:51.469556+0000","last_unstale":"2024-09-16T05:25:06.021819+0000","last_undegraded":"2024-09-16T05:25:06.021819+0000","last_fullsized":"2024-09-16T05:25:06.021819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:14.956562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1345,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.462705+0000","last_change":"2024-09-16T05:08:51.533742+0000","last_active":"2024-09-16T05:25:04.462705+0000","last_peered":"2024-09-16T05:25:04.462705+0000","last_clean":"2024-09-16T05:25:04.462705+0000","last_became_active":"2024-09-16T05:08:51.533473+0000","last_became_peered":"2024-09-16T05:08:51.533473+0000","last_unstale":"2024-09-16T05:25:04.462705+0000","last_undegraded":"2024-09-16T05:25:04.462705+0000","last_fullsized":"2024-09-16T05:25:04.462705+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:21.233490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1328,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522976+0000","last_change":"2024-09-16T05:08:51.509275+0000","last_active":"2024-09-16T05:25:05.522976+0000","last_peered":"2024-09-16T05:25:05.522976+0000","last_clean":"2024-09-16T05:25:05.522976+0000","last_became_active":"2024-09-16T05:08:51.493446+0000","last_became_peered":"2024-09-16T05:08:51.493446+0000","last_unstale":"2024-09-16T05:25:05.522976+0000","last_undegraded":"2024-09-16T05:25:05.522976+0000","last_fullsized":"2024-09-16T05:25:05.522976+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:19.908189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834575083,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73084,"kb_used_data":9940,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298756,"statfs":{"total":96636764160,"available":96561926144,"internally_reserved":0,"allocated":10178560,"data_stored":5202552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575084,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77260,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294580,"statfs":{"total":96636764160,"available":96557649920,"internally_reserved":0,"allocated":10182656,"data_stored":5202571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607787,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73604,"kb_used_data":10460,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298236,"statfs":{"total":96636764160,"available":96561393664,"internally_reserved":0,"allocated":10711040,"data_stored":5731480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607787,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73608,"kb_used_data":10464,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298232,"statfs":{"total":96636764160,"available":96561389568,"internally_reserved":0,"allocated":10715136,"data_stored":5731485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640492,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77248,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294592,"statfs":{"total":96636764160,"available":96557662208,"internally_reserved":0,"allocated":10182656,"data_stored":5202571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640492,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89716,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94282124,"statfs":{"total":96636764160,"available":96544894976,"internally_reserved":0,"allocated":10182656,"data_stored":5202557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81656263},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:27:02.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T05:27:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 10 monclient: get_monmap_and_config 2024-09-16T05:27:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 10 monclient: build_initial_monmap 2024-09-16T05:27:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 10 monclient: monmap: 2024-09-16T05:27:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:02.125801+0000 2024-09-16T05:27:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:02.125801+0000 2024-09-16T05:27:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:02.121 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:02.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 1 Processor -- start 2024-09-16T05:27:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 1 -- start start 2024-09-16T05:27:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 10 monclient: init 2024-09-16T05:27:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T05:27:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fccd009b6c0 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fccd00a15d0 0x7fccd013a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 10 monclient(hunting): picked mon.noname-a con 0x7fccd00a15d0 addr v2:172.21.15.114:3300/0 2024-09-16T05:27:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fccd00a19e0 addr v1:172.21.15.149:6789/0 2024-09-16T05:27:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccd009d4e0 con 0x7fccd009b6c0 2024-09-16T05:27:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccd009f430 con 0x7fccd00a19e0 2024-09-16T05:27:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fccd009e390 con 0x7fccd00a15d0 2024-09-16T05:27:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fccd6db6640 10 monclient(hunting): authenticate will time out at 2106.754882s 2024-09-16T05:27:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fccd00a15d0 0x7fccd013a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fccd00a15d0 0x7fccd013a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35160/0 (socket says 172.21.15.114:35160) 2024-09-16T05:27:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 1 -- 172.21.15.114:0/1947685634 learned_addr learned my addr 172.21.15.114:0/1947685634 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 10 monclient(hunting): get_auth_request con 0x7fccd00a15d0 auth_method 0 2024-09-16T05:27:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 10 monclient(hunting): handle_auth_done global_id 134181 payload 274 2024-09-16T05:27:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 1 -- 172.21.15.114:0/1947685634 >> v1:172.21.15.149:6789/0 conn(0x7fccd00a19e0 legacy=0x7fccd013cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:02.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 1 -- 172.21.15.114:0/1947685634 >> v1:172.21.15.114:6789/0 conn(0x7fccd009b6c0 legacy=0x7fccd00a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:02.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 10 monclient: _finish_hunting 0 2024-09-16T05:27:02.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 1 monclient: found mon.noname-a 2024-09-16T05:27:02.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:02.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:02.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.123+0000 7fcccffff640 1 -- 172.21.15.114:0/1947685634 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd009c2f0 con 0x7fccd00a15d0 2024-09-16T05:27:02.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccffff640 1 --2- 172.21.15.114:0/1947685634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd00a15d0 0x7fccd013a650 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fccd009c630 tx=0x7fccc001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb902d6d7f14687 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:02.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccf7fe640 1 -- 172.21.15.114:0/1947685634 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc00b0070 con 0x7fccd00a15d0 2024-09-16T05:27:02.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccf7fe640 10 monclient: dump: 2024-09-16T05:27:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:02.131 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccf7fe640 10 monclient: _finish_auth 0 2024-09-16T05:27:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccf7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:27:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 5 monclient: authenticate success, global_id 134181 2024-09-16T05:27:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccf7fe640 1 -- 172.21.15.114:0/1947685634 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccc00ab600 con 0x7fccd00a15d0 2024-09-16T05:27:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccf7fe640 1 -- 172.21.15.114:0/1947685634 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc00ab9d0 con 0x7fccd00a15d0 2024-09-16T05:27:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 4 set_mon_vals no callback set 2024-09-16T05:27:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccf7fe640 10 monclient: dump: 2024-09-16T05:27:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:02.133 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 10 monclient: shutdown 2024-09-16T05:27:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 -- 172.21.15.114:0/1947685634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd00a15d0 msgr2=0x7fccd013a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 --2- 172.21.15.114:0/1947685634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd00a15d0 0x7fccd013a650 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fccd009c630 tx=0x7fccc001c410 comp rx=0 tx=0).stop 2024-09-16T05:27:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 -- 172.21.15.114:0/1947685634 shutdown_connections 2024-09-16T05:27:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 --2- 172.21.15.114:0/1947685634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd00a15d0 0x7fccd013a650 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 -- 172.21.15.114:0/1947685634 >> 172.21.15.114:0/1947685634 conn(0x7fccd00a1fb0 msgr2=0x7fccd00a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 -- 172.21.15.114:0/1947685634 shutdown_connections 2024-09-16T05:27:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 -- 172.21.15.114:0/1947685634 wait complete. 2024-09-16T05:27:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 10 monclient: build_initial_monmap 2024-09-16T05:27:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 10 monclient: monmap: 2024-09-16T05:27:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:02.135 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 librados: starting msgr at 2024-09-16T05:27:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 librados: starting objecter 2024-09-16T05:27:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 Processor -- start 2024-09-16T05:27:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 -- start start 2024-09-16T05:27:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 librados: setting wanted keys 2024-09-16T05:27:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 librados: calling monclient init 2024-09-16T05:27:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 10 monclient: init 2024-09-16T05:27:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:27:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccd009b6c0 0x7fccd0098640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 10 monclient(hunting): picked mon.noname-c con 0x7fccd009b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fccd00a15d0 0x7fccd0091c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 10 monclient(hunting): picked mon.noname-b con 0x7fccd00a15d0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd00a19e0 0x7fccd00921d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 10 monclient(hunting): picked mon.noname-a con 0x7fccd00a19e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fccd0140c60 con 0x7fccd00a19e0 2024-09-16T05:27:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fccd0140ae0 con 0x7fccd00a15d0 2024-09-16T05:27:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fccd0140de0 con 0x7fccd009b6c0 2024-09-16T05:27:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd6db6640 10 monclient(hunting): authenticate will time out at 2106.759584s 2024-09-16T05:27:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd5353640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd00a19e0 0x7fccd00921d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd5353640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd00a19e0 0x7fccd00921d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35168/0 (socket says 172.21.15.114:35168) 2024-09-16T05:27:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd5353640 1 -- 172.21.15.114:0/3129215125 learned_addr learned my addr 172.21.15.114:0/3129215125 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd5353640 10 monclient(hunting): get_auth_request con 0x7fccd00a19e0 auth_method 0 2024-09-16T05:27:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd5353640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd5353640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd5353640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd4b52640 1 --2- 172.21.15.114:0/3129215125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccd009b6c0 0x7fccd0098640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd4b52640 10 monclient(hunting): get_auth_request con 0x7fccd009b6c0 auth_method 0 2024-09-16T05:27:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd4b52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd4b52640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd4b52640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd5353640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd5353640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd5353640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd4b52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd4b52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccffff640 1 --2- 172.21.15.114:0/3129215125 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fccd00a15d0 0x7fccd0091c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd4b52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccffff640 10 monclient(hunting): get_auth_request con 0x7fccd00a15d0 auth_method 0 2024-09-16T05:27:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fcccffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:02.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.127+0000 7fccd5353640 10 monclient(hunting): handle_auth_done global_id 134184 payload 824 2024-09-16T05:27:02.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd5353640 1 -- 172.21.15.114:0/3129215125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccd009b6c0 msgr2=0x7fccd0098640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:02.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd5353640 1 --2- 172.21.15.114:0/3129215125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccd009b6c0 0x7fccd0098640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:02.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd5353640 1 -- 172.21.15.114:0/3129215125 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fccd00a15d0 msgr2=0x7fccd0091c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:02.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd5353640 1 --2- 172.21.15.114:0/3129215125 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fccd00a15d0 0x7fccd0091c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:02.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd5353640 10 monclient: _finish_hunting 0 2024-09-16T05:27:02.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd5353640 1 monclient: found mon.noname-a 2024-09-16T05:27:02.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd5353640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:02.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd5353640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:02.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd5353640 1 -- 172.21.15.114:0/3129215125 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd0095940 con 0x7fccd00a19e0 2024-09-16T05:27:02.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd4b52640 1 --2- 172.21.15.114:0/3129215125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccd009b6c0 0x7fccd0098640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:27:02.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd5353640 1 --2- 172.21.15.114:0/3129215125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd00a19e0 0x7fccd00921d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fccc800c7b0 tx=0x7fccc800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:02.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 10 client.?.objecter ms_handle_connect 0x7fccd00a19e0 2024-09-16T05:27:02.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:02.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 1 -- 172.21.15.114:0/3129215125 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc8017070 con 0x7fccd00a19e0 2024-09-16T05:27:02.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:02.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:02.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 10 monclient: dump: 2024-09-16T05:27:02.140 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:02.140 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:02.140 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:02.140 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:02.140 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:02.140 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:02.140 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:02.140 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:02.140 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:02.141 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:02.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 10 monclient: _finish_auth 0 2024-09-16T05:27:02.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:27:02.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:02.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 1 -- 172.21.15.114:0/3129215125 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccc8003db0 con 0x7fccd00a19e0 2024-09-16T05:27:02.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd6db6640 5 monclient: authenticate success, global_id 134184 2024-09-16T05:27:02.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd6db6640 10 monclient: _renew_subs 2024-09-16T05:27:02.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd6db6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:02.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd6db6640 1 -- 172.21.15.114:0/3129215125 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccd0095c20 con 0x7fccd00a19e0 2024-09-16T05:27:02.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:02.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 1 -- 172.21.15.114:0/3129215125 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc800ce60 con 0x7fccd00a19e0 2024-09-16T05:27:02.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:02.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:02.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 10 monclient: dump: 2024-09-16T05:27:02.142 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:02.142 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:02.142 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:02.142 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:02.142 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:02.142 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:02.142 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:02.142 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:02.142 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:02.142 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:02.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd6db6640 10 client.134184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:02.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd6db6640 10 monclient: _renew_subs 2024-09-16T05:27:02.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd6db6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:02.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccceffd640 4 set_mon_vals no callback set 2024-09-16T05:27:02.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd6db6640 1 -- 172.21.15.114:0/3129215125 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccd0095660 con 0x7fccd00a19e0 2024-09-16T05:27:02.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd6db6640 1 librados: init done 2024-09-16T05:27:02.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd6db6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:02.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd6db6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:02.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd6db6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:02.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd6db6640 1 -- 172.21.15.114:0/3129215125 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccd00a0ec0 con 0x7fccd00a19e0 2024-09-16T05:27:02.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 1 -- 172.21.15.114:0/3129215125 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fccc801a070 con 0x7fccd00a19e0 2024-09-16T05:27:02.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 1 --2- 172.21.15.114:0/3129215125 >> v2:172.21.15.149:6812/1185531135 conn(0x7fccbc037810 0x7fccbc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:02.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 1 -- 172.21.15.114:0/3129215125 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7fccc8048130 con 0x7fccd00a19e0 2024-09-16T05:27:02.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 10 client.134184.objecter ms_dispatch 0x7fccd0000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:02.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 3 client.134184.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:02.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 3 client.134184.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:02.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 20 client.134184.objecter dump_active .. 0 homeless 2024-09-16T05:27:02.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd4b52640 1 --2- 172.21.15.114:0/3129215125 >> v2:172.21.15.149:6812/1185531135 conn(0x7fccbc037810 0x7fccbc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:02.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd4b52640 10 monclient: get_auth_request con 0x7fccbc037810 auth_method 0 2024-09-16T05:27:02.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fcccd7fa640 10 client.134184.objecter ms_handle_connect 0x7fccbc037810 2024-09-16T05:27:02.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.131+0000 7fccd4b52640 1 --2- 172.21.15.114:0/3129215125 >> v2:172.21.15.149:6812/1185531135 conn(0x7fccbc037810 0x7fccbc039cd0 secure :-1 s=READY pgs=3632 cs=0 l=1 rev1=1 crypto rx=0x7fccc4009cd0 tx=0x7fccc4009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:02.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.135+0000 7fcccd7fa640 1 -- 172.21.15.114:0/3129215125 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fccc8005770 con 0x7fccd00a19e0 2024-09-16T05:27:02.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.135+0000 7fcccd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:02.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.135+0000 7fcccd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:27:02.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.299+0000 7fccd6db6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:27:02.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.299+0000 7fccd6db6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:27:02.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.299+0000 7fccd6db6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:02.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.299+0000 7fccd6db6640 1 -- 172.21.15.114:0/3129215125 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fccd0094210 con 0x7fccd00a19e0 2024-09-16T05:27:02.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.299+0000 7fcccd7fa640 1 -- 172.21.15.114:0/3129215125 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v695) ==== 74+0+14064 (secure 0 0 0) 0x7fccc8005490 con 0x7fccd00a19e0 2024-09-16T05:27:02.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.299+0000 7fcccd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:27:02.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.299+0000 7fcccd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:27:02.297 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:27:02.297 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":695,"fsid":"c6dbfe9e-ab8c-4165-ae76-3f2273766e2c","created":"2024-09-16T05:07:27.305861+0000","modified":"2024-09-16T05:25:04.445692+0000","last_up_change":"2024-09-16T05:07:39.180058+0000","last_in_change":"2024-09-16T05:07:29.028596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":97,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":341,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:07:38.142989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T05:07:44.273942+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-16T05:08:49.485710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6ffdab41-0191-4764-ad62-922ec1091569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":690,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":269070361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":269070361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6811","nonce":269070361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":269070361}]},"public_addr":"172.21.15.114:6808/269070361","cluster_addr":"172.21.15.114:6809/269070361","heartbeat_back_addr":"172.21.15.114:6811/269070361","heartbeat_front_addr":"172.21.15.114:6810/269070361","state":["exists","up"]},{"osd":1,"uuid":"7de579ee-87a3-4463-87a5-6980b56cd113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":690,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2903720358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":2903720358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6803","nonce":2903720358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2903720358}]},"public_addr":"172.21.15.114:6800/2903720358","cluster_addr":"172.21.15.114:6801/2903720358","heartbeat_back_addr":"172.21.15.114:6803/2903720358","heartbeat_front_addr":"172.21.15.114:6802/2903720358","state":["exists","up"]},{"osd":2,"uuid":"258c8bf4-58c5-4c13-a0e5-ef9931efd698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":690,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1932657732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":1932657732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6807","nonce":1932657732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1932657732}]},"public_addr":"172.21.15.114:6804/1932657732","cluster_addr":"172.21.15.114:6805/1932657732","heartbeat_back_addr":"172.21.15.114:6807/1932657732","heartbeat_front_addr":"172.21.15.114:6806/1932657732","state":["exists","up"]},{"osd":3,"uuid":"acc1e03f-8a6c-4c85-9525-88707597d732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":690,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6805","nonce":1362772130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6807","nonce":1362772130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6811","nonce":1362772130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6809","nonce":1362772130}]},"public_addr":"172.21.15.149:6805/1362772130","cluster_addr":"172.21.15.149:6807/1362772130","heartbeat_back_addr":"172.21.15.149:6811/1362772130","heartbeat_front_addr":"172.21.15.149:6809/1362772130","state":["exists","up"]},{"osd":4,"uuid":"92cca4cf-19e9-44a1-9588-7c301a2c2065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":690,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":1092505261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6801","nonce":1092505261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6803","nonce":1092505261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":1092505261}]},"public_addr":"172.21.15.149:6800/1092505261","cluster_addr":"172.21.15.149:6801/1092505261","heartbeat_back_addr":"172.21.15.149:6803/1092505261","heartbeat_front_addr":"172.21.15.149:6802/1092505261","state":["exists","up"]},{"osd":5,"uuid":"ac717f4e-83fc-46ea-9816-7e35ef75a674","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":690,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":1244703827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":1244703827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":1244703827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":1244703827}]},"public_addr":"172.21.15.149:6804/1244703827","cluster_addr":"172.21.15.149:6806/1244703827","heartbeat_back_addr":"172.21.15.149:6810/1244703827","heartbeat_front_addr":"172.21.15.149:6808/1244703827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.554731+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.453254+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.446266+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:35.709335+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.069934+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:07:36.433872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":14,"snaps":[{"begin":2,"length":5}]},{"pool":15,"snaps":[{"begin":1,"length":4}]},{"pool":26,"snaps":[{"begin":1,"length":1}]},{"pool":66,"snaps":[{"begin":1,"length":1}]},{"pool":92,"snaps":[{"begin":4,"length":1},{"begin":7,"length":1},{"begin":9,"length":1}]},{"pool":103,"snaps":[{"begin":1,"length":1}]},{"pool":122,"snaps":[{"begin":1,"length":2}]},{"pool":140,"snaps":[{"begin":3,"length":1}]},{"pool":147,"snaps":[{"begin":2,"length":3}]},{"pool":152,"snaps":[{"begin":2,"length":1}]},{"pool":153,"snaps":[{"begin":2,"length":2}]},{"pool":164,"snaps":[{"begin":2,"length":3}]},{"pool":169,"snaps":[{"begin":3,"length":4}]},{"pool":173,"snaps":[{"begin":2,"length":4}]},{"pool":175,"snaps":[{"begin":2,"length":2}]},{"pool":183,"snaps":[{"begin":4,"length":1}]},{"pool":184,"snaps":[{"begin":2,"length":1}]},{"pool":189,"snaps":[{"begin":1,"length":4}]},{"pool":196,"snaps":[{"begin":7,"length":1}]},{"pool":205,"snaps":[{"begin":2,"length":1},{"begin":4,"length":4}]},{"pool":223,"snaps":[{"begin":8,"length":1}]},{"pool":234,"snaps":[{"begin":10,"length":1}]},{"pool":240,"snaps":[{"begin":11,"length":2}]},{"pool":256,"snaps":[{"begin":14,"length":1}]},{"pool":322,"snaps":[{"begin":2,"length":1}]},{"pool":323,"snaps":[{"begin":3,"length":1}]},{"pool":326,"snaps":[{"begin":4,"length":1}]},{"pool":330,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:27:02.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 10 librados: watch_flush enter 2024-09-16T05:27:02.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 10 librados: watch_flush exit 2024-09-16T05:27:02.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 20 client.134184.objecter shutdown clearing up homeless session... 2024-09-16T05:27:02.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 10 client.134184.objecter successfully canceled tick 2024-09-16T05:27:02.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 1 -- 172.21.15.114:0/3129215125 >> v2:172.21.15.149:6812/1185531135 conn(0x7fccbc037810 msgr2=0x7fccbc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:02.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 1 --2- 172.21.15.114:0/3129215125 >> v2:172.21.15.149:6812/1185531135 conn(0x7fccbc037810 0x7fccbc039cd0 secure :-1 s=READY pgs=3632 cs=0 l=1 rev1=1 crypto rx=0x7fccc4009cd0 tx=0x7fccc4009060 comp rx=0 tx=0).stop 2024-09-16T05:27:02.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 10 monclient: shutdown 2024-09-16T05:27:02.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:02.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 1 -- 172.21.15.114:0/3129215125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd00a19e0 msgr2=0x7fccd00921d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:02.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 1 --2- 172.21.15.114:0/3129215125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd00a19e0 0x7fccd00921d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fccc800c7b0 tx=0x7fccc800cc70 comp rx=0 tx=0).stop 2024-09-16T05:27:02.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 1 -- 172.21.15.114:0/3129215125 shutdown_connections 2024-09-16T05:27:02.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 1 --2- 172.21.15.114:0/3129215125 >> v2:172.21.15.149:6812/1185531135 conn(0x7fccbc037810 0x7fccbc039cd0 unknown :-1 s=CLOSED pgs=3632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:02.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 1 --2- 172.21.15.114:0/3129215125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccd00a19e0 0x7fccd00921d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:02.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 1 --2- 172.21.15.114:0/3129215125 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fccd00a15d0 0x7fccd0091c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:02.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 1 --2- 172.21.15.114:0/3129215125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccd009b6c0 0x7fccd0098640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:02.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 1 -- 172.21.15.114:0/3129215125 >> 172.21.15.114:0/3129215125 conn(0x7fccd00a1fb0 msgr2=0x7fccd0071330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:02.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 1 -- 172.21.15.114:0/3129215125 shutdown_connections 2024-09-16T05:27:02.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 1 -- 172.21.15.114:0/3129215125 wait complete. 2024-09-16T05:27:02.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:02.303+0000 7fccd6db6640 1 librados: shutdown 2024-09-16T05:27:03.316 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T05:27:03.317 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T05:27:03.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 10 monclient: get_monmap_and_config 2024-09-16T05:27:03.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 10 monclient: build_initial_monmap 2024-09-16T05:27:03.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 10 monclient: monmap: 2024-09-16T05:27:03.456 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:03.456 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.456 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:03.460986+0000 2024-09-16T05:27:03.456 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:03.460986+0000 2024-09-16T05:27:03.456 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:03.456 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.456 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:03.456 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:03.456 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:03.456 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:03.456 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:03.456 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 1 Processor -- start 2024-09-16T05:27:03.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 1 -- start start 2024-09-16T05:27:03.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 10 monclient: init 2024-09-16T05:27:03.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:03.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-16T05:27:03.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f86380a11a0 0x7f863809ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 10 monclient(hunting): picked mon.noname-a con 0x7f86380a11a0 addr v2:172.21.15.114:3300/0 2024-09-16T05:27:03.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f863809cfd0 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:03.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f863809d3e0 0x7f863813ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 10 monclient(hunting): picked mon.noname-b con 0x7f863809d3e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:03.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f863809df90 con 0x7f863809cfd0 2024-09-16T05:27:03.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f86380a1580 con 0x7f86380a11a0 2024-09-16T05:27:03.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f863809dcb0 con 0x7f863809d3e0 2024-09-16T05:27:03.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:03.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863f937640 10 monclient(hunting): authenticate will time out at 2108.090732s 2024-09-16T05:27:03.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f863809d3e0 0x7f863813ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ced2640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f863809cfd0 0x7f8638139720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37626/0 (socket says 172.21.15.114:37626) 2024-09-16T05:27:03.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ced2640 1 -- 172.21.15.114:0/3654963049 learned_addr learned my addr 172.21.15.114:0/3654963049 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:27:03.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 10 monclient(hunting): get_auth_request con 0x7f863809d3e0 auth_method 0 2024-09-16T05:27:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863d6d3640 1 --2- 172.21.15.114:0/3654963049 >> v2:172.21.15.114:3300/0 conn(0x7f86380a11a0 0x7f863809ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863d6d3640 10 monclient(hunting): get_auth_request con 0x7f86380a11a0 auth_method 0 2024-09-16T05:27:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863d6d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863d6d3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863d6d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863d6d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863d6d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863d6d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 10 monclient(hunting): handle_auth_done global_id 104414 payload 274 2024-09-16T05:27:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 1 -- 172.21.15.114:0/3654963049 >> v2:172.21.15.114:3300/0 conn(0x7f86380a11a0 msgr2=0x7f863809ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 1 --2- 172.21.15.114:0/3654963049 >> v2:172.21.15.114:3300/0 conn(0x7f86380a11a0 0x7f863809ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 1 -- 172.21.15.114:0/3654963049 >> v1:172.21.15.114:6789/0 conn(0x7f863809cfd0 legacy=0x7f8638139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 10 monclient: _finish_hunting 0 2024-09-16T05:27:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 1 monclient: found mon.noname-b 2024-09-16T05:27:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:27:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.459+0000 7f863ded4640 1 -- 172.21.15.114:0/3654963049 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f863809d850 con 0x7f863809d3e0 2024-09-16T05:27:03.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863ded4640 1 --2- 172.21.15.114:0/3654963049 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f863809d3e0 0x7f863813ce50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8630009d90 tx=0x7f863001c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa7b263d6c2e5958 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:03.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f862ffff640 1 -- 172.21.15.114:0/3654963049 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86300b2070 con 0x7f863809d3e0 2024-09-16T05:27:03.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f862ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:03.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f862ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:27:03.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f862ffff640 10 monclient: dump: 2024-09-16T05:27:03.468 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:03.468 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.468 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.469 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.469 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:03.469 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.469 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:03.469 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:03.469 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:03.469 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f862ffff640 10 monclient: _finish_auth 0 2024-09-16T05:27:03.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f862ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:27:03.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f862ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:03.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f862ffff640 1 -- 172.21.15.114:0/3654963049 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86300a5890 con 0x7f863809d3e0 2024-09-16T05:27:03.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 5 monclient: authenticate success, global_id 104414 2024-09-16T05:27:03.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:03.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f862ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f862ffff640 1 -- 172.21.15.114:0/3654963049 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86300a5c60 con 0x7f863809d3e0 2024-09-16T05:27:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f862ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 4 set_mon_vals no callback set 2024-09-16T05:27:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f862ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f862ffff640 10 monclient: dump: 2024-09-16T05:27:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:03.470 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:03.471 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 10 monclient: shutdown 2024-09-16T05:27:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 -- 172.21.15.114:0/3654963049 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f863809d3e0 msgr2=0x7f863813ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 --2- 172.21.15.114:0/3654963049 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f863809d3e0 0x7f863813ce50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8630009d90 tx=0x7f863001c8c0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 -- 172.21.15.114:0/3654963049 shutdown_connections 2024-09-16T05:27:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 --2- 172.21.15.114:0/3654963049 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f863809d3e0 0x7f863813ce50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 --2- 172.21.15.114:0/3654963049 >> v2:172.21.15.114:3300/0 conn(0x7f86380a11a0 0x7f863809ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 -- 172.21.15.114:0/3654963049 >> 172.21.15.114:0/3654963049 conn(0x7f86380a2490 msgr2=0x7f86380a28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 -- 172.21.15.114:0/3654963049 shutdown_connections 2024-09-16T05:27:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 -- 172.21.15.114:0/3654963049 wait complete. 2024-09-16T05:27:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 10 monclient: build_initial_monmap 2024-09-16T05:27:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 10 monclient: monmap: 2024-09-16T05:27:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:03.472 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 librados: starting msgr at 2024-09-16T05:27:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 librados: starting objecter 2024-09-16T05:27:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 Processor -- start 2024-09-16T05:27:03.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 -- start start 2024-09-16T05:27:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 librados: setting wanted keys 2024-09-16T05:27:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 librados: calling monclient init 2024-09-16T05:27:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 10 monclient: init 2024-09-16T05:27:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:27:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f863809cfd0 0x7f86380982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 10 monclient(hunting): picked mon.noname-b con 0x7f863809cfd0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f863809d3e0 0x7f863808fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 10 monclient(hunting): picked mon.noname-c con 0x7f863809d3e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86380a11a0 0x7f863808ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 10 monclient(hunting): picked mon.noname-a con 0x7f86380a11a0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f863809f0b0 con 0x7f86380a11a0 2024-09-16T05:27:03.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f863809ef30 con 0x7f863809cfd0 2024-09-16T05:27:03.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8638099590 con 0x7f863809d3e0 2024-09-16T05:27:03.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:03.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863f937640 10 monclient(hunting): authenticate will time out at 2108.095856s 2024-09-16T05:27:03.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863ced2640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f863809d3e0 0x7f863808fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.463+0000 7f863ded4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86380a11a0 0x7f863808ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ded4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86380a11a0 0x7f863808ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35178/0 (socket says 172.21.15.114:35178) 2024-09-16T05:27:03.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f863809d3e0 0x7f863808fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51760/0 (socket says 172.21.15.114:51760) 2024-09-16T05:27:03.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ded4640 1 -- 172.21.15.114:0/2711596378 learned_addr learned my addr 172.21.15.114:0/2711596378 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:03.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ded4640 10 monclient(hunting): get_auth_request con 0x7f86380a11a0 auth_method 0 2024-09-16T05:27:03.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ded4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:03.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ded4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ded4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 10 monclient(hunting): get_auth_request con 0x7f863809d3e0 auth_method 0 2024-09-16T05:27:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ded4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ded4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ded4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863d6d3640 1 --2- 172.21.15.114:0/2711596378 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f863809cfd0 0x7f86380982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 10 monclient(hunting): handle_auth_done global_id 104417 payload 824 2024-09-16T05:27:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 1 -- 172.21.15.114:0/2711596378 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f863809cfd0 msgr2=0x7f86380982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 1 --2- 172.21.15.114:0/2711596378 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f863809cfd0 0x7f86380982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 1 -- 172.21.15.114:0/2711596378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86380a11a0 msgr2=0x7f863808ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 1 --2- 172.21.15.114:0/2711596378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86380a11a0 0x7f863808ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 10 monclient: _finish_hunting 0 2024-09-16T05:27:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 1 monclient: found mon.noname-c 2024-09-16T05:27:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:27:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 1 -- 172.21.15.114:0/2711596378 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8638093460 con 0x7f863809d3e0 2024-09-16T05:27:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863d6d3640 10 monclient: get_auth_request con 0x7f863809cfd0 auth_method 0 2024-09-16T05:27:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ded4640 1 --2- 172.21.15.114:0/2711596378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86380a11a0 0x7f863808ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:27:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863d6d3640 1 --2- 172.21.15.114:0/2711596378 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f863809cfd0 0x7f86380982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:27:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ced2640 1 --2- 172.21.15.114:0/2711596378 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f863809d3e0 0x7f863808fa90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f862800c390 tx=0x7f862800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 client.?.objecter ms_handle_connect 0x7f863809d3e0 2024-09-16T05:27:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:03.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 1 -- 172.21.15.114:0/2711596378 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8628017070 con 0x7f863809d3e0 2024-09-16T05:27:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:27:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 monclient: dump: 2024-09-16T05:27:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:03.477 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 monclient: _finish_auth 0 2024-09-16T05:27:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:27:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863f937640 5 monclient: authenticate success, global_id 104417 2024-09-16T05:27:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 1 -- 172.21.15.114:0/2711596378 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8628005cf0 con 0x7f863809d3e0 2024-09-16T05:27:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863f937640 10 monclient: _renew_subs 2024-09-16T05:27:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863f937640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863f937640 1 -- 172.21.15.114:0/2711596378 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8638093740 con 0x7f863809d3e0 2024-09-16T05:27:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 1 -- 172.21.15.114:0/2711596378 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86280072f0 con 0x7f863809d3e0 2024-09-16T05:27:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 monclient: dump: 2024-09-16T05:27:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:03.478 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:03.479 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862f7fe640 4 set_mon_vals no callback set 2024-09-16T05:27:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863f937640 10 client.104417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863f937640 10 monclient: _renew_subs 2024-09-16T05:27:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863f937640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863f937640 1 -- 172.21.15.114:0/2711596378 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8638093180 con 0x7f863809d3e0 2024-09-16T05:27:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863f937640 1 librados: init done 2024-09-16T05:27:03.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863f937640 10 client.104417.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863f937640 15 client.104417.objecter _session_command_op_assign -1 1 2024-09-16T05:27:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863f937640 20 client.104417.objecter _get_session osd=-1 returning homeless 2024-09-16T05:27:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863f937640 10 client.104417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863f937640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863f937640 1 -- 172.21.15.114:0/2711596378 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f863809ca90 con 0x7f863809d3e0 2024-09-16T05:27:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 1 -- 172.21.15.114:0/2711596378 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f862801a070 con 0x7f863809d3e0 2024-09-16T05:27:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 1 --2- 172.21.15.114:0/2711596378 >> v2:172.21.15.149:6812/1185531135 conn(0x7f861c037840 0x7f861c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 1 -- 172.21.15.114:0/2711596378 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7f8628047eb0 con 0x7f863809d3e0 2024-09-16T05:27:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 client.104417.objecter ms_dispatch 0x7f8638000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 3 client.104417.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 3 client.104417.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 client.104417.objecter checking command 1 2024-09-16T05:27:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 1 --2- 172.21.15.114:0/2711596378 >> v2:172.21.15.114:6808/269070361 conn(0x7f861c03ece0 0x7f861c041180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 20 client.104417.objecter _get_session s=0x7f861c03e630 osd=0 3 2024-09-16T05:27:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 20 client.104417.objecter put_session s=0x7f861c03e630 osd=0 3 2024-09-16T05:27:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 15 client.104417.objecter _session_command_op_remove -1 1 2024-09-16T05:27:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 20 client.104417.objecter _get_session s=0x7f861c03e630 osd=0 3 2024-09-16T05:27:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 20 client.104417.objecter get_session s=0x7f861c03e630 osd=0 3 2024-09-16T05:27:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 15 client.104417.objecter _session_command_op_assign 0 1 2024-09-16T05:27:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 20 client.104417.objecter put_session s=0x7f861c03e630 osd=0 4 2024-09-16T05:27:03.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 client.104417.objecter _send_command 1 2024-09-16T05:27:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 1 -- 172.21.15.114:0/2711596378 --> v2:172.21.15.114:6808/269070361 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8628016ec0 con 0x7f861c03ece0 2024-09-16T05:27:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 20 client.104417.objecter dump_active .. 0 homeless 2024-09-16T05:27:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 1 -- 172.21.15.114:0/2711596378 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_get_version_reply(handle=1 version=695) ==== 24+0+0 (secure 0 0 0) 0x7f862804a580 con 0x7f863809d3e0 2024-09-16T05:27:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 monclient: handle_get_version_reply finishing 1 version 695 2024-09-16T05:27:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ded4640 1 --2- 172.21.15.114:0/2711596378 >> v2:172.21.15.114:6808/269070361 conn(0x7f861c03ece0 0x7f861c041180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ded4640 10 monclient: get_auth_request con 0x7f861c03ece0 auth_method 0 2024-09-16T05:27:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863d6d3640 1 --2- 172.21.15.114:0/2711596378 >> v2:172.21.15.149:6812/1185531135 conn(0x7f861c037840 0x7f861c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863d6d3640 10 monclient: get_auth_request con 0x7f861c037840 auth_method 0 2024-09-16T05:27:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 client.104417.objecter ms_handle_connect 0x7f861c03ece0 2024-09-16T05:27:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863ded4640 1 --2- 172.21.15.114:0/2711596378 >> v2:172.21.15.114:6808/269070361 conn(0x7f861c03ece0 0x7f861c041180 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 client.104417.objecter ms_handle_connect 0x7f861c037840 2024-09-16T05:27:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f863d6d3640 1 --2- 172.21.15.114:0/2711596378 >> v2:172.21.15.149:6812/1185531135 conn(0x7f861c037840 0x7f861c039d00 secure :-1 s=READY pgs=3633 cs=0 l=1 rev1=1 crypto rx=0x7f8634009ae0 tx=0x7f8634008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 1 -- 172.21.15.114:0/2711596378 <== osd.0 v2:172.21.15.114:6808/269070361 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8628016ec0 con 0x7f861c03ece0 2024-09-16T05:27:03.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 client.104417.objecter ms_dispatch 0x7f8638000ba0 command_reply(tid 1: 0 ) 2024-09-16T05:27:03.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 10 client.104417.objecter _finish_command 1 = system:0 2024-09-16T05:27:03.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 20 client.104417.objecter put_session s=0x7f861c03e630 osd=0 4 2024-09-16T05:27:03.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.467+0000 7f862dffb640 15 client.104417.objecter _session_command_op_remove 0 1 2024-09-16T05:27:03.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 10 client.104417.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T05:27:03.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 15 client.104417.objecter _session_command_op_assign -1 2 2024-09-16T05:27:03.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 20 client.104417.objecter _get_session s=0x7f861c03e630 osd=0 3 2024-09-16T05:27:03.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 20 client.104417.objecter put_session s=0x7f861c03e630 osd=0 3 2024-09-16T05:27:03.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 20 client.104417.objecter _get_session s=0x7f861c03e630 osd=0 3 2024-09-16T05:27:03.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 15 client.104417.objecter _session_command_op_remove -1 2 2024-09-16T05:27:03.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 20 client.104417.objecter get_session s=0x7f861c03e630 osd=0 3 2024-09-16T05:27:03.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 15 client.104417.objecter _session_command_op_assign 0 2 2024-09-16T05:27:03.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 20 client.104417.objecter put_session s=0x7f861c03e630 osd=0 4 2024-09-16T05:27:03.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 10 client.104417.objecter _send_command 2 2024-09-16T05:27:03.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 1 -- 172.21.15.114:0/2711596378 --> v2:172.21.15.114:6808/269070361 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8638092010 con 0x7f861c03ece0 2024-09-16T05:27:03.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f862dffb640 1 -- 172.21.15.114:0/2711596378 <== osd.0 v2:172.21.15.114:6808/269070361 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8638092010 con 0x7f861c03ece0 2024-09-16T05:27:03.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f862dffb640 10 client.104417.objecter ms_dispatch 0x7f8638000ba0 command_reply(tid 2: 0 ) 2024-09-16T05:27:03.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f862dffb640 10 client.104417.objecter _finish_command 2 = system:0 2024-09-16T05:27:03.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f862dffb640 20 client.104417.objecter put_session s=0x7f861c03e630 osd=0 4 2024-09-16T05:27:03.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f862dffb640 15 client.104417.objecter _session_command_op_remove 0 2 2024-09-16T05:27:03.485 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:27:03.485 INFO:teuthology.orchestra.run.smithi114.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T05:27:03.485 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:27:03.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 10 librados: watch_flush enter 2024-09-16T05:27:03.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 10 librados: watch_flush exit 2024-09-16T05:27:03.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 10 client.104417.objecter close_session for osd.0 2024-09-16T05:27:03.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 1 -- 172.21.15.114:0/2711596378 >> v2:172.21.15.114:6808/269070361 conn(0x7f861c03ece0 msgr2=0x7f861c041180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 1 --2- 172.21.15.114:0/2711596378 >> v2:172.21.15.114:6808/269070361 conn(0x7f861c03ece0 0x7f861c041180 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 20 client.104417.objecter put_session s=0x7f861c03e630 osd=0 1 2024-09-16T05:27:03.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 20 client.104417.objecter shutdown clearing up homeless session... 2024-09-16T05:27:03.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 10 client.104417.objecter successfully canceled tick 2024-09-16T05:27:03.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 1 -- 172.21.15.114:0/2711596378 >> v2:172.21.15.149:6812/1185531135 conn(0x7f861c037840 msgr2=0x7f861c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 1 --2- 172.21.15.114:0/2711596378 >> v2:172.21.15.149:6812/1185531135 conn(0x7f861c037840 0x7f861c039d00 secure :-1 s=READY pgs=3633 cs=0 l=1 rev1=1 crypto rx=0x7f8634009ae0 tx=0x7f8634008040 comp rx=0 tx=0).stop 2024-09-16T05:27:03.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 10 monclient: shutdown 2024-09-16T05:27:03.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:03.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 1 -- 172.21.15.114:0/2711596378 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f863809d3e0 msgr2=0x7f863808fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 1 --2- 172.21.15.114:0/2711596378 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f863809d3e0 0x7f863808fa90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f862800c390 tx=0x7f862800c850 comp rx=0 tx=0).stop 2024-09-16T05:27:03.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863ded4640 1 -- 172.21.15.114:0/2711596378 reap_dead start 2024-09-16T05:27:03.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 1 -- 172.21.15.114:0/2711596378 shutdown_connections 2024-09-16T05:27:03.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 1 -- 172.21.15.114:0/2711596378 >> 172.21.15.114:0/2711596378 conn(0x7f86380a2490 msgr2=0x7f86380a43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 1 -- 172.21.15.114:0/2711596378 shutdown_connections 2024-09-16T05:27:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 1 -- 172.21.15.114:0/2711596378 wait complete. 2024-09-16T05:27:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.483+0000 7f863f937640 1 librados: shutdown 2024-09-16T05:27:03.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T05:27:03.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.607+0000 7f36fa022640 10 monclient: get_monmap_and_config 2024-09-16T05:27:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.607+0000 7f36fa022640 10 monclient: build_initial_monmap 2024-09-16T05:27:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.607+0000 7f36fa022640 10 monclient: monmap: 2024-09-16T05:27:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:03.612256+0000 2024-09-16T05:27:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:03.612256+0000 2024-09-16T05:27:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:03.608 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:03.608 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 1 Processor -- start 2024-09-16T05:27:03.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 1 -- start start 2024-09-16T05:27:03.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 10 monclient: init 2024-09-16T05:27:03.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:03.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-16T05:27:03.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f36f4072580 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:03.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f36f4072960 addr v1:172.21.15.149:6789/0 2024-09-16T05:27:03.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f36f413d5b0 0x7f36f413f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 10 monclient(hunting): picked mon.noname-c con 0x7f36f413d5b0 addr v2:172.21.15.149:3300/0 2024-09-16T05:27:03.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36f409cab0 con 0x7f36f4072580 2024-09-16T05:27:03.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36f40a13a0 con 0x7f36f4072960 2024-09-16T05:27:03.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f36f40a0700 con 0x7f36f413d5b0 2024-09-16T05:27:03.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:03.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 10 monclient(hunting): authenticate will time out at 2108.241749s 2024-09-16T05:27:03.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f37fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f36f4072580 0x7f36f40a1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37638/0 (socket says 172.21.15.114:37638) 2024-09-16T05:27:03.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f37fe640 1 -- 172.21.15.114:0/846957007 learned_addr learned my addr 172.21.15.114:0/846957007 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:27:03.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1158952490 0 0) 0x7f36f409cab0 con 0x7f36f4072580 2024-09-16T05:27:03.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:27:03.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:03.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 1 --2- 172.21.15.114:0/846957007 >> v2:172.21.15.149:3300/0 conn(0x7f36f413d5b0 0x7f36f413f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): my global_id is 134196 2024-09-16T05:27:03.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36d4003530 con 0x7f36f4072580 2024-09-16T05:27:03.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient(hunting): get_auth_request con 0x7f36f413d5b0 auth_method 0 2024-09-16T05:27:03.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:03.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:03.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:03.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:03.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:03.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 486259161 0 0) 0x7f36f40a13a0 con 0x7f36f4072960 2024-09-16T05:27:03.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:27:03.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:03.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): my global_id is 124204 2024-09-16T05:27:03.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36f409cab0 con 0x7f36f4072960 2024-09-16T05:27:03.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3388690741 0 0) 0x7f36d4003530 con 0x7f36f4072580 2024-09-16T05:27:03.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 >> v2:172.21.15.149:3300/0 conn(0x7f36f413d5b0 msgr2=0x7f36f413f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:03.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 --2- 172.21.15.114:0/846957007 >> v2:172.21.15.149:3300/0 conn(0x7f36f413d5b0 0x7f36f413f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 >> v1:172.21.15.149:6789/0 conn(0x7f36f4072960 legacy=0x7f36f4072dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:27:03.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:27:03.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:03.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:27:03.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f40a1680 con 0x7f36f4072580 2024-09-16T05:27:03.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:27:03.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:27:03.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:03.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f36dc001600 con 0x7f36f4072580 2024-09-16T05:27:03.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 5 monclient: authenticate success, global_id 134196 2024-09-16T05:27:03.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:03.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:03.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:27:03.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient: dump: 2024-09-16T05:27:03.626 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:03.626 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.626 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.626 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.626 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:03.626 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:03.627 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:27:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 >> v1:172.21.15.114:6789/0 conn(0x7f36f4072580 legacy=0x7f36f40a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:27:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 --2- 172.21.15.114:0/846957007 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f36d4003ba0 0x7f36d4003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): picked mon.b con 0x7f36d4003ba0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 --2- 172.21.15.114:0/846957007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d4004810 0x7f36d4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): picked mon.a con 0x7f36d4004810 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 --2- 172.21.15.114:0/846957007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f36d4005290 0x7f36d400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): picked mon.c con 0x7f36d4005290 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f36f40a0700 con 0x7f36d4004810 2024-09-16T05:27:03.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f36d400c1d0 con 0x7f36d4003ba0 2024-09-16T05:27:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f36d400c3d0 con 0x7f36d4005290 2024-09-16T05:27:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:27:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36fa022640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 1 --2- 172.21.15.114:0/846957007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f36d4005290 0x7f36d400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36ebfff640 1 --2- 172.21.15.114:0/846957007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d4004810 0x7f36d4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient(hunting): get_auth_request con 0x7f36d4005290 auth_method 0 2024-09-16T05:27:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:27:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36ebfff640 10 monclient(hunting): get_auth_request con 0x7f36d4004810 auth_method 0 2024-09-16T05:27:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36ebfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:27:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:27:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:27:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient(hunting): handle_auth_done global_id 134196 payload 293 2024-09-16T05:27:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 1 -- 172.21.15.114:0/846957007 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f36d4003ba0 msgr2=0x7f36d4003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f37fe640 1 --2- 172.21.15.114:0/846957007 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f36d4003ba0 0x7f36d4003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 1 --2- 172.21.15.114:0/846957007 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f36d4003ba0 0x7f36d4003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 1 -- 172.21.15.114:0/846957007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d4004810 msgr2=0x7f36d4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 1 --2- 172.21.15.114:0/846957007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d4004810 0x7f36d4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient: _finish_hunting 0 2024-09-16T05:27:03.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 1 monclient: found mon.c 2024-09-16T05:27:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 1 -- 172.21.15.114:0/846957007 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d4003530 con 0x7f36d4005290 2024-09-16T05:27:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient: _finish_auth 0 2024-09-16T05:27:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 10 monclient: _check_auth_tickets 2024-09-16T05:27:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f37fe640 10 monclient: get_auth_request con 0x7f36d4003ba0 auth_method 0 2024-09-16T05:27:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36ebfff640 1 --2- 172.21.15.114:0/846957007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d4004810 0x7f36d4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:27:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.611+0000 7f36f37fe640 1 --2- 172.21.15.114:0/846957007 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f36d4003ba0 0x7f36d4003fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:27:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f3fff640 1 -- 172.21.15.114:0/846957007 reap_dead start 2024-09-16T05:27:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f3fff640 1 --2- 172.21.15.114:0/846957007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f36d4005290 0x7f36d400b9d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009fc0 tx=0x7f36ec01cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8c82b0417e2ea02 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:03.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e4001520 con 0x7f36d4005290 2024-09-16T05:27:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f2ffd640 10 monclient: dump: 2024-09-16T05:27:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:03.634 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36ec0a5c30 con 0x7f36d4005290 2024-09-16T05:27:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f2ffd640 1 -- 172.21.15.114:0/846957007 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec009c10 con 0x7f36d4005290 2024-09-16T05:27:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 4 set_mon_vals no callback set 2024-09-16T05:27:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f2ffd640 10 monclient: dump: 2024-09-16T05:27:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:03.636 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 10 monclient: shutdown 2024-09-16T05:27:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 -- 172.21.15.114:0/846957007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f36d4005290 msgr2=0x7f36d400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 --2- 172.21.15.114:0/846957007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f36d4005290 0x7f36d400b9d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009fc0 tx=0x7f36ec01cca0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 -- 172.21.15.114:0/846957007 shutdown_connections 2024-09-16T05:27:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 --2- 172.21.15.114:0/846957007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f36d4005290 0x7f36d400b9d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 -- 172.21.15.114:0/846957007 >> 172.21.15.114:0/846957007 conn(0x7f36f40a63b0 msgr2=0x7f36f40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 -- 172.21.15.114:0/846957007 shutdown_connections 2024-09-16T05:27:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 -- 172.21.15.114:0/846957007 wait complete. 2024-09-16T05:27:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 10 monclient: build_initial_monmap 2024-09-16T05:27:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 10 monclient: monmap: 2024-09-16T05:27:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:03.638 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 librados: starting msgr at 2024-09-16T05:27:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 librados: starting objecter 2024-09-16T05:27:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 Processor -- start 2024-09-16T05:27:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 -- start start 2024-09-16T05:27:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 librados: setting wanted keys 2024-09-16T05:27:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 librados: calling monclient init 2024-09-16T05:27:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 10 monclient: init 2024-09-16T05:27:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:27:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f36d4005290 0x7f36f4137430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 10 monclient(hunting): picked mon.noname-c con 0x7f36d4005290 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4131e90 0x7f36f41322b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 10 monclient(hunting): picked mon.noname-a con 0x7f36f4131e90 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f36f41327f0 0x7f36f41d54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 10 monclient(hunting): picked mon.noname-b con 0x7f36f41327f0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f36f40a2110 con 0x7f36f4131e90 2024-09-16T05:27:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f36f40a1f90 con 0x7f36f41327f0 2024-09-16T05:27:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f36f40a2290 con 0x7f36d4005290 2024-09-16T05:27:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36fa022640 10 monclient(hunting): authenticate will time out at 2108.247775s 2024-09-16T05:27:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f37fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f36d4005290 0x7f36f4137430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36ebfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4131e90 0x7f36f41322b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f37fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f36d4005290 0x7f36f4137430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51776/0 (socket says 172.21.15.114:51776) 2024-09-16T05:27:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f37fe640 1 -- 172.21.15.114:0/3539837687 learned_addr learned my addr 172.21.15.114:0/3539837687 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f37fe640 10 monclient(hunting): get_auth_request con 0x7f36d4005290 auth_method 0 2024-09-16T05:27:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36ebfff640 10 monclient(hunting): get_auth_request con 0x7f36f4131e90 auth_method 0 2024-09-16T05:27:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.615+0000 7f36ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f3fff640 1 --2- 172.21.15.114:0/3539837687 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f36f41327f0 0x7f36f41d54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f37fe640 10 monclient(hunting): handle_auth_done global_id 104420 payload 824 2024-09-16T05:27:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f37fe640 1 -- 172.21.15.114:0/3539837687 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f36f41327f0 msgr2=0x7f36f41d54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f37fe640 1 --2- 172.21.15.114:0/3539837687 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f36f41327f0 0x7f36f41d54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f37fe640 1 -- 172.21.15.114:0/3539837687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4131e90 msgr2=0x7f36f41322b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f37fe640 1 --2- 172.21.15.114:0/3539837687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4131e90 0x7f36f41322b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:27:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f37fe640 1 monclient: found mon.noname-c 2024-09-16T05:27:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:27:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f37fe640 1 -- 172.21.15.114:0/3539837687 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f4135780 con 0x7f36d4005290 2024-09-16T05:27:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f3fff640 10 monclient: get_auth_request con 0x7f36f41327f0 auth_method 0 2024-09-16T05:27:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f3fff640 1 --2- 172.21.15.114:0/3539837687 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f36f41327f0 0x7f36f41d54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:27:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36ebfff640 1 --2- 172.21.15.114:0/3539837687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4131e90 0x7f36f41322b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:27:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f37fe640 1 --2- 172.21.15.114:0/3539837687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f36d4005290 0x7f36f4137430 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f36dc007fd0 tx=0x7f36dc00dc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 10 client.?.objecter ms_handle_connect 0x7f36d4005290 2024-09-16T05:27:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 1 -- 172.21.15.114:0/3539837687 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc019070 con 0x7f36d4005290 2024-09-16T05:27:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:27:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 10 monclient: dump: 2024-09-16T05:27:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:03.646 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 10 monclient: _finish_auth 0 2024-09-16T05:27:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:27:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 1 -- 172.21.15.114:0/3539837687 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36dc0072d0 con 0x7f36d4005290 2024-09-16T05:27:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 1 -- 172.21.15.114:0/3539837687 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc007680 con 0x7f36d4005290 2024-09-16T05:27:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 10 monclient: dump: 2024-09-16T05:27:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:03.648 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f27fc640 4 set_mon_vals no callback set 2024-09-16T05:27:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36fa022640 5 monclient: authenticate success, global_id 104420 2024-09-16T05:27:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36fa022640 10 monclient: _renew_subs 2024-09-16T05:27:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36fa022640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36fa022640 1 -- 172.21.15.114:0/3539837687 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f406d1b0 con 0x7f36d4005290 2024-09-16T05:27:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36fa022640 10 client.104420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36fa022640 10 monclient: _renew_subs 2024-09-16T05:27:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36fa022640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36fa022640 1 -- 172.21.15.114:0/3539837687 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f4135a60 con 0x7f36d4005290 2024-09-16T05:27:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36fa022640 1 librados: init done 2024-09-16T05:27:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 1 -- 172.21.15.114:0/3539837687 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f36dc0047f0 con 0x7f36d4005290 2024-09-16T05:27:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36fa022640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36fa022640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36fa022640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36fa022640 1 -- 172.21.15.114:0/3539837687 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36f4072e00 con 0x7f36d4005290 2024-09-16T05:27:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 1 --2- 172.21.15.114:0/3539837687 >> v2:172.21.15.149:6812/1185531135 conn(0x7f36d8037810 0x7f36d8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 1 -- 172.21.15.114:0/3539837687 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7f36dc048430 con 0x7f36d4005290 2024-09-16T05:27:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 10 client.104420.objecter ms_dispatch 0x7f36f4000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 3 client.104420.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 3 client.104420.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 20 client.104420.objecter dump_active .. 0 homeless 2024-09-16T05:27:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36ebfff640 1 --2- 172.21.15.114:0/3539837687 >> v2:172.21.15.149:6812/1185531135 conn(0x7f36d8037810 0x7f36d8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36ebfff640 10 monclient: get_auth_request con 0x7f36d8037810 auth_method 0 2024-09-16T05:27:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36f0ff9640 10 client.104420.objecter ms_handle_connect 0x7f36d8037810 2024-09-16T05:27:03.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.619+0000 7f36ebfff640 1 --2- 172.21.15.114:0/3539837687 >> v2:172.21.15.149:6812/1185531135 conn(0x7f36d8037810 0x7f36d8039cd0 secure :-1 s=READY pgs=3634 cs=0 l=1 rev1=1 crypto rx=0x7f36f4133670 tx=0x7f36e4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:03.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.623+0000 7f36f0ff9640 1 -- 172.21.15.114:0/3539837687 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f36dc01c360 con 0x7f36d4005290 2024-09-16T05:27:03.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.623+0000 7f36f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:03.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.623+0000 7f36f0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:27:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.787+0000 7f36fa022640 1 -- 172.21.15.114:0/3539837687 --> v2:172.21.15.149:6812/1185531135 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f36f4134330 con 0x7f36d8037810 2024-09-16T05:27:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.787+0000 7f36f0ff9640 1 -- 172.21.15.114:0/3539837687 <== mgr.4108 v2:172.21.15.149:6812/1185531135 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f36f4134330 con 0x7f36d8037810 2024-09-16T05:27:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T05:27:03.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 10 librados: watch_flush enter 2024-09-16T05:27:03.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 10 librados: watch_flush exit 2024-09-16T05:27:03.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 20 client.104420.objecter shutdown clearing up homeless session... 2024-09-16T05:27:03.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 10 client.104420.objecter successfully canceled tick 2024-09-16T05:27:03.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 1 -- 172.21.15.114:0/3539837687 >> v2:172.21.15.149:6812/1185531135 conn(0x7f36d8037810 msgr2=0x7f36d8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 1 --2- 172.21.15.114:0/3539837687 >> v2:172.21.15.149:6812/1185531135 conn(0x7f36d8037810 0x7f36d8039cd0 secure :-1 s=READY pgs=3634 cs=0 l=1 rev1=1 crypto rx=0x7f36f4133670 tx=0x7f36e4009040 comp rx=0 tx=0).stop 2024-09-16T05:27:03.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 10 monclient: shutdown 2024-09-16T05:27:03.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:03.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 1 -- 172.21.15.114:0/3539837687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f36d4005290 msgr2=0x7f36f4137430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 1 --2- 172.21.15.114:0/3539837687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f36d4005290 0x7f36f4137430 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f36dc007fd0 tx=0x7f36dc00dc00 comp rx=0 tx=0).stop 2024-09-16T05:27:03.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 1 -- 172.21.15.114:0/3539837687 shutdown_connections 2024-09-16T05:27:03.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 1 --2- 172.21.15.114:0/3539837687 >> v2:172.21.15.149:6812/1185531135 conn(0x7f36d8037810 0x7f36d8039cd0 unknown :-1 s=CLOSED pgs=3634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 1 --2- 172.21.15.114:0/3539837687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4131e90 0x7f36f41322b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 1 --2- 172.21.15.114:0/3539837687 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f36f41327f0 0x7f36f41d54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 1 --2- 172.21.15.114:0/3539837687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f36d4005290 0x7f36f4137430 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 1 -- 172.21.15.114:0/3539837687 >> 172.21.15.114:0/3539837687 conn(0x7f36f40a63b0 msgr2=0x7f36f409f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:03.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 1 -- 172.21.15.114:0/3539837687 shutdown_connections 2024-09-16T05:27:03.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 1 -- 172.21.15.114:0/3539837687 wait complete. 2024-09-16T05:27:03.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.791+0000 7f36fa022640 1 librados: shutdown 2024-09-16T05:27:03.803 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T05:27:03.803 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T05:27:03.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.915+0000 7ffbdc7de640 10 monclient: get_monmap_and_config 2024-09-16T05:27:03.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.915+0000 7ffbdc7de640 10 monclient: build_initial_monmap 2024-09-16T05:27:03.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.915+0000 7ffbdc7de640 10 monclient: monmap: 2024-09-16T05:27:03.915 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:03.915 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.915 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:03.920064+0000 2024-09-16T05:27:03.915 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:03.920064+0000 2024-09-16T05:27:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:03.916 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 1 Processor -- start 2024-09-16T05:27:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 1 -- start start 2024-09-16T05:27:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 10 monclient: init 2024-09-16T05:27:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-16T05:27:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffbd40a1250 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ffbd409e2b0 0x7ffbd4139720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 10 monclient(hunting): picked mon.noname-c con 0x7ffbd409e2b0 addr v2:172.21.15.149:3300/0 2024-09-16T05:27:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffbd409e6c0 addr v1:172.21.15.149:6789/0 2024-09-16T05:27:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd40a1aa0 con 0x7ffbd40a1250 2024-09-16T05:27:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd409ecd0 con 0x7ffbd409e6c0 2024-09-16T05:27:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7ffbd409bba0 con 0x7ffbd409e2b0 2024-09-16T05:27:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 10 monclient(hunting): authenticate will time out at 2108.549300s 2024-09-16T05:27:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ffbd40a1250 0x7ffbd409dba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37640/0 (socket says 172.21.15.114:37640) 2024-09-16T05:27:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 1 -- 172.21.15.114:0/1175768023 learned_addr learned my addr 172.21.15.114:0/1175768023 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:27:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9d79640 1 --2- 172.21.15.114:0/1175768023 >> v2:172.21.15.149:3300/0 conn(0x7ffbd409e2b0 0x7ffbd4139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9d79640 10 monclient(hunting): get_auth_request con 0x7ffbd409e2b0 auth_method 0 2024-09-16T05:27:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 -- 172.21.15.114:0/1175768023 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1228184955 0 0) 0x7ffbd40a1aa0 con 0x7ffbd40a1250 2024-09-16T05:27:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9d79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9d79640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9d79640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient(hunting): _negotiate 2024-09-16T05:27:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient(hunting): my global_id is 134202 2024-09-16T05:27:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 -- 172.21.15.114:0/1175768023 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbc0003530 con 0x7ffbd40a1250 2024-09-16T05:27:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9d79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9d79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9d79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 -- 172.21.15.114:0/1175768023 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2433692014 0 0) 0x7ffbc0003530 con 0x7ffbd40a1250 2024-09-16T05:27:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 -- 172.21.15.114:0/1175768023 >> v2:172.21.15.149:3300/0 conn(0x7ffbd409e2b0 msgr2=0x7ffbd4139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 --2- 172.21.15.114:0/1175768023 >> v2:172.21.15.149:3300/0 conn(0x7ffbd409e2b0 0x7ffbd4139720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 -- 172.21.15.114:0/1175768023 >> v1:172.21.15.149:6789/0 conn(0x7ffbd409e6c0 legacy=0x7ffbd413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient: _finish_hunting 0 2024-09-16T05:27:03.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:27:03.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:03.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:27:03.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 -- 172.21.15.114:0/1175768023 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd40996c0 con 0x7ffbd40a1250 2024-09-16T05:27:03.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient: _finish_auth 0 2024-09-16T05:27:03.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient: _check_auth_tickets 2024-09-16T05:27:03.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:03.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 -- 172.21.15.114:0/1175768023 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7ffbc8001680 con 0x7ffbd40a1250 2024-09-16T05:27:03.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:27:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient: dump: 2024-09-16T05:27:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:03.934 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:27:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 -- 172.21.15.114:0/1175768023 >> v1:172.21.15.114:6789/0 conn(0x7ffbd40a1250 legacy=0x7ffbd409dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:27:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 --2- 172.21.15.114:0/1175768023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc00039c0 0x7ffbc0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient(hunting): picked mon.a con 0x7ffbc00039c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 --2- 172.21.15.114:0/1175768023 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffbc0004810 0x7ffbc0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient(hunting): picked mon.b con 0x7ffbc0004810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 --2- 172.21.15.114:0/1175768023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc00052f0 0x7ffbc000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient(hunting): picked mon.c con 0x7ffbc00052f0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:03.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 -- 172.21.15.114:0/1175768023 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc000c190 con 0x7ffbc00039c0 2024-09-16T05:27:03.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 -- 172.21.15.114:0/1175768023 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc000c340 con 0x7ffbc0004810 2024-09-16T05:27:03.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 1 -- 172.21.15.114:0/1175768023 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ffbc000e750 con 0x7ffbc00052f0 2024-09-16T05:27:03.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:03.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9578640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:27:03.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 1 --2- 172.21.15.114:0/1175768023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc00039c0 0x7ffbc0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 10 monclient(hunting): get_auth_request con 0x7ffbc00039c0 auth_method 0 2024-09-16T05:27:03.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:27:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdad7b640 1 --2- 172.21.15.114:0/1175768023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc00052f0 0x7ffbc000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdad7b640 10 monclient(hunting): get_auth_request con 0x7ffbc00052f0 auth_method 0 2024-09-16T05:27:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdad7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdad7b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdad7b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:27:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:27:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdad7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdad7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdad7b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:27:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9d79640 1 --2- 172.21.15.114:0/1175768023 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffbc0004810 0x7ffbc0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 10 monclient(hunting): handle_auth_done global_id 134202 payload 293 2024-09-16T05:27:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 1 -- 172.21.15.114:0/1175768023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc00052f0 msgr2=0x7ffbc000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 1 --2- 172.21.15.114:0/1175768023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc00052f0 0x7ffbc000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 1 -- 172.21.15.114:0/1175768023 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffbc0004810 msgr2=0x7ffbc0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 1 --2- 172.21.15.114:0/1175768023 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffbc0004810 0x7ffbc0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 10 monclient: _finish_hunting 0 2024-09-16T05:27:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 1 monclient: found mon.a 2024-09-16T05:27:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:03.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 1 -- 172.21.15.114:0/1175768023 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc0003530 con 0x7ffbc00039c0 2024-09-16T05:27:03.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 10 monclient: _finish_auth 0 2024-09-16T05:27:03.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdad7b640 1 -- 172.21.15.114:0/1175768023 reap_dead start 2024-09-16T05:27:03.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 10 monclient: _check_auth_tickets 2024-09-16T05:27:03.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbda57a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:03.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9d79640 10 monclient: get_auth_request con 0x7ffbc0004810 auth_method 0 2024-09-16T05:27:03.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 5 monclient: authenticate success, global_id 134202 2024-09-16T05:27:03.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbdc7de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:03.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.919+0000 7ffbd9d79640 1 --2- 172.21.15.114:0/1175768023 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffbc0004810 0x7ffbc0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:27:03.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbda57a640 1 --2- 172.21.15.114:0/1175768023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc00039c0 0x7ffbc0003de0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8001ec0 tx=0x7ffbc801cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1fc948cb30999df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:03.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9578640 1 -- 172.21.15.114:0/1175768023 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8007d40 con 0x7ffbc00039c0 2024-09-16T05:27:03.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9578640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:03.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9578640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:03.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9578640 10 monclient: dump: 2024-09-16T05:27:03.939 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:03.939 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.939 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.939 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.939 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:03.939 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.939 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:03.939 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:03.939 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:03.939 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:03.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9578640 1 -- 172.21.15.114:0/1175768023 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbc80a9070 con 0x7ffbc00039c0 2024-09-16T05:27:03.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9578640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:03.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:03.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 4 set_mon_vals no callback set 2024-09-16T05:27:03.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9578640 1 -- 172.21.15.114:0/1175768023 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8005d80 con 0x7ffbc00039c0 2024-09-16T05:27:03.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9578640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:03.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9578640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:03.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9578640 10 monclient: dump: 2024-09-16T05:27:03.940 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:03.940 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.940 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.941 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.941 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:03.941 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.941 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:03.941 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:03.941 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:03.941 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient: shutdown 2024-09-16T05:27:03.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:03.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 -- 172.21.15.114:0/1175768023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc00039c0 msgr2=0x7ffbc0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 --2- 172.21.15.114:0/1175768023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc00039c0 0x7ffbc0003de0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8001ec0 tx=0x7ffbc801cbe0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 -- 172.21.15.114:0/1175768023 shutdown_connections 2024-09-16T05:27:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 --2- 172.21.15.114:0/1175768023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc00039c0 0x7ffbc0003de0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 -- 172.21.15.114:0/1175768023 >> 172.21.15.114:0/1175768023 conn(0x7ffbd40a2490 msgr2=0x7ffbd40a28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 -- 172.21.15.114:0/1175768023 shutdown_connections 2024-09-16T05:27:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 -- 172.21.15.114:0/1175768023 wait complete. 2024-09-16T05:27:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient: build_initial_monmap 2024-09-16T05:27:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient: monmap: 2024-09-16T05:27:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:03.943 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 librados: starting msgr at 2024-09-16T05:27:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 librados: starting objecter 2024-09-16T05:27:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 Processor -- start 2024-09-16T05:27:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 -- start start 2024-09-16T05:27:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 librados: setting wanted keys 2024-09-16T05:27:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 librados: calling monclient init 2024-09-16T05:27:03.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient: init 2024-09-16T05:27:03.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:03.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:27:03.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc00039c0 0x7ffbd412ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient(hunting): picked mon.noname-c con 0x7ffbc00039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:03.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbd412f370 0x7ffbd412f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient(hunting): picked mon.noname-a con 0x7ffbd412f370 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:03.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffbd41d3680 0x7ffbd41d5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient(hunting): picked mon.noname-b con 0x7ffbd41d3680 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:03.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd409d420 con 0x7ffbd412f370 2024-09-16T05:27:03.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd409d2a0 con 0x7ffbd41d3680 2024-09-16T05:27:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ffbd409d5a0 con 0x7ffbc00039c0 2024-09-16T05:27:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient(hunting): authenticate will time out at 2108.555133s 2024-09-16T05:27:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbd412f370 0x7ffbd412f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbda57a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc00039c0 0x7ffbd412ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbd412f370 0x7ffbd412f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35222/0 (socket says 172.21.15.114:35222) 2024-09-16T05:27:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 1 -- 172.21.15.114:0/144967235 learned_addr learned my addr 172.21.15.114:0/144967235 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 10 monclient(hunting): get_auth_request con 0x7ffbd412f370 auth_method 0 2024-09-16T05:27:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbda57a640 10 monclient(hunting): get_auth_request con 0x7ffbc00039c0 auth_method 0 2024-09-16T05:27:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbda57a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbda57a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbda57a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbda57a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbda57a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbda57a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdad7b640 1 --2- 172.21.15.114:0/144967235 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffbd41d3680 0x7ffbd41d5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdad7b640 10 monclient(hunting): get_auth_request con 0x7ffbd41d3680 auth_method 0 2024-09-16T05:27:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdad7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdad7b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdad7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 10 monclient(hunting): handle_auth_done global_id 134205 payload 824 2024-09-16T05:27:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 1 -- 172.21.15.114:0/144967235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc00039c0 msgr2=0x7ffbd412ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 1 --2- 172.21.15.114:0/144967235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc00039c0 0x7ffbd412ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 1 -- 172.21.15.114:0/144967235 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffbd41d3680 msgr2=0x7ffbd41d5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 1 --2- 172.21.15.114:0/144967235 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffbd41d3680 0x7ffbd41d5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 10 monclient: _finish_hunting 0 2024-09-16T05:27:03.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 1 monclient: found mon.noname-a 2024-09-16T05:27:03.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:03.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:03.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 1 -- 172.21.15.114:0/144967235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd40a4590 con 0x7ffbd412f370 2024-09-16T05:27:03.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbda57a640 1 --2- 172.21.15.114:0/144967235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc00039c0 0x7ffbd412ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:27:03.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd9d79640 1 --2- 172.21.15.114:0/144967235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbd412f370 0x7ffbd412f7b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00107f0 tx=0x7ffbd0010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:03.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbc6ffd640 10 client.?.objecter ms_handle_connect 0x7ffbd412f370 2024-09-16T05:27:03.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbc6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:03.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbc6ffd640 1 -- 172.21.15.114:0/144967235 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd000f040 con 0x7ffbd412f370 2024-09-16T05:27:03.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:03.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbc6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbc6ffd640 10 monclient: dump: 2024-09-16T05:27:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:03.949 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbc6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:27:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbc6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:27:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbc6ffd640 1 -- 172.21.15.114:0/144967235 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbd0004220 con 0x7ffbd412f370 2024-09-16T05:27:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbc6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbc6ffd640 1 -- 172.21.15.114:0/144967235 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0010ea0 con 0x7ffbd412f370 2024-09-16T05:27:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbc6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbc6ffd640 10 monclient: dump: 2024-09-16T05:27:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:03.951 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 5 monclient: authenticate success, global_id 134205 2024-09-16T05:27:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbd8d77640 4 set_mon_vals no callback set 2024-09-16T05:27:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient: _renew_subs 2024-09-16T05:27:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.923+0000 7ffbdc7de640 1 -- 172.21.15.114:0/144967235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd4132690 con 0x7ffbd412f370 2024-09-16T05:27:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbdc7de640 10 client.134205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbdc7de640 10 monclient: _renew_subs 2024-09-16T05:27:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbdc7de640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbdc7de640 1 -- 172.21.15.114:0/144967235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd40a4a30 con 0x7ffbd412f370 2024-09-16T05:27:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbdc7de640 1 librados: init done 2024-09-16T05:27:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbdc7de640 10 client.134205.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbdc7de640 15 client.134205.objecter _session_command_op_assign -1 1 2024-09-16T05:27:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbdc7de640 20 client.134205.objecter _get_session osd=-1 returning homeless 2024-09-16T05:27:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbdc7de640 10 client.134205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbdc7de640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbdc7de640 1 -- 172.21.15.114:0/144967235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffbd409c780 con 0x7ffbd412f370 2024-09-16T05:27:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 1 -- 172.21.15.114:0/144967235 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ffbd000e900 con 0x7ffbd412f370 2024-09-16T05:27:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 1 --2- 172.21.15.114:0/144967235 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffbb8037810 0x7ffbb8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 1 -- 172.21.15.114:0/144967235 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7ffbd0048c40 con 0x7ffbd412f370 2024-09-16T05:27:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 10 client.134205.objecter ms_dispatch 0x7ffbd4000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 3 client.134205.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 3 client.134205.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 10 client.134205.objecter checking command 1 2024-09-16T05:27:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 1 --2- 172.21.15.114:0/144967235 >> v2:172.21.15.114:6800/2903720358 conn(0x7ffbb803ecb0 0x7ffbb8041150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 20 client.134205.objecter _get_session s=0x7ffbb803e600 osd=1 3 2024-09-16T05:27:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 20 client.134205.objecter put_session s=0x7ffbb803e600 osd=1 3 2024-09-16T05:27:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 15 client.134205.objecter _session_command_op_remove -1 1 2024-09-16T05:27:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 20 client.134205.objecter _get_session s=0x7ffbb803e600 osd=1 3 2024-09-16T05:27:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 20 client.134205.objecter get_session s=0x7ffbb803e600 osd=1 3 2024-09-16T05:27:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 15 client.134205.objecter _session_command_op_assign 1 1 2024-09-16T05:27:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 20 client.134205.objecter put_session s=0x7ffbb803e600 osd=1 4 2024-09-16T05:27:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 10 client.134205.objecter _send_command 1 2024-09-16T05:27:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 1 -- 172.21.15.114:0/144967235 --> v2:172.21.15.114:6800/2903720358 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffbd0006800 con 0x7ffbb803ecb0 2024-09-16T05:27:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 20 client.134205.objecter dump_active .. 0 homeless 2024-09-16T05:27:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 1 -- 172.21.15.114:0/144967235 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=695) ==== 24+0+0 (secure 0 0 0) 0x7ffbd004b310 con 0x7ffbd412f370 2024-09-16T05:27:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 10 monclient: handle_get_version_reply finishing 1 version 695 2024-09-16T05:27:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbdad7b640 1 --2- 172.21.15.114:0/144967235 >> v2:172.21.15.114:6800/2903720358 conn(0x7ffbb803ecb0 0x7ffbb8041150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbdad7b640 10 monclient: get_auth_request con 0x7ffbb803ecb0 auth_method 0 2024-09-16T05:27:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbda57a640 1 --2- 172.21.15.114:0/144967235 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffbb8037810 0x7ffbb8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbda57a640 10 monclient: get_auth_request con 0x7ffbb8037810 auth_method 0 2024-09-16T05:27:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbdad7b640 1 --2- 172.21.15.114:0/144967235 >> v2:172.21.15.114:6800/2903720358 conn(0x7ffbb803ecb0 0x7ffbb8041150 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 10 client.134205.objecter ms_handle_connect 0x7ffbb803ecb0 2024-09-16T05:27:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 10 client.134205.objecter ms_handle_connect 0x7ffbb8037810 2024-09-16T05:27:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbda57a640 1 --2- 172.21.15.114:0/144967235 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffbb8037810 0x7ffbb8039cd0 secure :-1 s=READY pgs=3635 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80ab000 tx=0x7ffbc80a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 1 -- 172.21.15.114:0/144967235 <== osd.1 v2:172.21.15.114:6800/2903720358 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffbd0006800 con 0x7ffbb803ecb0 2024-09-16T05:27:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 10 client.134205.objecter ms_dispatch 0x7ffbd4000ba0 command_reply(tid 1: 0 ) 2024-09-16T05:27:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 10 client.134205.objecter _finish_command 1 = system:0 2024-09-16T05:27:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 20 client.134205.objecter put_session s=0x7ffbb803e600 osd=1 4 2024-09-16T05:27:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.927+0000 7ffbc6ffd640 15 client.134205.objecter _session_command_op_remove 1 1 2024-09-16T05:27:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 10 client.134205.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T05:27:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 15 client.134205.objecter _session_command_op_assign -1 2 2024-09-16T05:27:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 20 client.134205.objecter _get_session s=0x7ffbb803e600 osd=1 3 2024-09-16T05:27:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 20 client.134205.objecter put_session s=0x7ffbb803e600 osd=1 3 2024-09-16T05:27:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 20 client.134205.objecter _get_session s=0x7ffbb803e600 osd=1 3 2024-09-16T05:27:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 15 client.134205.objecter _session_command_op_remove -1 2 2024-09-16T05:27:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 20 client.134205.objecter get_session s=0x7ffbb803e600 osd=1 3 2024-09-16T05:27:03.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 15 client.134205.objecter _session_command_op_assign 1 2 2024-09-16T05:27:03.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 20 client.134205.objecter put_session s=0x7ffbb803e600 osd=1 4 2024-09-16T05:27:03.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 10 client.134205.objecter _send_command 2 2024-09-16T05:27:03.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 1 -- 172.21.15.114:0/144967235 --> v2:172.21.15.114:6800/2903720358 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ffbd409d930 con 0x7ffbb803ecb0 2024-09-16T05:27:03.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbc6ffd640 1 -- 172.21.15.114:0/144967235 <== osd.1 v2:172.21.15.114:6800/2903720358 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ffbd409d930 con 0x7ffbb803ecb0 2024-09-16T05:27:03.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbc6ffd640 10 client.134205.objecter ms_dispatch 0x7ffbd4000ba0 command_reply(tid 2: 0 ) 2024-09-16T05:27:03.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbc6ffd640 10 client.134205.objecter _finish_command 2 = system:0 2024-09-16T05:27:03.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbc6ffd640 20 client.134205.objecter put_session s=0x7ffbb803e600 osd=1 4 2024-09-16T05:27:03.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbc6ffd640 15 client.134205.objecter _session_command_op_remove 1 2 2024-09-16T05:27:03.958 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:27:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T05:27:03.959 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:27:03.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 10 librados: watch_flush enter 2024-09-16T05:27:03.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 10 librados: watch_flush exit 2024-09-16T05:27:03.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 10 client.134205.objecter close_session for osd.1 2024-09-16T05:27:03.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 1 -- 172.21.15.114:0/144967235 >> v2:172.21.15.114:6800/2903720358 conn(0x7ffbb803ecb0 msgr2=0x7ffbb8041150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 1 --2- 172.21.15.114:0/144967235 >> v2:172.21.15.114:6800/2903720358 conn(0x7ffbb803ecb0 0x7ffbb8041150 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 20 client.134205.objecter put_session s=0x7ffbb803e600 osd=1 1 2024-09-16T05:27:03.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 20 client.134205.objecter shutdown clearing up homeless session... 2024-09-16T05:27:03.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 10 client.134205.objecter successfully canceled tick 2024-09-16T05:27:03.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 1 -- 172.21.15.114:0/144967235 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffbb8037810 msgr2=0x7ffbb8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 1 --2- 172.21.15.114:0/144967235 >> v2:172.21.15.149:6812/1185531135 conn(0x7ffbb8037810 0x7ffbb8039cd0 secure :-1 s=READY pgs=3635 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80ab000 tx=0x7ffbc80a7070 comp rx=0 tx=0).stop 2024-09-16T05:27:03.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 10 monclient: shutdown 2024-09-16T05:27:03.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:03.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 1 -- 172.21.15.114:0/144967235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbd412f370 msgr2=0x7ffbd412f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:03.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 1 --2- 172.21.15.114:0/144967235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbd412f370 0x7ffbd412f7b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00107f0 tx=0x7ffbd0010cb0 comp rx=0 tx=0).stop 2024-09-16T05:27:03.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdad7b640 1 -- 172.21.15.114:0/144967235 reap_dead start 2024-09-16T05:27:03.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 1 -- 172.21.15.114:0/144967235 shutdown_connections 2024-09-16T05:27:03.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 1 -- 172.21.15.114:0/144967235 >> 172.21.15.114:0/144967235 conn(0x7ffbd40a2490 msgr2=0x7ffbd409fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:03.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 1 -- 172.21.15.114:0/144967235 shutdown_connections 2024-09-16T05:27:03.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 1 -- 172.21.15.114:0/144967235 wait complete. 2024-09-16T05:27:03.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:03.939+0000 7ffbdc7de640 1 librados: shutdown 2024-09-16T05:27:03.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T05:27:04.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 10 monclient: get_monmap_and_config 2024-09-16T05:27:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 10 monclient: build_initial_monmap 2024-09-16T05:27:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 10 monclient: monmap: 2024-09-16T05:27:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:04.072401+0000 2024-09-16T05:27:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:04.072401+0000 2024-09-16T05:27:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:04.068 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 1 Processor -- start 2024-09-16T05:27:04.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 1 -- start start 2024-09-16T05:27:04.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 10 monclient: init 2024-09-16T05:27:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-16T05:27:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f730c1376e0 0x7f730c137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 10 monclient(hunting): picked mon.noname-a con 0x7f730c1376e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:27:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f730c1370a0 0x7f730c0a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 10 monclient(hunting): picked mon.noname-c con 0x7f730c1370a0 addr v2:172.21.15.149:3300/0 2024-09-16T05:27:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f730c0a5d00 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f730c09bd20 con 0x7f730c0a5d00 2024-09-16T05:27:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f730c138000 con 0x7f730c1376e0 2024-09-16T05:27:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f730c09c5c0 con 0x7f730c1370a0 2024-09-16T05:27:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f73111a7640 10 monclient(hunting): authenticate will time out at 2108.701566s 2024-09-16T05:27:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f730c1376e0 0x7f730c137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f730c1376e0 0x7f730c137ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35230/0 (socket says 172.21.15.114:35230) 2024-09-16T05:27:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 1 -- 172.21.15.114:0/3369125019 learned_addr learned my addr 172.21.15.114:0/3369125019 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 10 monclient(hunting): get_auth_request con 0x7f730c1376e0 auth_method 0 2024-09-16T05:27:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730a59c640 1 --2- 172.21.15.114:0/3369125019 >> v2:172.21.15.149:3300/0 conn(0x7f730c1370a0 0x7f730c0a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 10 monclient(hunting): handle_auth_done global_id 134208 payload 274 2024-09-16T05:27:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 1 -- 172.21.15.114:0/3369125019 >> v2:172.21.15.149:3300/0 conn(0x7f730c1370a0 msgr2=0x7f730c0a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 1 --2- 172.21.15.114:0/3369125019 >> v2:172.21.15.149:3300/0 conn(0x7f730c1370a0 0x7f730c0a4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 1 -- 172.21.15.114:0/3369125019 >> v1:172.21.15.114:6789/0 conn(0x7f730c0a5d00 legacy=0x7f730c0a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 10 monclient: _finish_hunting 0 2024-09-16T05:27:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 1 monclient: found mon.noname-a 2024-09-16T05:27:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:04.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730ad9d640 1 -- 172.21.15.114:0/3369125019 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730c09c000 con 0x7f730c1376e0 2024-09-16T05:27:04.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730a59c640 10 monclient: get_auth_request con 0x7f730c1370a0 auth_method 0 2024-09-16T05:27:04.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.071+0000 7f730a59c640 1 --2- 172.21.15.114:0/3369125019 >> v2:172.21.15.149:3300/0 conn(0x7f730c1370a0 0x7f730c0a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:27:04.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 1 --2- 172.21.15.114:0/3369125019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f730c1376e0 0x7f730c137ac0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f730c073760 tx=0x7f72fc01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69769baaadf2f335 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:04.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f7309d9b640 1 -- 172.21.15.114:0/3369125019 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72fc0b0070 con 0x7f730c1376e0 2024-09-16T05:27:04.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f7309d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f7309d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f7309d9b640 10 monclient: dump: 2024-09-16T05:27:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.079 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f7309d9b640 10 monclient: _finish_auth 0 2024-09-16T05:27:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f7309d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:27:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f7309d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f7309d9b640 1 -- 172.21.15.114:0/3369125019 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72fc0ab600 con 0x7f730c1376e0 2024-09-16T05:27:04.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f7309d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:04.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f7309d9b640 1 -- 172.21.15.114:0/3369125019 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72fc0ab9d0 con 0x7f730c1376e0 2024-09-16T05:27:04.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 5 monclient: authenticate success, global_id 134208 2024-09-16T05:27:04.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:04.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 4 set_mon_vals no callback set 2024-09-16T05:27:04.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f7309d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f7309d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:04.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f7309d9b640 10 monclient: dump: 2024-09-16T05:27:04.080 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.081 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.081 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.081 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.081 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.081 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.082 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient: shutdown 2024-09-16T05:27:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 -- 172.21.15.114:0/3369125019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f730c1376e0 msgr2=0x7f730c137ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 --2- 172.21.15.114:0/3369125019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f730c1376e0 0x7f730c137ac0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f730c073760 tx=0x7f72fc01c3e0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 -- 172.21.15.114:0/3369125019 shutdown_connections 2024-09-16T05:27:04.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 --2- 172.21.15.114:0/3369125019 >> v2:172.21.15.149:3300/0 conn(0x7f730c1370a0 0x7f730c0a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 --2- 172.21.15.114:0/3369125019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f730c1376e0 0x7f730c137ac0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 -- 172.21.15.114:0/3369125019 >> 172.21.15.114:0/3369125019 conn(0x7f730c0a9a40 msgr2=0x7f730c0a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:04.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 -- 172.21.15.114:0/3369125019 shutdown_connections 2024-09-16T05:27:04.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 -- 172.21.15.114:0/3369125019 wait complete. 2024-09-16T05:27:04.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient: build_initial_monmap 2024-09-16T05:27:04.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient: monmap: 2024-09-16T05:27:04.083 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:04.084 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:04.084 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:04.084 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:04.084 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:04.084 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.084 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:04.084 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:04.084 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:04.084 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 librados: starting msgr at 2024-09-16T05:27:04.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 librados: starting objecter 2024-09-16T05:27:04.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 Processor -- start 2024-09-16T05:27:04.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 -- start start 2024-09-16T05:27:04.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 librados: setting wanted keys 2024-09-16T05:27:04.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 librados: calling monclient init 2024-09-16T05:27:04.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient: init 2024-09-16T05:27:04.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:04.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:27:04.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f730c0a5d00 0x7f730c135bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient(hunting): picked mon.noname-a con 0x7f730c0a5d00 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:04.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f730c1370a0 0x7f730c136130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient(hunting): picked mon.noname-c con 0x7f730c1370a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:04.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f730c1376e0 0x7f730c13bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient(hunting): picked mon.noname-b con 0x7f730c1376e0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f730c141d80 con 0x7f730c0a5d00 2024-09-16T05:27:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f730c141c00 con 0x7f730c1376e0 2024-09-16T05:27:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f730c141f00 con 0x7f730c1370a0 2024-09-16T05:27:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient(hunting): authenticate will time out at 2108.706358s 2024-09-16T05:27:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f730c0a5d00 0x7f730c135bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730a59c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f730c1370a0 0x7f730c136130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f730c0a5d00 0x7f730c135bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35240/0 (socket says 172.21.15.114:35240) 2024-09-16T05:27:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 1 -- 172.21.15.114:0/3166295334 learned_addr learned my addr 172.21.15.114:0/3166295334 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 10 monclient(hunting): get_auth_request con 0x7f730c0a5d00 auth_method 0 2024-09-16T05:27:04.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730a59c640 10 monclient(hunting): get_auth_request con 0x7f730c1370a0 auth_method 0 2024-09-16T05:27:04.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:04.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:04.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:04.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:04.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:04.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 10 monclient(hunting): handle_auth_done global_id 134211 payload 824 2024-09-16T05:27:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 1 -- 172.21.15.114:0/3166295334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f730c1370a0 msgr2=0x7f730c136130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 1 --2- 172.21.15.114:0/3166295334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f730c1370a0 0x7f730c136130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 1 -- 172.21.15.114:0/3166295334 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f730c1376e0 msgr2=0x7f730c13bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730b59e640 1 --2- 172.21.15.114:0/3166295334 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f730c1376e0 0x7f730c13bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 1 --2- 172.21.15.114:0/3166295334 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f730c1376e0 0x7f730c13bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 10 monclient: _finish_hunting 0 2024-09-16T05:27:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 1 monclient: found mon.noname-a 2024-09-16T05:27:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 1 -- 172.21.15.114:0/3166295334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730c073d20 con 0x7f730c0a5d00 2024-09-16T05:27:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730b59e640 10 monclient: get_auth_request con 0x7f730c1376e0 auth_method 0 2024-09-16T05:27:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730b59e640 1 --2- 172.21.15.114:0/3166295334 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f730c1376e0 0x7f730c13bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:27:04.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730ad9d640 1 --2- 172.21.15.114:0/3166295334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f730c0a5d00 0x7f730c135bf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f72fc018ff0 tx=0x7f72fc019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:04.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 10 client.?.objecter ms_handle_connect 0x7f730c0a5d00 2024-09-16T05:27:04.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:04.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 1 -- 172.21.15.114:0/3166295334 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72fc0b9070 con 0x7f730c0a5d00 2024-09-16T05:27:04.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:04.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 10 monclient: dump: 2024-09-16T05:27:04.090 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.090 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.090 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.090 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.091 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.091 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.091 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.091 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.091 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.091 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 10 monclient: _finish_auth 0 2024-09-16T05:27:04.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:27:04.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:04.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 5 monclient: authenticate success, global_id 134211 2024-09-16T05:27:04.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 1 -- 172.21.15.114:0/3166295334 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72fc0b4600 con 0x7f730c0a5d00 2024-09-16T05:27:04.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient: _renew_subs 2024-09-16T05:27:04.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:04.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 -- 172.21.15.114:0/3166295334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f730c120180 con 0x7f730c0a5d00 2024-09-16T05:27:04.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:04.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 1 -- 172.21.15.114:0/3166295334 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72fc0b0040 con 0x7f730c0a5d00 2024-09-16T05:27:04.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:04.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 10 monclient: dump: 2024-09-16T05:27:04.092 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.092 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.093 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.093 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.093 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.093 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.093 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.093 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.093 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.093 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 client.134211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:04.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f730959a640 4 set_mon_vals no callback set 2024-09-16T05:27:04.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient: _renew_subs 2024-09-16T05:27:04.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:04.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 -- 172.21.15.114:0/3166295334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f730c0a8f30 con 0x7f730c0a5d00 2024-09-16T05:27:04.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 librados: init done 2024-09-16T05:27:04.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:04.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:04.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:04.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f73111a7640 1 -- 172.21.15.114:0/3166295334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f730c137ac0 con 0x7f730c0a5d00 2024-09-16T05:27:04.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.075+0000 7f72f37fe640 1 -- 172.21.15.114:0/3166295334 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f72fc0bc070 con 0x7f730c0a5d00 2024-09-16T05:27:04.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.079+0000 7f72f37fe640 1 --2- 172.21.15.114:0/3166295334 >> v2:172.21.15.149:6812/1185531135 conn(0x7f72f8037840 0x7f72f8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.079+0000 7f72f37fe640 1 -- 172.21.15.114:0/3166295334 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7f72fc0e1e10 con 0x7f730c0a5d00 2024-09-16T05:27:04.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.079+0000 7f72f37fe640 10 client.134211.objecter ms_dispatch 0x7f730c000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:04.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.079+0000 7f72f37fe640 3 client.134211.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:04.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.079+0000 7f72f37fe640 3 client.134211.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:04.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.079+0000 7f72f37fe640 20 client.134211.objecter dump_active .. 0 homeless 2024-09-16T05:27:04.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.079+0000 7f730a59c640 1 --2- 172.21.15.114:0/3166295334 >> v2:172.21.15.149:6812/1185531135 conn(0x7f72f8037840 0x7f72f8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.079+0000 7f730a59c640 10 monclient: get_auth_request con 0x7f72f8037840 auth_method 0 2024-09-16T05:27:04.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.079+0000 7f72f37fe640 10 client.134211.objecter ms_handle_connect 0x7f72f8037840 2024-09-16T05:27:04.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.079+0000 7f730a59c640 1 --2- 172.21.15.114:0/3166295334 >> v2:172.21.15.149:6812/1185531135 conn(0x7f72f8037840 0x7f72f8039d00 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 crypto rx=0x7f730c1d0080 tx=0x7f7300007810 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:04.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.083+0000 7f72f37fe640 1 -- 172.21.15.114:0/3166295334 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f730c073d20 con 0x7f730c0a5d00 2024-09-16T05:27:04.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.083+0000 7f72f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:04.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.083+0000 7f72f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:27:04.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.231+0000 7f73111a7640 1 -- 172.21.15.114:0/3166295334 --> v2:172.21.15.149:6812/1185531135 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f730c0a2510 con 0x7f72f8037840 2024-09-16T05:27:04.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.231+0000 7f72f37fe640 1 -- 172.21.15.114:0/3166295334 <== mgr.4108 v2:172.21.15.149:6812/1185531135 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f730c0a2510 con 0x7f72f8037840 2024-09-16T05:27:04.230 INFO:teuthology.orchestra.run.smithi114.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T05:27:04.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 10 librados: watch_flush enter 2024-09-16T05:27:04.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 10 librados: watch_flush exit 2024-09-16T05:27:04.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 20 client.134211.objecter shutdown clearing up homeless session... 2024-09-16T05:27:04.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 10 client.134211.objecter successfully canceled tick 2024-09-16T05:27:04.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 1 -- 172.21.15.114:0/3166295334 >> v2:172.21.15.149:6812/1185531135 conn(0x7f72f8037840 msgr2=0x7f72f8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 1 --2- 172.21.15.114:0/3166295334 >> v2:172.21.15.149:6812/1185531135 conn(0x7f72f8037840 0x7f72f8039d00 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 crypto rx=0x7f730c1d0080 tx=0x7f7300007810 comp rx=0 tx=0).stop 2024-09-16T05:27:04.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 10 monclient: shutdown 2024-09-16T05:27:04.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:04.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 1 -- 172.21.15.114:0/3166295334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f730c0a5d00 msgr2=0x7f730c135bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 1 --2- 172.21.15.114:0/3166295334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f730c0a5d00 0x7f730c135bf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f72fc018ff0 tx=0x7f72fc019180 comp rx=0 tx=0).stop 2024-09-16T05:27:04.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 1 -- 172.21.15.114:0/3166295334 shutdown_connections 2024-09-16T05:27:04.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 1 --2- 172.21.15.114:0/3166295334 >> v2:172.21.15.149:6812/1185531135 conn(0x7f72f8037840 0x7f72f8039d00 unknown :-1 s=CLOSED pgs=3636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 1 --2- 172.21.15.114:0/3166295334 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f730c1376e0 0x7f730c13bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 1 --2- 172.21.15.114:0/3166295334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f730c1370a0 0x7f730c136130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 1 --2- 172.21.15.114:0/3166295334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f730c0a5d00 0x7f730c135bf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 1 -- 172.21.15.114:0/3166295334 >> 172.21.15.114:0/3166295334 conn(0x7f730c0a9a40 msgr2=0x7f730c0a8040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:04.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 1 -- 172.21.15.114:0/3166295334 shutdown_connections 2024-09-16T05:27:04.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 1 -- 172.21.15.114:0/3166295334 wait complete. 2024-09-16T05:27:04.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.235+0000 7f73111a7640 1 librados: shutdown 2024-09-16T05:27:04.247 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T05:27:04.248 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T05:27:04.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 10 monclient: get_monmap_and_config 2024-09-16T05:27:04.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 10 monclient: build_initial_monmap 2024-09-16T05:27:04.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 10 monclient: monmap: 2024-09-16T05:27:04.367 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:04.367 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.367 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:04.372498+0000 2024-09-16T05:27:04.367 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:04.372498+0000 2024-09-16T05:27:04.367 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:04.367 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.367 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:04.368 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:04.368 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:04.368 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:04.368 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:04.368 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 1 Processor -- start 2024-09-16T05:27:04.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 1 -- start start 2024-09-16T05:27:04.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 10 monclient: init 2024-09-16T05:27:04.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:04.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-16T05:27:04.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9354137480 0x7f93540a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 10 monclient(hunting): picked mon.noname-b con 0x7f9354137480 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:04.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9354137860 0x7f93540a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 10 monclient(hunting): picked mon.noname-a con 0x7f9354137860 addr v2:172.21.15.114:3300/0 2024-09-16T05:27:04.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9354137c40 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:04.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f935406c250 con 0x7f9354137c40 2024-09-16T05:27:04.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f935406d940 con 0x7f9354137860 2024-09-16T05:27:04.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f93540a0af0 con 0x7f9354137480 2024-09-16T05:27:04.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:04.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935c61c640 10 monclient(hunting): authenticate will time out at 2109.001710s 2024-09-16T05:27:04.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9354137480 0x7f93540a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f9359bb7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9354137860 0x7f93540a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9354137480 0x7f93540a05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51816/0 (socket says 172.21.15.114:51816) 2024-09-16T05:27:04.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 1 -- 172.21.15.114:0/3135951961 learned_addr learned my addr 172.21.15.114:0/3135951961 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:04.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f9359bb7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9354137860 0x7f93540a66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35252/0 (socket says 172.21.15.114:35252) 2024-09-16T05:27:04.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 10 monclient(hunting): get_auth_request con 0x7f9354137480 auth_method 0 2024-09-16T05:27:04.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f9359bb7640 10 monclient(hunting): get_auth_request con 0x7f9354137860 auth_method 0 2024-09-16T05:27:04.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f9359bb7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f9359bb7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f9359bb7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:04.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:04.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:04.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f9359bb7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:04.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f9359bb7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:04.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f9359bb7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:04.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 10 monclient(hunting): handle_auth_done global_id 104429 payload 274 2024-09-16T05:27:04.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 1 -- 172.21.15.114:0/3135951961 >> v2:172.21.15.114:3300/0 conn(0x7f9354137860 msgr2=0x7f93540a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:04.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 1 --2- 172.21.15.114:0/3135951961 >> v2:172.21.15.114:3300/0 conn(0x7f9354137860 0x7f93540a66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 1 -- 172.21.15.114:0/3135951961 >> v1:172.21.15.114:6789/0 conn(0x7f9354137c40 legacy=0x7f93540a6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 10 monclient: _finish_hunting 0 2024-09-16T05:27:04.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 1 monclient: found mon.noname-b 2024-09-16T05:27:04.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:04.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:27:04.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.371+0000 7f935a3b8640 1 -- 172.21.15.114:0/3135951961 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f935406d020 con 0x7f9354137480 2024-09-16T05:27:04.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935a3b8640 1 --2- 172.21.15.114:0/3135951961 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9354137480 0x7f93540a05b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9350009870 tx=0x7f935001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa26c2130487119 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:04.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f93593b6640 1 -- 172.21.15.114:0/3135951961 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93500ab640 con 0x7f9354137480 2024-09-16T05:27:04.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f93593b6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f93593b6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:27:04.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f93593b6640 10 monclient: dump: 2024-09-16T05:27:04.381 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.381 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.381 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.381 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.381 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.381 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.381 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.382 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.382 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.382 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f93593b6640 10 monclient: _finish_auth 0 2024-09-16T05:27:04.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f93593b6640 10 monclient: _check_auth_tickets 2024-09-16T05:27:04.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f93593b6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:04.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 5 monclient: authenticate success, global_id 104429 2024-09-16T05:27:04.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f93593b6640 1 -- 172.21.15.114:0/3135951961 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93500b0070 con 0x7f9354137480 2024-09-16T05:27:04.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:04.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f93593b6640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:04.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:04.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 4 set_mon_vals no callback set 2024-09-16T05:27:04.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f93593b6640 1 -- 172.21.15.114:0/3135951961 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93500aba00 con 0x7f9354137480 2024-09-16T05:27:04.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f93593b6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f93593b6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:04.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f93593b6640 10 monclient: dump: 2024-09-16T05:27:04.383 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.383 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.383 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.383 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.383 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.383 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.383 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.383 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.383 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.383 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient: shutdown 2024-09-16T05:27:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 -- 172.21.15.114:0/3135951961 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9354137480 msgr2=0x7f93540a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 --2- 172.21.15.114:0/3135951961 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9354137480 0x7f93540a05b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9350009870 tx=0x7f935001c420 comp rx=0 tx=0).stop 2024-09-16T05:27:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 -- 172.21.15.114:0/3135951961 shutdown_connections 2024-09-16T05:27:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 --2- 172.21.15.114:0/3135951961 >> v2:172.21.15.114:3300/0 conn(0x7f9354137860 0x7f93540a66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 --2- 172.21.15.114:0/3135951961 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9354137480 0x7f93540a05b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 -- 172.21.15.114:0/3135951961 >> 172.21.15.114:0/3135951961 conn(0x7f93540a42e0 msgr2=0x7f93540a46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 -- 172.21.15.114:0/3135951961 shutdown_connections 2024-09-16T05:27:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 -- 172.21.15.114:0/3135951961 wait complete. 2024-09-16T05:27:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient: build_initial_monmap 2024-09-16T05:27:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient: monmap: 2024-09-16T05:27:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:04.385 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 librados: starting msgr at 2024-09-16T05:27:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 librados: starting objecter 2024-09-16T05:27:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 Processor -- start 2024-09-16T05:27:04.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 -- start start 2024-09-16T05:27:04.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 librados: setting wanted keys 2024-09-16T05:27:04.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 librados: calling monclient init 2024-09-16T05:27:04.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient: init 2024-09-16T05:27:04.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:04.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:27:04.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9354137480 0x7f93541ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient(hunting): picked mon.noname-c con 0x7f9354137480 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:04.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9354137860 0x7f93541352f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient(hunting): picked mon.noname-a con 0x7f9354137860 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:04.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9354137c40 0x7f935413d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient(hunting): picked mon.noname-b con 0x7f9354137c40 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:04.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9354141ca0 con 0x7f9354137860 2024-09-16T05:27:04.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f9354141b20 con 0x7f9354137c40 2024-09-16T05:27:04.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9354141e20 con 0x7f9354137480 2024-09-16T05:27:04.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:04.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient(hunting): authenticate will time out at 2109.007193s 2024-09-16T05:27:04.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935a3b8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9354137480 0x7f93541ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9354137860 0x7f93541352f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935a3b8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9354137480 0x7f93541ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51818/0 (socket says 172.21.15.114:51818) 2024-09-16T05:27:04.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935a3b8640 1 -- 172.21.15.114:0/2794041073 learned_addr learned my addr 172.21.15.114:0/2794041073 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:04.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935a3b8640 10 monclient(hunting): get_auth_request con 0x7f9354137480 auth_method 0 2024-09-16T05:27:04.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935a3b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935a3b8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935a3b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 10 monclient(hunting): get_auth_request con 0x7f9354137860 auth_method 0 2024-09-16T05:27:04.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935a3b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:04.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935a3b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:04.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935a3b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:04.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:04.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 10 monclient(hunting): handle_auth_done global_id 134220 payload 824 2024-09-16T05:27:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 1 -- 172.21.15.114:0/2794041073 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9354137480 msgr2=0x7f93541ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 1 --2- 172.21.15.114:0/2794041073 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9354137480 0x7f93541ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 1 -- 172.21.15.114:0/2794041073 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9354137c40 msgr2=0x7f935413d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 1 --2- 172.21.15.114:0/2794041073 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9354137c40 0x7f935413d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 10 monclient: _finish_hunting 0 2024-09-16T05:27:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 1 monclient: found mon.noname-a 2024-09-16T05:27:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 1 -- 172.21.15.114:0/2794041073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f935411fc80 con 0x7f9354137860 2024-09-16T05:27:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935a3b8640 1 --2- 172.21.15.114:0/2794041073 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9354137480 0x7f93541ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:27:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9359bb7640 1 --2- 172.21.15.114:0/2794041073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9354137860 0x7f93541352f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f934400e870 tx=0x7f934400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f934affd640 10 client.?.objecter ms_handle_connect 0x7f9354137860 2024-09-16T05:27:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f934affd640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f934affd640 1 -- 172.21.15.114:0/2794041073 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344017070 con 0x7f9354137860 2024-09-16T05:27:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f934affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f934affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f934affd640 10 monclient: dump: 2024-09-16T05:27:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.391 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.391 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.391 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.391 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f934affd640 10 monclient: _finish_auth 0 2024-09-16T05:27:04.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f934affd640 10 monclient: _check_auth_tickets 2024-09-16T05:27:04.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f934affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:04.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 5 monclient: authenticate success, global_id 134220 2024-09-16T05:27:04.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f934affd640 1 -- 172.21.15.114:0/2794041073 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f934400c940 con 0x7f9354137860 2024-09-16T05:27:04.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient: _renew_subs 2024-09-16T05:27:04.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:04.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 -- 172.21.15.114:0/2794041073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93540a3eb0 con 0x7f9354137860 2024-09-16T05:27:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f934affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f934affd640 1 -- 172.21.15.114:0/2794041073 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934400ccf0 con 0x7f9354137860 2024-09-16T05:27:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f934affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f934affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f934affd640 10 monclient: dump: 2024-09-16T05:27:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.393 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f9358bb5640 4 set_mon_vals no callback set 2024-09-16T05:27:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 client.134220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient: _renew_subs 2024-09-16T05:27:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 -- 172.21.15.114:0/2794041073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93540a6f90 con 0x7f9354137860 2024-09-16T05:27:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.375+0000 7f935c61c640 1 librados: init done 2024-09-16T05:27:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f935c61c640 10 client.134220.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 1 -- 172.21.15.114:0/2794041073 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f934401a070 con 0x7f9354137860 2024-09-16T05:27:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f935c61c640 15 client.134220.objecter _session_command_op_assign -1 1 2024-09-16T05:27:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f935c61c640 20 client.134220.objecter _get_session osd=-1 returning homeless 2024-09-16T05:27:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f935c61c640 10 client.134220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f935c61c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f935c61c640 1 -- 172.21.15.114:0/2794041073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9354068250 con 0x7f9354137860 2024-09-16T05:27:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 1 --2- 172.21.15.114:0/2794041073 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9338037840 0x7f9338039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 1 -- 172.21.15.114:0/2794041073 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7f9344048c00 con 0x7f9354137860 2024-09-16T05:27:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 10 client.134220.objecter ms_dispatch 0x7f9354000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 3 client.134220.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 3 client.134220.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 10 client.134220.objecter checking command 1 2024-09-16T05:27:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 1 --2- 172.21.15.114:0/2794041073 >> v2:172.21.15.114:6804/1932657732 conn(0x7f933803ecf0 0x7f9338041190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 20 client.134220.objecter _get_session s=0x7f933803e640 osd=2 3 2024-09-16T05:27:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 20 client.134220.objecter put_session s=0x7f933803e640 osd=2 3 2024-09-16T05:27:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 15 client.134220.objecter _session_command_op_remove -1 1 2024-09-16T05:27:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 20 client.134220.objecter _get_session s=0x7f933803e640 osd=2 3 2024-09-16T05:27:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 20 client.134220.objecter get_session s=0x7f933803e640 osd=2 3 2024-09-16T05:27:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 15 client.134220.objecter _session_command_op_assign 2 1 2024-09-16T05:27:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 20 client.134220.objecter put_session s=0x7f933803e640 osd=2 4 2024-09-16T05:27:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 10 client.134220.objecter _send_command 1 2024-09-16T05:27:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 1 -- 172.21.15.114:0/2794041073 --> v2:172.21.15.114:6804/1932657732 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9344022e50 con 0x7f933803ecf0 2024-09-16T05:27:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 20 client.134220.objecter dump_active .. 0 homeless 2024-09-16T05:27:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 1 -- 172.21.15.114:0/2794041073 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=695) ==== 24+0+0 (secure 0 0 0) 0x7f934404f050 con 0x7f9354137860 2024-09-16T05:27:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 10 monclient: handle_get_version_reply finishing 1 version 695 2024-09-16T05:27:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f935abb9640 1 --2- 172.21.15.114:0/2794041073 >> v2:172.21.15.114:6804/1932657732 conn(0x7f933803ecf0 0x7f9338041190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f935abb9640 10 monclient: get_auth_request con 0x7f933803ecf0 auth_method 0 2024-09-16T05:27:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f935a3b8640 1 --2- 172.21.15.114:0/2794041073 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9338037840 0x7f9338039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f935a3b8640 10 monclient: get_auth_request con 0x7f9338037840 auth_method 0 2024-09-16T05:27:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 10 client.134220.objecter ms_handle_connect 0x7f933803ecf0 2024-09-16T05:27:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f935abb9640 1 --2- 172.21.15.114:0/2794041073 >> v2:172.21.15.114:6804/1932657732 conn(0x7f933803ecf0 0x7f9338041190 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f935a3b8640 1 --2- 172.21.15.114:0/2794041073 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9338037840 0x7f9338039d00 secure :-1 s=READY pgs=3637 cs=0 l=1 rev1=1 crypto rx=0x7f935001c460 tx=0x7f93500a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 10 client.134220.objecter ms_handle_connect 0x7f9338037840 2024-09-16T05:27:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 1 -- 172.21.15.114:0/2794041073 <== osd.2 v2:172.21.15.114:6804/1932657732 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9344022e50 con 0x7f933803ecf0 2024-09-16T05:27:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 10 client.134220.objecter ms_dispatch 0x7f9354000ba0 command_reply(tid 1: 0 ) 2024-09-16T05:27:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 10 client.134220.objecter _finish_command 1 = system:0 2024-09-16T05:27:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 20 client.134220.objecter put_session s=0x7f933803e640 osd=2 4 2024-09-16T05:27:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.379+0000 7f934affd640 15 client.134220.objecter _session_command_op_remove 2 1 2024-09-16T05:27:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 10 client.134220.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T05:27:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 15 client.134220.objecter _session_command_op_assign -1 2 2024-09-16T05:27:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 20 client.134220.objecter _get_session s=0x7f933803e640 osd=2 3 2024-09-16T05:27:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 20 client.134220.objecter put_session s=0x7f933803e640 osd=2 3 2024-09-16T05:27:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 20 client.134220.objecter _get_session s=0x7f933803e640 osd=2 3 2024-09-16T05:27:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 15 client.134220.objecter _session_command_op_remove -1 2 2024-09-16T05:27:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 20 client.134220.objecter get_session s=0x7f933803e640 osd=2 3 2024-09-16T05:27:04.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 15 client.134220.objecter _session_command_op_assign 2 2 2024-09-16T05:27:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 20 client.134220.objecter put_session s=0x7f933803e640 osd=2 4 2024-09-16T05:27:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 10 client.134220.objecter _send_command 2 2024-09-16T05:27:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 1 -- 172.21.15.114:0/2794041073 --> v2:172.21.15.114:6804/1932657732 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f93540a05b0 con 0x7f933803ecf0 2024-09-16T05:27:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f934affd640 1 -- 172.21.15.114:0/2794041073 <== osd.2 v2:172.21.15.114:6804/1932657732 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f93540a05b0 con 0x7f933803ecf0 2024-09-16T05:27:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f934affd640 10 client.134220.objecter ms_dispatch 0x7f9354000ba0 command_reply(tid 2: 0 ) 2024-09-16T05:27:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f934affd640 10 client.134220.objecter _finish_command 2 = system:0 2024-09-16T05:27:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f934affd640 20 client.134220.objecter put_session s=0x7f933803e640 osd=2 4 2024-09-16T05:27:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f934affd640 15 client.134220.objecter _session_command_op_remove 2 2 2024-09-16T05:27:04.398 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:27:04.398 INFO:teuthology.orchestra.run.smithi114.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T05:27:04.399 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:27:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 10 librados: watch_flush enter 2024-09-16T05:27:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 10 librados: watch_flush exit 2024-09-16T05:27:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 10 client.134220.objecter close_session for osd.2 2024-09-16T05:27:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 1 -- 172.21.15.114:0/2794041073 >> v2:172.21.15.114:6804/1932657732 conn(0x7f933803ecf0 msgr2=0x7f9338041190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 1 --2- 172.21.15.114:0/2794041073 >> v2:172.21.15.114:6804/1932657732 conn(0x7f933803ecf0 0x7f9338041190 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 20 client.134220.objecter put_session s=0x7f933803e640 osd=2 1 2024-09-16T05:27:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 20 client.134220.objecter shutdown clearing up homeless session... 2024-09-16T05:27:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 10 client.134220.objecter successfully canceled tick 2024-09-16T05:27:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 1 -- 172.21.15.114:0/2794041073 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9338037840 msgr2=0x7f9338039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 1 --2- 172.21.15.114:0/2794041073 >> v2:172.21.15.149:6812/1185531135 conn(0x7f9338037840 0x7f9338039d00 secure :-1 s=READY pgs=3637 cs=0 l=1 rev1=1 crypto rx=0x7f935001c460 tx=0x7f93500a7000 comp rx=0 tx=0).stop 2024-09-16T05:27:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 10 monclient: shutdown 2024-09-16T05:27:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 1 -- 172.21.15.114:0/2794041073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9354137860 msgr2=0x7f93541352f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 1 --2- 172.21.15.114:0/2794041073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9354137860 0x7f93541352f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f934400e870 tx=0x7f934400ed30 comp rx=0 tx=0).stop 2024-09-16T05:27:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935abb9640 1 -- 172.21.15.114:0/2794041073 reap_dead start 2024-09-16T05:27:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 1 -- 172.21.15.114:0/2794041073 shutdown_connections 2024-09-16T05:27:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 1 -- 172.21.15.114:0/2794041073 >> 172.21.15.114:0/2794041073 conn(0x7f93540a42e0 msgr2=0x7f93540a1860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 1 -- 172.21.15.114:0/2794041073 shutdown_connections 2024-09-16T05:27:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 1 -- 172.21.15.114:0/2794041073 wait complete. 2024-09-16T05:27:04.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.391+0000 7f935c61c640 1 librados: shutdown 2024-09-16T05:27:04.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T05:27:04.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 10 monclient: get_monmap_and_config 2024-09-16T05:27:04.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 10 monclient: build_initial_monmap 2024-09-16T05:27:04.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 10 monclient: monmap: 2024-09-16T05:27:04.515 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:04.515 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.515 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:04.520791+0000 2024-09-16T05:27:04.515 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:04.520791+0000 2024-09-16T05:27:04.516 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:04.516 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.516 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:04.516 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:04.516 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:04.516 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:04.516 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:04.516 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 1 Processor -- start 2024-09-16T05:27:04.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 1 -- start start 2024-09-16T05:27:04.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 10 monclient: init 2024-09-16T05:27:04.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:04.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-16T05:27:04.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6bf4098500 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:04.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6bf41368a0 addr v1:172.21.15.149:6789/0 2024-09-16T05:27:04.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bf413a3b0 0x7f6bf413c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 10 monclient(hunting): picked mon.noname-b con 0x7f6bf413a3b0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:04.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bf409db80 con 0x7f6bf4098500 2024-09-16T05:27:04.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bf409e2d0 con 0x7f6bf41368a0 2024-09-16T05:27:04.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf409e070 con 0x7f6bf413a3b0 2024-09-16T05:27:04.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:04.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bfb161640 10 monclient(hunting): authenticate will time out at 2109.150595s 2024-09-16T05:27:04.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bf413a3b0 0x7f6bf413c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf8efd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6bf4098500 0x7f6bf40988e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37662/0 (socket says 172.21.15.114:37662) 2024-09-16T05:27:04.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf8efd640 1 -- 172.21.15.114:0/2454801804 learned_addr learned my addr 172.21.15.114:0/2454801804 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:27:04.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 10 monclient(hunting): get_auth_request con 0x7f6bf413a3b0 auth_method 0 2024-09-16T05:27:04.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 10 monclient(hunting): handle_auth_done global_id 104435 payload 274 2024-09-16T05:27:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 1 -- 172.21.15.114:0/2454801804 >> v1:172.21.15.149:6789/0 conn(0x7f6bf41368a0 legacy=0x7f6bf4138c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 1 -- 172.21.15.114:0/2454801804 >> v1:172.21.15.114:6789/0 conn(0x7f6bf4098500 legacy=0x7f6bf40988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 10 monclient: _finish_hunting 0 2024-09-16T05:27:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 1 monclient: found mon.noname-b 2024-09-16T05:27:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:27:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf96fe640 1 -- 172.21.15.114:0/2454801804 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf409f270 con 0x7f6bf413a3b0 2024-09-16T05:27:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf37fe640 1 -- 172.21.15.114:0/2454801804 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4046222087 0 0) 0x7f6bf409db80 con 0x7f6bf4098500 2024-09-16T05:27:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.519+0000 7f6bf37fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:27:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf96fe640 1 --2- 172.21.15.114:0/2454801804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bf413a3b0 0x7f6bf413c7a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6bec009dc0 tx=0x7f6bec01d920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d85d6e265a75b17 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf37fe640 1 -- 172.21.15.114:0/2454801804 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bec0b3070 con 0x7f6bf413a3b0 2024-09-16T05:27:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:27:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf37fe640 10 monclient: dump: 2024-09-16T05:27:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.530 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf37fe640 10 monclient: _finish_auth 0 2024-09-16T05:27:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:27:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf37fe640 1 -- 172.21.15.114:0/2454801804 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bec0a6780 con 0x7f6bf413a3b0 2024-09-16T05:27:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 5 monclient: authenticate success, global_id 104435 2024-09-16T05:27:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 4 set_mon_vals no callback set 2024-09-16T05:27:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf37fe640 1 -- 172.21.15.114:0/2454801804 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bec0a6b50 con 0x7f6bf413a3b0 2024-09-16T05:27:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf37fe640 10 monclient: dump: 2024-09-16T05:27:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.532 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.532 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 10 monclient: shutdown 2024-09-16T05:27:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 -- 172.21.15.114:0/2454801804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bf413a3b0 msgr2=0x7f6bf413c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 --2- 172.21.15.114:0/2454801804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bf413a3b0 0x7f6bf413c7a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6bec009dc0 tx=0x7f6bec01d920 comp rx=0 tx=0).stop 2024-09-16T05:27:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 -- 172.21.15.114:0/2454801804 shutdown_connections 2024-09-16T05:27:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 --2- 172.21.15.114:0/2454801804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bf413a3b0 0x7f6bf413c7a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 -- 172.21.15.114:0/2454801804 >> 172.21.15.114:0/2454801804 conn(0x7f6bf40a1fb0 msgr2=0x7f6bf40a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 -- 172.21.15.114:0/2454801804 shutdown_connections 2024-09-16T05:27:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 -- 172.21.15.114:0/2454801804 wait complete. 2024-09-16T05:27:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 10 monclient: build_initial_monmap 2024-09-16T05:27:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 10 monclient: monmap: 2024-09-16T05:27:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:04.537 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 librados: starting msgr at 2024-09-16T05:27:04.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 librados: starting objecter 2024-09-16T05:27:04.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 Processor -- start 2024-09-16T05:27:04.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 -- start start 2024-09-16T05:27:04.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 librados: setting wanted keys 2024-09-16T05:27:04.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 librados: calling monclient init 2024-09-16T05:27:04.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 10 monclient: init 2024-09-16T05:27:04.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:04.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:27:04.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6bf4098500 0x7f6bf406f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 10 monclient(hunting): picked mon.noname-b con 0x7f6bf4098500 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:04.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf41368a0 0x7f6bf406ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 10 monclient(hunting): picked mon.noname-a con 0x7f6bf41368a0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bf413a3b0 0x7f6bf4070470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 10 monclient(hunting): picked mon.noname-c con 0x7f6bf413a3b0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf409f550 con 0x7f6bf41368a0 2024-09-16T05:27:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf409a030 con 0x7f6bf4098500 2024-09-16T05:27:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf413ecf0 con 0x7f6bf413a3b0 2024-09-16T05:27:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bfb161640 10 monclient(hunting): authenticate will time out at 2109.155476s 2024-09-16T05:27:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf41368a0 0x7f6bf406ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf96fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bf413a3b0 0x7f6bf4070470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf41368a0 0x7f6bf406ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35262/0 (socket says 172.21.15.114:35262) 2024-09-16T05:27:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 1 -- 172.21.15.114:0/983320514 learned_addr learned my addr 172.21.15.114:0/983320514 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:04.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 10 monclient(hunting): get_auth_request con 0x7f6bf41368a0 auth_method 0 2024-09-16T05:27:04.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf96fe640 10 monclient(hunting): get_auth_request con 0x7f6bf413a3b0 auth_method 0 2024-09-16T05:27:04.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf96fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf96fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf96fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:04.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:04.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:04.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf96fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:04.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf96fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:04.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf96fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:04.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 10 monclient(hunting): handle_auth_done global_id 134226 payload 824 2024-09-16T05:27:04.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf8efd640 1 --2- 172.21.15.114:0/983320514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6bf4098500 0x7f6bf406f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 1 -- 172.21.15.114:0/983320514 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bf413a3b0 msgr2=0x7f6bf4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 1 --2- 172.21.15.114:0/983320514 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bf413a3b0 0x7f6bf4070470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 1 -- 172.21.15.114:0/983320514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6bf4098500 msgr2=0x7f6bf406f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 1 --2- 172.21.15.114:0/983320514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6bf4098500 0x7f6bf406f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 10 monclient: _finish_hunting 0 2024-09-16T05:27:04.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 1 monclient: found mon.noname-a 2024-09-16T05:27:04.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:04.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:04.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.523+0000 7f6bf3fff640 1 -- 172.21.15.114:0/983320514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf4071940 con 0x7f6bf41368a0 2024-09-16T05:27:04.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf3fff640 1 --2- 172.21.15.114:0/983320514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf41368a0 0x7f6bf406ff30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6be800c430 tx=0x7f6be800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 10 client.?.objecter ms_handle_connect 0x7f6bf41368a0 2024-09-16T05:27:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 1 -- 172.21.15.114:0/983320514 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8011790 con 0x7f6bf41368a0 2024-09-16T05:27:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 10 monclient: dump: 2024-09-16T05:27:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.543 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 10 monclient: _finish_auth 0 2024-09-16T05:27:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 10 monclient: _check_auth_tickets 2024-09-16T05:27:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 1 -- 172.21.15.114:0/983320514 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be8017070 con 0x7f6bf41368a0 2024-09-16T05:27:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bfb161640 5 monclient: authenticate success, global_id 134226 2024-09-16T05:27:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 1 -- 172.21.15.114:0/983320514 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8011bb0 con 0x7f6bf41368a0 2024-09-16T05:27:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 10 monclient: dump: 2024-09-16T05:27:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.544 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.544 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.544 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.544 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.544 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.544 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.544 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.544 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.544 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf2ffd640 4 set_mon_vals no callback set 2024-09-16T05:27:04.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bfb161640 10 monclient: _renew_subs 2024-09-16T05:27:04.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bfb161640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:04.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bfb161640 1 -- 172.21.15.114:0/983320514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf40725e0 con 0x7f6bf41368a0 2024-09-16T05:27:04.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bfb161640 10 client.134226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:04.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bfb161640 10 monclient: _renew_subs 2024-09-16T05:27:04.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bfb161640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:04.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bfb161640 1 -- 172.21.15.114:0/983320514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf409b5b0 con 0x7f6bf41368a0 2024-09-16T05:27:04.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bfb161640 1 librados: init done 2024-09-16T05:27:04.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bfb161640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:04.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bfb161640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:04.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bfb161640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:04.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 1 -- 172.21.15.114:0/983320514 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6be801c460 con 0x7f6bf41368a0 2024-09-16T05:27:04.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bfb161640 1 -- 172.21.15.114:0/983320514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf40988e0 con 0x7f6bf41368a0 2024-09-16T05:27:04.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 1 --2- 172.21.15.114:0/983320514 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6be0037810 0x7f6be0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 1 -- 172.21.15.114:0/983320514 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7f6be8048b00 con 0x7f6bf41368a0 2024-09-16T05:27:04.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 10 client.134226.objecter ms_dispatch 0x7f6bf4000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:04.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 3 client.134226.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:04.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 3 client.134226.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:04.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 20 client.134226.objecter dump_active .. 0 homeless 2024-09-16T05:27:04.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf8efd640 1 --2- 172.21.15.114:0/983320514 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6be0037810 0x7f6be0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf8efd640 10 monclient: get_auth_request con 0x7f6be0037810 auth_method 0 2024-09-16T05:27:04.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf8efd640 1 --2- 172.21.15.114:0/983320514 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6be0037810 0x7f6be0039cd0 secure :-1 s=READY pgs=3638 cs=0 l=1 rev1=1 crypto rx=0x7f6be4000f70 tx=0x7f6be4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:04.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.527+0000 7f6bf17fa640 10 client.134226.objecter ms_handle_connect 0x7f6be0037810 2024-09-16T05:27:04.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.531+0000 7f6bf17fa640 1 -- 172.21.15.114:0/983320514 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f6be801a520 con 0x7f6bf41368a0 2024-09-16T05:27:04.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.531+0000 7f6bf17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:04.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.531+0000 7f6bf17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:27:04.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 1 -- 172.21.15.114:0/983320514 --> v2:172.21.15.149:6812/1185531135 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f6bf40714a0 con 0x7f6be0037810 2024-09-16T05:27:04.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bf17fa640 1 -- 172.21.15.114:0/983320514 <== mgr.4108 v2:172.21.15.149:6812/1185531135 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6be4001cc0 con 0x7f6be0037810 2024-09-16T05:27:04.696 INFO:teuthology.orchestra.run.smithi114.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T05:27:04.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 10 librados: watch_flush enter 2024-09-16T05:27:04.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 10 librados: watch_flush exit 2024-09-16T05:27:04.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 20 client.134226.objecter shutdown clearing up homeless session... 2024-09-16T05:27:04.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 10 client.134226.objecter successfully canceled tick 2024-09-16T05:27:04.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 1 -- 172.21.15.114:0/983320514 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6be0037810 msgr2=0x7f6be0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 1 --2- 172.21.15.114:0/983320514 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6be0037810 0x7f6be0039cd0 secure :-1 s=READY pgs=3638 cs=0 l=1 rev1=1 crypto rx=0x7f6be4000f70 tx=0x7f6be4009040 comp rx=0 tx=0).stop 2024-09-16T05:27:04.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 10 monclient: shutdown 2024-09-16T05:27:04.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:04.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 1 -- 172.21.15.114:0/983320514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf41368a0 msgr2=0x7f6bf406ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 1 --2- 172.21.15.114:0/983320514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf41368a0 0x7f6bf406ff30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6be800c430 tx=0x7f6be800c8f0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 1 -- 172.21.15.114:0/983320514 shutdown_connections 2024-09-16T05:27:04.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 1 --2- 172.21.15.114:0/983320514 >> v2:172.21.15.149:6812/1185531135 conn(0x7f6be0037810 0x7f6be0039cd0 unknown :-1 s=CLOSED pgs=3638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 1 --2- 172.21.15.114:0/983320514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf41368a0 0x7f6bf406ff30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 1 --2- 172.21.15.114:0/983320514 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bf413a3b0 0x7f6bf4070470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 1 --2- 172.21.15.114:0/983320514 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6bf4098500 0x7f6bf406f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.699+0000 7f6bfb161640 1 -- 172.21.15.114:0/983320514 >> 172.21.15.114:0/983320514 conn(0x7f6bf40a1fb0 msgr2=0x7f6bf409a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:04.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.703+0000 7f6bfb161640 1 -- 172.21.15.114:0/983320514 shutdown_connections 2024-09-16T05:27:04.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.703+0000 7f6bfb161640 1 -- 172.21.15.114:0/983320514 wait complete. 2024-09-16T05:27:04.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.703+0000 7f6bfb161640 1 librados: shutdown 2024-09-16T05:27:04.715 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T05:27:04.715 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T05:27:04.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 10 monclient: get_monmap_and_config 2024-09-16T05:27:04.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 10 monclient: build_initial_monmap 2024-09-16T05:27:04.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 10 monclient: monmap: 2024-09-16T05:27:04.836 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:04.836 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.837 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:04.841129+0000 2024-09-16T05:27:04.837 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:04.841129+0000 2024-09-16T05:27:04.837 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:04.837 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.837 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:04.837 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:04.838 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:04.838 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:04.838 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:04.838 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 1 Processor -- start 2024-09-16T05:27:04.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 1 -- start start 2024-09-16T05:27:04.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 10 monclient: init 2024-09-16T05:27:04.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:04.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T05:27:04.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fa12409ca90 0x7fa12409ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 10 monclient(hunting): picked mon.noname-c con 0x7fa12409ca90 addr v2:172.21.15.149:3300/0 2024-09-16T05:27:04.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa12409d3b0 0x7fa1240980b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 10 monclient(hunting): picked mon.noname-a con 0x7fa12409d3b0 addr v2:172.21.15.114:3300/0 2024-09-16T05:27:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1240985f0 0x7fa124098a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 10 monclient(hunting): picked mon.noname-b con 0x7fa1240985f0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa12409de00 con 0x7fa12409d3b0 2024-09-16T05:27:04.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fa12409dfd0 con 0x7fa12409ca90 2024-09-16T05:27:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa12409f480 con 0x7fa1240985f0 2024-09-16T05:27:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa12b10b640 10 monclient(hunting): authenticate will time out at 2109.470714s 2024-09-16T05:27:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa12409d3b0 0x7fa1240980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa1296a8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1240985f0 0x7fa124098a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa1296a8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1240985f0 0x7fa124098a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51844/0 (socket says 172.21.15.114:51844) 2024-09-16T05:27:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa12409d3b0 0x7fa1240980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35276/0 (socket says 172.21.15.114:35276) 2024-09-16T05:27:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa1296a8640 1 -- 172.21.15.114:0/3398939478 learned_addr learned my addr 172.21.15.114:0/3398939478 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa1296a8640 10 monclient(hunting): get_auth_request con 0x7fa1240985f0 auth_method 0 2024-09-16T05:27:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa1296a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa1296a8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa1296a8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 10 monclient(hunting): get_auth_request con 0x7fa12409d3b0 auth_method 0 2024-09-16T05:27:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa1296a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:04.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa1296a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:04.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa1296a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa128ea7640 1 --2- 172.21.15.114:0/3398939478 >> v2:172.21.15.149:3300/0 conn(0x7fa12409ca90 0x7fa12409ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa128ea7640 10 monclient(hunting): get_auth_request con 0x7fa12409ca90 auth_method 0 2024-09-16T05:27:04.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa128ea7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa128ea7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa128ea7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 10 monclient(hunting): handle_auth_done global_id 134229 payload 274 2024-09-16T05:27:04.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 1 -- 172.21.15.114:0/3398939478 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1240985f0 msgr2=0x7fa124098a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:04.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 1 --2- 172.21.15.114:0/3398939478 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1240985f0 0x7fa124098a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 1 -- 172.21.15.114:0/3398939478 >> v2:172.21.15.149:3300/0 conn(0x7fa12409ca90 msgr2=0x7fa12409ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:04.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 1 --2- 172.21.15.114:0/3398939478 >> v2:172.21.15.149:3300/0 conn(0x7fa12409ca90 0x7fa12409ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:27:04.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 1 monclient: found mon.noname-a 2024-09-16T05:27:04.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:04.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:04.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa11bfff640 1 -- 172.21.15.114:0/3398939478 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa12409f760 con 0x7fa12409d3b0 2024-09-16T05:27:04.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.839+0000 7fa1296a8640 1 --2- 172.21.15.114:0/3398939478 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1240985f0 0x7fa124098a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:27:04.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa11bfff640 1 --2- 172.21.15.114:0/3398939478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa12409d3b0 0x7fa1240980b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa1240a1fd0 tx=0x7fa11401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47825077dad9b8fd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa11b7fe640 1 -- 172.21.15.114:0/3398939478 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1140b0070 con 0x7fa12409d3b0 2024-09-16T05:27:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa11b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa11b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa11b7fe640 10 monclient: dump: 2024-09-16T05:27:04.870 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.870 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.870 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.870 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.870 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.871 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa11b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:27:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa11b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:27:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa11b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa11b7fe640 1 -- 172.21.15.114:0/3398939478 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1140ab600 con 0x7fa12409d3b0 2024-09-16T05:27:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa11b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa11b7fe640 1 -- 172.21.15.114:0/3398939478 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1140ab9d0 con 0x7fa12409d3b0 2024-09-16T05:27:04.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 5 monclient: authenticate success, global_id 134229 2024-09-16T05:27:04.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:04.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 4 set_mon_vals no callback set 2024-09-16T05:27:04.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa11b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa11b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:04.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa11b7fe640 10 monclient: dump: 2024-09-16T05:27:04.874 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.874 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.874 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.874 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.874 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.874 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.875 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.875 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.875 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.875 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 10 monclient: shutdown 2024-09-16T05:27:04.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:04.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 -- 172.21.15.114:0/3398939478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa12409d3b0 msgr2=0x7fa1240980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 --2- 172.21.15.114:0/3398939478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa12409d3b0 0x7fa1240980b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa1240a1fd0 tx=0x7fa11401c340 comp rx=0 tx=0).stop 2024-09-16T05:27:04.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 -- 172.21.15.114:0/3398939478 shutdown_connections 2024-09-16T05:27:04.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 --2- 172.21.15.114:0/3398939478 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1240985f0 0x7fa124098a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 --2- 172.21.15.114:0/3398939478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa12409d3b0 0x7fa1240980b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 --2- 172.21.15.114:0/3398939478 >> v2:172.21.15.149:3300/0 conn(0x7fa12409ca90 0x7fa12409ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 -- 172.21.15.114:0/3398939478 >> 172.21.15.114:0/3398939478 conn(0x7fa1240a2490 msgr2=0x7fa1240a28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 -- 172.21.15.114:0/3398939478 shutdown_connections 2024-09-16T05:27:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 -- 172.21.15.114:0/3398939478 wait complete. 2024-09-16T05:27:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 10 monclient: build_initial_monmap 2024-09-16T05:27:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 10 monclient: monmap: 2024-09-16T05:27:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:04.879 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:04.879 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:04.879 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.879 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:04.879 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:04.879 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:04.879 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 librados: starting msgr at 2024-09-16T05:27:04.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 librados: starting objecter 2024-09-16T05:27:04.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 Processor -- start 2024-09-16T05:27:04.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 -- start start 2024-09-16T05:27:04.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 librados: setting wanted keys 2024-09-16T05:27:04.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 librados: calling monclient init 2024-09-16T05:27:04.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 10 monclient: init 2024-09-16T05:27:04.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:04.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:27:04.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1240985f0 0x7fa12406f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 10 monclient(hunting): picked mon.noname-c con 0x7fa1240985f0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:04.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa12409ca90 0x7fa12406fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 10 monclient(hunting): picked mon.noname-b con 0x7fa12409ca90 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa12409d3b0 0x7fa124070410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 10 monclient(hunting): picked mon.noname-a con 0x7fa12409d3b0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.843+0000 7fa12b10b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa12409a010 con 0x7fa12409d3b0 2024-09-16T05:27:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fa1240a1c90 con 0x7fa12409ca90 2024-09-16T05:27:04.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa124072750 con 0x7fa1240985f0 2024-09-16T05:27:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 10 monclient(hunting): authenticate will time out at 2109.476021s 2024-09-16T05:27:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1240985f0 0x7fa12406f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1240985f0 0x7fa12406f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51858/0 (socket says 172.21.15.114:51858) 2024-09-16T05:27:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1296a8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa12409d3b0 0x7fa124070410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 1 -- 172.21.15.114:0/205836007 learned_addr learned my addr 172.21.15.114:0/205836007 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 10 monclient(hunting): get_auth_request con 0x7fa1240985f0 auth_method 0 2024-09-16T05:27:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1296a8640 10 monclient(hunting): get_auth_request con 0x7fa12409d3b0 auth_method 0 2024-09-16T05:27:04.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1296a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1296a8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1296a8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1296a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1296a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1296a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa11bfff640 1 --2- 172.21.15.114:0/205836007 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa12409ca90 0x7fa12406fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa11bfff640 10 monclient(hunting): get_auth_request con 0x7fa12409ca90 auth_method 0 2024-09-16T05:27:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa11bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa11bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa11bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 10 monclient(hunting): handle_auth_done global_id 104444 payload 824 2024-09-16T05:27:04.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 1 -- 172.21.15.114:0/205836007 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa12409ca90 msgr2=0x7fa12406fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 1 --2- 172.21.15.114:0/205836007 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa12409ca90 0x7fa12406fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 1 -- 172.21.15.114:0/205836007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa12409d3b0 msgr2=0x7fa124070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 1 --2- 172.21.15.114:0/205836007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa12409d3b0 0x7fa124070410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 10 monclient: _finish_hunting 0 2024-09-16T05:27:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 1 monclient: found mon.noname-c 2024-09-16T05:27:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:27:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 1 -- 172.21.15.114:0/205836007 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1240a52d0 con 0x7fa1240985f0 2024-09-16T05:27:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1296a8640 1 --2- 172.21.15.114:0/205836007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa12409d3b0 0x7fa124070410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:27:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa128ea7640 1 --2- 172.21.15.114:0/205836007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1240985f0 0x7fa12406f990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa12000d800 tx=0x7fa12000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 client.?.objecter ms_handle_connect 0x7fa1240985f0 2024-09-16T05:27:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:04.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 1 -- 172.21.15.114:0/205836007 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12000b6c0 con 0x7fa1240985f0 2024-09-16T05:27:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:27:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 monclient: dump: 2024-09-16T05:27:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.885 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 monclient: _finish_auth 0 2024-09-16T05:27:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 monclient: _check_auth_tickets 2024-09-16T05:27:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 5 monclient: authenticate success, global_id 104444 2024-09-16T05:27:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 10 monclient: _renew_subs 2024-09-16T05:27:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 1 -- 172.21.15.114:0/205836007 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa120019070 con 0x7fa1240985f0 2024-09-16T05:27:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 1 -- 172.21.15.114:0/205836007 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1240a4d30 con 0x7fa1240985f0 2024-09-16T05:27:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 1 -- 172.21.15.114:0/205836007 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12000ba60 con 0x7fa1240985f0 2024-09-16T05:27:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 10 client.104444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:04.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 monclient: dump: 2024-09-16T05:27:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:04.887 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa11affd640 4 set_mon_vals no callback set 2024-09-16T05:27:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 10 monclient: _renew_subs 2024-09-16T05:27:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:04.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 1 -- 172.21.15.114:0/205836007 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa124073260 con 0x7fa1240985f0 2024-09-16T05:27:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 1 librados: init done 2024-09-16T05:27:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 10 client.104444.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 15 client.104444.objecter _session_command_op_assign -1 1 2024-09-16T05:27:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 20 client.104444.objecter _get_session osd=-1 returning homeless 2024-09-16T05:27:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 10 client.104444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa12b10b640 1 -- 172.21.15.114:0/205836007 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa12409ce70 con 0x7fa1240985f0 2024-09-16T05:27:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 1 -- 172.21.15.114:0/205836007 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa12000bc00 con 0x7fa1240985f0 2024-09-16T05:27:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 1 --2- 172.21.15.114:0/205836007 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa110037810 0x7fa110039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 1 -- 172.21.15.114:0/205836007 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7fa120048f00 con 0x7fa1240985f0 2024-09-16T05:27:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 client.104444.objecter ms_dispatch 0x7fa124000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 3 client.104444.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 3 client.104444.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 client.104444.objecter checking command 1 2024-09-16T05:27:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 1 --2- 172.21.15.114:0/205836007 >> v2:172.21.15.149:6805/1362772130 conn(0x7fa11003ecb0 0x7fa110041150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 20 client.104444.objecter _get_session s=0x7fa11003e600 osd=3 3 2024-09-16T05:27:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 20 client.104444.objecter put_session s=0x7fa11003e600 osd=3 3 2024-09-16T05:27:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 15 client.104444.objecter _session_command_op_remove -1 1 2024-09-16T05:27:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 20 client.104444.objecter _get_session s=0x7fa11003e600 osd=3 3 2024-09-16T05:27:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 20 client.104444.objecter get_session s=0x7fa11003e600 osd=3 3 2024-09-16T05:27:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 15 client.104444.objecter _session_command_op_assign 3 1 2024-09-16T05:27:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 20 client.104444.objecter put_session s=0x7fa11003e600 osd=3 4 2024-09-16T05:27:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 client.104444.objecter _send_command 1 2024-09-16T05:27:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 1 -- 172.21.15.114:0/205836007 --> v2:172.21.15.149:6805/1362772130 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa1100439f0 con 0x7fa11003ecb0 2024-09-16T05:27:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 20 client.104444.objecter dump_active .. 0 homeless 2024-09-16T05:27:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 1 -- 172.21.15.114:0/205836007 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_get_version_reply(handle=1 version=695) ==== 24+0+0 (secure 0 0 0) 0x7fa12004b5d0 con 0x7fa1240985f0 2024-09-16T05:27:04.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 monclient: handle_get_version_reply finishing 1 version 695 2024-09-16T05:27:04.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa11bfff640 1 --2- 172.21.15.114:0/205836007 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa110037810 0x7fa110039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa11bfff640 10 monclient: get_auth_request con 0x7fa110037810 auth_method 0 2024-09-16T05:27:04.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1296a8640 1 --2- 172.21.15.114:0/205836007 >> v2:172.21.15.149:6805/1362772130 conn(0x7fa11003ecb0 0x7fa110041150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:04.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1296a8640 10 monclient: get_auth_request con 0x7fa11003ecb0 auth_method 0 2024-09-16T05:27:04.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa11bfff640 1 --2- 172.21.15.114:0/205836007 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa110037810 0x7fa110039cd0 secure :-1 s=READY pgs=3639 cs=0 l=1 rev1=1 crypto rx=0x7fa1140ad000 tx=0x7fa114008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:04.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 client.104444.objecter ms_handle_connect 0x7fa110037810 2024-09-16T05:27:04.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1197fa640 10 client.104444.objecter ms_handle_connect 0x7fa11003ecb0 2024-09-16T05:27:04.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.847+0000 7fa1296a8640 1 --2- 172.21.15.114:0/205836007 >> v2:172.21.15.149:6805/1362772130 conn(0x7fa11003ecb0 0x7fa110041150 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:04.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.851+0000 7fa1197fa640 1 -- 172.21.15.114:0/205836007 <== osd.3 v2:172.21.15.149:6805/1362772130 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa1100439f0 con 0x7fa11003ecb0 2024-09-16T05:27:04.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.851+0000 7fa1197fa640 10 client.104444.objecter ms_dispatch 0x7fa124000ba0 command_reply(tid 1: 0 ) 2024-09-16T05:27:04.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.851+0000 7fa1197fa640 10 client.104444.objecter _finish_command 1 = system:0 2024-09-16T05:27:04.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.851+0000 7fa1197fa640 20 client.104444.objecter put_session s=0x7fa11003e600 osd=3 4 2024-09-16T05:27:04.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.851+0000 7fa1197fa640 15 client.104444.objecter _session_command_op_remove 3 1 2024-09-16T05:27:04.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 10 client.104444.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T05:27:04.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 15 client.104444.objecter _session_command_op_assign -1 2 2024-09-16T05:27:04.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 20 client.104444.objecter _get_session s=0x7fa11003e600 osd=3 3 2024-09-16T05:27:04.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 20 client.104444.objecter put_session s=0x7fa11003e600 osd=3 3 2024-09-16T05:27:04.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 20 client.104444.objecter _get_session s=0x7fa11003e600 osd=3 3 2024-09-16T05:27:04.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 15 client.104444.objecter _session_command_op_remove -1 2 2024-09-16T05:27:04.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 20 client.104444.objecter get_session s=0x7fa11003e600 osd=3 3 2024-09-16T05:27:04.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 15 client.104444.objecter _session_command_op_assign 3 2 2024-09-16T05:27:04.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 20 client.104444.objecter put_session s=0x7fa11003e600 osd=3 4 2024-09-16T05:27:04.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 10 client.104444.objecter _send_command 2 2024-09-16T05:27:04.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 1 -- 172.21.15.114:0/205836007 --> v2:172.21.15.149:6805/1362772130 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa124070950 con 0x7fa11003ecb0 2024-09-16T05:27:04.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa1197fa640 1 -- 172.21.15.114:0/205836007 <== osd.3 v2:172.21.15.149:6805/1362772130 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa124070950 con 0x7fa11003ecb0 2024-09-16T05:27:04.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa1197fa640 10 client.104444.objecter ms_dispatch 0x7fa124000ba0 command_reply(tid 2: 0 ) 2024-09-16T05:27:04.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa1197fa640 10 client.104444.objecter _finish_command 2 = system:0 2024-09-16T05:27:04.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa1197fa640 20 client.104444.objecter put_session s=0x7fa11003e600 osd=3 4 2024-09-16T05:27:04.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa1197fa640 15 client.104444.objecter _session_command_op_remove 3 2 2024-09-16T05:27:04.893 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:27:04.893 INFO:teuthology.orchestra.run.smithi114.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T05:27:04.894 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:27:04.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 10 librados: watch_flush enter 2024-09-16T05:27:04.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 10 librados: watch_flush exit 2024-09-16T05:27:04.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 10 client.104444.objecter close_session for osd.3 2024-09-16T05:27:04.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 1 -- 172.21.15.114:0/205836007 >> v2:172.21.15.149:6805/1362772130 conn(0x7fa11003ecb0 msgr2=0x7fa110041150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 1 --2- 172.21.15.114:0/205836007 >> v2:172.21.15.149:6805/1362772130 conn(0x7fa11003ecb0 0x7fa110041150 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 20 client.104444.objecter put_session s=0x7fa11003e600 osd=3 1 2024-09-16T05:27:04.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 20 client.104444.objecter shutdown clearing up homeless session... 2024-09-16T05:27:04.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 10 client.104444.objecter successfully canceled tick 2024-09-16T05:27:04.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 1 -- 172.21.15.114:0/205836007 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa110037810 msgr2=0x7fa110039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 1 --2- 172.21.15.114:0/205836007 >> v2:172.21.15.149:6812/1185531135 conn(0x7fa110037810 0x7fa110039cd0 secure :-1 s=READY pgs=3639 cs=0 l=1 rev1=1 crypto rx=0x7fa1140ad000 tx=0x7fa114008040 comp rx=0 tx=0).stop 2024-09-16T05:27:04.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 10 monclient: shutdown 2024-09-16T05:27:04.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:04.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 1 -- 172.21.15.114:0/205836007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1240985f0 msgr2=0x7fa12406f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:04.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 1 --2- 172.21.15.114:0/205836007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1240985f0 0x7fa12406f990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa12000d800 tx=0x7fa12000dcc0 comp rx=0 tx=0).stop 2024-09-16T05:27:04.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa1296a8640 1 -- 172.21.15.114:0/205836007 reap_dead start 2024-09-16T05:27:04.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 1 -- 172.21.15.114:0/205836007 shutdown_connections 2024-09-16T05:27:04.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 1 -- 172.21.15.114:0/205836007 >> 172.21.15.114:0/205836007 conn(0x7fa1240a2490 msgr2=0x7fa12409a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:04.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 1 -- 172.21.15.114:0/205836007 shutdown_connections 2024-09-16T05:27:04.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 1 -- 172.21.15.114:0/205836007 wait complete. 2024-09-16T05:27:04.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:04.863+0000 7fa12b10b640 1 librados: shutdown 2024-09-16T05:27:04.897 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T05:27:05.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 10 monclient: get_monmap_and_config 2024-09-16T05:27:05.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 10 monclient: build_initial_monmap 2024-09-16T05:27:05.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 10 monclient: monmap: 2024-09-16T05:27:05.012 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:05.012 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.012 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:05.017002+0000 2024-09-16T05:27:05.012 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:05.017002+0000 2024-09-16T05:27:05.012 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:05.012 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.012 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:05.012 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:05.012 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:05.012 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:05.012 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:05.012 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 1 Processor -- start 2024-09-16T05:27:05.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 1 -- start start 2024-09-16T05:27:05.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 10 monclient: init 2024-09-16T05:27:05.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:05.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T05:27:05.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd13409b6c0 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:05.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1340a15d0 0x7fd13413a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 10 monclient(hunting): picked mon.noname-b con 0x7fd1340a15d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:05.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1340a19e0 addr v1:172.21.15.149:6789/0 2024-09-16T05:27:05.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd13409d4e0 con 0x7fd13409b6c0 2024-09-16T05:27:05.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd13409f430 con 0x7fd1340a19e0 2024-09-16T05:27:05.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd13409e390 con 0x7fd1340a15d0 2024-09-16T05:27:05.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:05.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd13a8ea640 10 monclient(hunting): authenticate will time out at 2109.646140s 2024-09-16T05:27:05.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1340a15d0 0x7fd13413a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd133fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd13409b6c0 0x7fd1340a0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37668/0 (socket says 172.21.15.114:37668) 2024-09-16T05:27:05.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd133fff640 1 -- 172.21.15.114:0/2002793741 learned_addr learned my addr 172.21.15.114:0/2002793741 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:27:05.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1340a15d0 0x7fd13413a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51862/0 (socket says 172.21.15.114:51862) 2024-09-16T05:27:05.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 10 monclient(hunting): get_auth_request con 0x7fd1340a15d0 auth_method 0 2024-09-16T05:27:05.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:05.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:05.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:05.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 10 monclient(hunting): handle_auth_done global_id 104447 payload 274 2024-09-16T05:27:05.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 1 -- 172.21.15.114:0/2002793741 >> v1:172.21.15.149:6789/0 conn(0x7fd1340a19e0 legacy=0x7fd13413cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 1 -- 172.21.15.114:0/2002793741 >> v1:172.21.15.114:6789/0 conn(0x7fd13409b6c0 legacy=0x7fd1340a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 10 monclient: _finish_hunting 0 2024-09-16T05:27:05.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 1 monclient: found mon.noname-b 2024-09-16T05:27:05.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:05.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:27:05.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd132ffd640 1 -- 172.21.15.114:0/2002793741 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1390904607 0 0) 0x7fd13409d4e0 con 0x7fd13409b6c0 2024-09-16T05:27:05.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd1337fe640 1 -- 172.21.15.114:0/2002793741 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd13409c2f0 con 0x7fd1340a15d0 2024-09-16T05:27:05.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.015+0000 7fd132ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:27:05.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd1337fe640 1 --2- 172.21.15.114:0/2002793741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1340a15d0 0x7fd13413a650 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd1240097d0 tx=0x7fd12401c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f7ab232e2aa0adf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd132ffd640 1 -- 172.21.15.114:0/2002793741 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1240b0070 con 0x7fd1340a15d0 2024-09-16T05:27:05.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd132ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd132ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:27:05.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd132ffd640 10 monclient: dump: 2024-09-16T05:27:05.025 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.025 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.025 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.025 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.025 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.025 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.026 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.026 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.026 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.026 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd132ffd640 10 monclient: _finish_auth 0 2024-09-16T05:27:05.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd132ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:27:05.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:05.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd132ffd640 1 -- 172.21.15.114:0/2002793741 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1240ab600 con 0x7fd1340a15d0 2024-09-16T05:27:05.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 5 monclient: authenticate success, global_id 104447 2024-09-16T05:27:05.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:05.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd132ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:05.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:05.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd132ffd640 1 -- 172.21.15.114:0/2002793741 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1240ab9d0 con 0x7fd1340a15d0 2024-09-16T05:27:05.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 4 set_mon_vals no callback set 2024-09-16T05:27:05.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd132ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd132ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:05.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd132ffd640 10 monclient: dump: 2024-09-16T05:27:05.027 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.027 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.027 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.027 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.028 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.028 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.028 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.028 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.028 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.028 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 10 monclient: shutdown 2024-09-16T05:27:05.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:05.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 -- 172.21.15.114:0/2002793741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1340a15d0 msgr2=0x7fd13413a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 --2- 172.21.15.114:0/2002793741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1340a15d0 0x7fd13413a650 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd1240097d0 tx=0x7fd12401c380 comp rx=0 tx=0).stop 2024-09-16T05:27:05.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 -- 172.21.15.114:0/2002793741 shutdown_connections 2024-09-16T05:27:05.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 --2- 172.21.15.114:0/2002793741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd1340a15d0 0x7fd13413a650 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 -- 172.21.15.114:0/2002793741 >> 172.21.15.114:0/2002793741 conn(0x7fd1340a1fb0 msgr2=0x7fd1340a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:05.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 -- 172.21.15.114:0/2002793741 shutdown_connections 2024-09-16T05:27:05.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 -- 172.21.15.114:0/2002793741 wait complete. 2024-09-16T05:27:05.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 10 monclient: build_initial_monmap 2024-09-16T05:27:05.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 10 monclient: monmap: 2024-09-16T05:27:05.029 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:05.029 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:05.029 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:05.029 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:05.029 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:05.029 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.030 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:05.030 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:05.030 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:05.030 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 librados: starting msgr at 2024-09-16T05:27:05.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 librados: starting objecter 2024-09-16T05:27:05.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 Processor -- start 2024-09-16T05:27:05.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 -- start start 2024-09-16T05:27:05.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 librados: setting wanted keys 2024-09-16T05:27:05.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 librados: calling monclient init 2024-09-16T05:27:05.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 10 monclient: init 2024-09-16T05:27:05.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:05.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:27:05.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13409b6c0 0x7fd13413db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 10 monclient(hunting): picked mon.noname-c con 0x7fd13409b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:05.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd1340a15d0 0x7fd1340918b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 10 monclient(hunting): picked mon.noname-b con 0x7fd1340a15d0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:05.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1340a19e0 0x7fd134091df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 10 monclient(hunting): picked mon.noname-a con 0x7fd1340a19e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:05.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd134140c60 con 0x7fd1340a19e0 2024-09-16T05:27:05.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fd134140ae0 con 0x7fd1340a15d0 2024-09-16T05:27:05.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd134140de0 con 0x7fd13409b6c0 2024-09-16T05:27:05.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:05.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd13a8ea640 10 monclient(hunting): authenticate will time out at 2109.651902s 2024-09-16T05:27:05.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.019+0000 7fd133fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13409b6c0 0x7fd13413db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1340a19e0 0x7fd134091df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd133fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13409b6c0 0x7fd13413db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51872/0 (socket says 172.21.15.114:51872) 2024-09-16T05:27:05.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd133fff640 1 -- 172.21.15.114:0/2796775609 learned_addr learned my addr 172.21.15.114:0/2796775609 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:05.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd133fff640 10 monclient(hunting): get_auth_request con 0x7fd13409b6c0 auth_method 0 2024-09-16T05:27:05.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd133fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd133fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd133fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 10 monclient(hunting): get_auth_request con 0x7fd1340a19e0 auth_method 0 2024-09-16T05:27:05.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:05.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:05.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:05.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd133fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd133fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd133fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd1337fe640 1 --2- 172.21.15.114:0/2796775609 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd1340a15d0 0x7fd1340918b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd1337fe640 10 monclient(hunting): get_auth_request con 0x7fd1340a15d0 auth_method 0 2024-09-16T05:27:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd1337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd1337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd1337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 10 monclient(hunting): handle_auth_done global_id 134238 payload 824 2024-09-16T05:27:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 1 -- 172.21.15.114:0/2796775609 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13409b6c0 msgr2=0x7fd13413db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 1 --2- 172.21.15.114:0/2796775609 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13409b6c0 0x7fd13413db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 1 -- 172.21.15.114:0/2796775609 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd1340a15d0 msgr2=0x7fd1340918b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 1 --2- 172.21.15.114:0/2796775609 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd1340a15d0 0x7fd1340918b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 10 monclient: _finish_hunting 0 2024-09-16T05:27:05.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 1 monclient: found mon.noname-a 2024-09-16T05:27:05.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:05.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:05.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 1 -- 172.21.15.114:0/2796775609 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd134094070 con 0x7fd1340a19e0 2024-09-16T05:27:05.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd138e87640 1 --2- 172.21.15.114:0/2796775609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1340a19e0 0x7fd134091df0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd12c00c700 tx=0x7fd12c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 10 client.?.objecter ms_handle_connect 0x7fd1340a19e0 2024-09-16T05:27:05.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:05.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 1 -- 172.21.15.114:0/2796775609 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c003b80 con 0x7fd1340a19e0 2024-09-16T05:27:05.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:05.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 10 monclient: dump: 2024-09-16T05:27:05.036 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.036 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.036 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.036 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.037 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 10 monclient: _finish_auth 0 2024-09-16T05:27:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:27:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 1 -- 172.21.15.114:0/2796775609 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd12c003d20 con 0x7fd1340a19e0 2024-09-16T05:27:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd13a8ea640 5 monclient: authenticate success, global_id 134238 2024-09-16T05:27:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd13a8ea640 10 monclient: _renew_subs 2024-09-16T05:27:05.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd13a8ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:05.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd13a8ea640 1 -- 172.21.15.114:0/2796775609 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd134095520 con 0x7fd1340a19e0 2024-09-16T05:27:05.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:05.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 1 -- 172.21.15.114:0/2796775609 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c00aa30 con 0x7fd1340a19e0 2024-09-16T05:27:05.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:05.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 10 monclient: dump: 2024-09-16T05:27:05.038 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.038 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.038 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.038 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.038 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.038 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.038 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.039 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.039 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.039 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd13a8ea640 10 client.134238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:05.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd13a8ea640 10 monclient: _renew_subs 2024-09-16T05:27:05.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd13a8ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:05.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd13a8ea640 1 -- 172.21.15.114:0/2796775609 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd134095240 con 0x7fd1340a19e0 2024-09-16T05:27:05.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd13a8ea640 1 librados: init done 2024-09-16T05:27:05.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd1327fc640 4 set_mon_vals no callback set 2024-09-16T05:27:05.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd13a8ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:05.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd13a8ea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:05.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd13a8ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:05.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd13a8ea640 1 -- 172.21.15.114:0/2796775609 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1340a0ec0 con 0x7fd1340a19e0 2024-09-16T05:27:05.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 1 -- 172.21.15.114:0/2796775609 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd12c00abd0 con 0x7fd1340a19e0 2024-09-16T05:27:05.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 1 --2- 172.21.15.114:0/2796775609 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd120037840 0x7fd120039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 1 -- 172.21.15.114:0/2796775609 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7fd12c0489b0 con 0x7fd1340a19e0 2024-09-16T05:27:05.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 10 client.134238.objecter ms_dispatch 0x7fd134000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:05.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 3 client.134238.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:05.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 3 client.134238.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:05.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 20 client.134238.objecter dump_active .. 0 homeless 2024-09-16T05:27:05.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd133fff640 1 --2- 172.21.15.114:0/2796775609 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd120037840 0x7fd120039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd133fff640 10 monclient: get_auth_request con 0x7fd120037840 auth_method 0 2024-09-16T05:27:05.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd130ff9640 10 client.134238.objecter ms_handle_connect 0x7fd120037840 2024-09-16T05:27:05.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.023+0000 7fd133fff640 1 --2- 172.21.15.114:0/2796775609 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd120037840 0x7fd120039d00 secure :-1 s=READY pgs=3640 cs=0 l=1 rev1=1 crypto rx=0x7fd128000f70 tx=0x7fd128007ee0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.027+0000 7fd130ff9640 1 -- 172.21.15.114:0/2796775609 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd12c005180 con 0x7fd1340a19e0 2024-09-16T05:27:05.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.027+0000 7fd130ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:05.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.027+0000 7fd130ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:27:05.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.179+0000 7fd13a8ea640 1 -- 172.21.15.114:0/2796775609 --> v2:172.21.15.149:6812/1185531135 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fd13413e070 con 0x7fd120037840 2024-09-16T05:27:05.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.179+0000 7fd130ff9640 1 -- 172.21.15.114:0/2796775609 <== mgr.4108 v2:172.21.15.149:6812/1185531135 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd13413e070 con 0x7fd120037840 2024-09-16T05:27:05.177 INFO:teuthology.orchestra.run.smithi114.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T05:27:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 10 librados: watch_flush enter 2024-09-16T05:27:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 10 librados: watch_flush exit 2024-09-16T05:27:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 20 client.134238.objecter shutdown clearing up homeless session... 2024-09-16T05:27:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 10 client.134238.objecter successfully canceled tick 2024-09-16T05:27:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 1 -- 172.21.15.114:0/2796775609 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd120037840 msgr2=0x7fd120039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 1 --2- 172.21.15.114:0/2796775609 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd120037840 0x7fd120039d00 secure :-1 s=READY pgs=3640 cs=0 l=1 rev1=1 crypto rx=0x7fd128000f70 tx=0x7fd128007ee0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 10 monclient: shutdown 2024-09-16T05:27:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:05.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 1 -- 172.21.15.114:0/2796775609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1340a19e0 msgr2=0x7fd134091df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 1 --2- 172.21.15.114:0/2796775609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1340a19e0 0x7fd134091df0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd12c00c700 tx=0x7fd12c00cbc0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 1 -- 172.21.15.114:0/2796775609 shutdown_connections 2024-09-16T05:27:05.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 1 --2- 172.21.15.114:0/2796775609 >> v2:172.21.15.149:6812/1185531135 conn(0x7fd120037840 0x7fd120039d00 unknown :-1 s=CLOSED pgs=3640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 1 --2- 172.21.15.114:0/2796775609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1340a19e0 0x7fd134091df0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 1 --2- 172.21.15.114:0/2796775609 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd1340a15d0 0x7fd1340918b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 1 --2- 172.21.15.114:0/2796775609 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd13409b6c0 0x7fd13413db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 1 -- 172.21.15.114:0/2796775609 >> 172.21.15.114:0/2796775609 conn(0x7fd1340a1fb0 msgr2=0x7fd134071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:05.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 1 -- 172.21.15.114:0/2796775609 shutdown_connections 2024-09-16T05:27:05.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 1 -- 172.21.15.114:0/2796775609 wait complete. 2024-09-16T05:27:05.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.183+0000 7fd13a8ea640 1 librados: shutdown 2024-09-16T05:27:05.194 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T05:27:05.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T05:27:05.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.307+0000 7fc512371640 10 monclient: get_monmap_and_config 2024-09-16T05:27:05.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.307+0000 7fc512371640 10 monclient: build_initial_monmap 2024-09-16T05:27:05.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.307+0000 7fc512371640 10 monclient: monmap: 2024-09-16T05:27:05.307 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:05.307 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.307 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:05.311601+0000 2024-09-16T05:27:05.307 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:05.311601+0000 2024-09-16T05:27:05.307 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:05.307 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.307 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:05.307 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:05.307 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:05.307 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:05.308 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:05.308 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 1 Processor -- start 2024-09-16T05:27:05.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 1 -- start start 2024-09-16T05:27:05.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 10 monclient: init 2024-09-16T05:27:05.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T05:27:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc50c0a11a0 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc50c09d1a0 0x7fc50c13a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 10 monclient(hunting): picked mon.noname-b con 0x7fc50c09d1a0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc50c09d580 addr v1:172.21.15.149:6789/0 2024-09-16T05:27:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc50c09df90 con 0x7fc50c0a11a0 2024-09-16T05:27:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc50c09dcb0 con 0x7fc50c09d580 2024-09-16T05:27:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc50c0a1580 con 0x7fc50c09d1a0 2024-09-16T05:27:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 10 monclient(hunting): authenticate will time out at 2109.940830s 2024-09-16T05:27:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc50c09d1a0 0x7fc50c13a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50bfff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc50c0a11a0 0x7fc50c09ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37676/0 (socket says 172.21.15.114:37676) 2024-09-16T05:27:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50bfff640 1 -- 172.21.15.114:0/238625079 learned_addr learned my addr 172.21.15.114:0/238625079 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:27:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 10 monclient(hunting): get_auth_request con 0x7fc50c09d1a0 auth_method 0 2024-09-16T05:27:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 10 monclient(hunting): handle_auth_done global_id 104453 payload 274 2024-09-16T05:27:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 1 -- 172.21.15.114:0/238625079 >> v1:172.21.15.149:6789/0 conn(0x7fc50c09d580 legacy=0x7fc50c13cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 1 -- 172.21.15.114:0/238625079 >> v1:172.21.15.114:6789/0 conn(0x7fc50c0a11a0 legacy=0x7fc50c09ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:27:05.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 1 monclient: found mon.noname-b 2024-09-16T05:27:05.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50affd640 1 -- 172.21.15.114:0/238625079 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1572806486 0 0) 0x7fc4f4001120 con 0x7fc50c0a11a0 2024-09-16T05:27:05.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:05.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:27:05.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 1 -- 172.21.15.114:0/238625079 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc50c099150 con 0x7fc50c09d1a0 2024-09-16T05:27:05.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50affd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:27:05.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50b7fe640 1 --2- 172.21.15.114:0/238625079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc50c09d1a0 0x7fc50c13a730 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc5040097f0 tx=0x7fc50401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44ac30b9f6b690ce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50affd640 1 -- 172.21.15.114:0/238625079 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5040b0070 con 0x7fc50c09d1a0 2024-09-16T05:27:05.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:27:05.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50affd640 10 monclient: dump: 2024-09-16T05:27:05.325 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.325 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.326 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.326 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.326 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.326 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.326 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.326 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.326 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.326 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50affd640 10 monclient: _finish_auth 0 2024-09-16T05:27:05.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50affd640 10 monclient: _check_auth_tickets 2024-09-16T05:27:05.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:05.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50affd640 1 -- 172.21.15.114:0/238625079 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5040ab600 con 0x7fc50c09d1a0 2024-09-16T05:27:05.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:05.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50affd640 1 -- 172.21.15.114:0/238625079 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5040ab9d0 con 0x7fc50c09d1a0 2024-09-16T05:27:05.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:05.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc50affd640 10 monclient: dump: 2024-09-16T05:27:05.327 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.327 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.327 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.327 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.327 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.327 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.327 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.327 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.327 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.327 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 5 monclient: authenticate success, global_id 104453 2024-09-16T05:27:05.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:05.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.311+0000 7fc512371640 4 set_mon_vals no callback set 2024-09-16T05:27:05.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient: shutdown 2024-09-16T05:27:05.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:05.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 -- 172.21.15.114:0/238625079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc50c09d1a0 msgr2=0x7fc50c13a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 --2- 172.21.15.114:0/238625079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc50c09d1a0 0x7fc50c13a730 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc5040097f0 tx=0x7fc50401c3f0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 -- 172.21.15.114:0/238625079 shutdown_connections 2024-09-16T05:27:05.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 --2- 172.21.15.114:0/238625079 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc50c09d1a0 0x7fc50c13a730 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 -- 172.21.15.114:0/238625079 >> 172.21.15.114:0/238625079 conn(0x7fc50c0a2490 msgr2=0x7fc50c0a28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:05.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 -- 172.21.15.114:0/238625079 shutdown_connections 2024-09-16T05:27:05.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 -- 172.21.15.114:0/238625079 wait complete. 2024-09-16T05:27:05.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient: build_initial_monmap 2024-09-16T05:27:05.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient: monmap: 2024-09-16T05:27:05.329 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:05.329 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:05.329 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:05.329 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:05.329 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:05.329 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.329 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:05.329 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:05.329 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:05.329 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 librados: starting msgr at 2024-09-16T05:27:05.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 librados: starting objecter 2024-09-16T05:27:05.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 Processor -- start 2024-09-16T05:27:05.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 -- start start 2024-09-16T05:27:05.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 librados: setting wanted keys 2024-09-16T05:27:05.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 librados: calling monclient init 2024-09-16T05:27:05.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient: init 2024-09-16T05:27:05.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:05.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:27:05.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc50c09d1a0 0x7fc50c13dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient(hunting): picked mon.noname-a con 0x7fc50c09d1a0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:05.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc50c09d580 0x7fc50c0983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient(hunting): picked mon.noname-c con 0x7fc50c09d580 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:05.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc50c0a11a0 0x7fc50c0995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient(hunting): picked mon.noname-b con 0x7fc50c0a11a0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:05.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c140d40 con 0x7fc50c09d1a0 2024-09-16T05:27:05.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c140bc0 con 0x7fc50c0a11a0 2024-09-16T05:27:05.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc50c140ec0 con 0x7fc50c09d580 2024-09-16T05:27:05.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:05.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient(hunting): authenticate will time out at 2109.945599s 2024-09-16T05:27:05.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc50c09d1a0 0x7fc50c13dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc50c09d1a0 0x7fc50c13dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35298/0 (socket says 172.21.15.114:35298) 2024-09-16T05:27:05.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 1 -- 172.21.15.114:0/4034565738 learned_addr learned my addr 172.21.15.114:0/4034565738 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:05.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 10 monclient(hunting): get_auth_request con 0x7fc50c09d1a0 auth_method 0 2024-09-16T05:27:05.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50b7fe640 1 --2- 172.21.15.114:0/4034565738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc50c09d580 0x7fc50c0983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50b7fe640 10 monclient(hunting): get_auth_request con 0x7fc50c09d580 auth_method 0 2024-09-16T05:27:05.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:05.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:05.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:05.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:05.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:05.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:05.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc51090e640 1 --2- 172.21.15.114:0/4034565738 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc50c0a11a0 0x7fc50c0995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 10 monclient(hunting): handle_auth_done global_id 134244 payload 824 2024-09-16T05:27:05.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 1 -- 172.21.15.114:0/4034565738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc50c09d580 msgr2=0x7fc50c0983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 1 --2- 172.21.15.114:0/4034565738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc50c09d580 0x7fc50c0983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 1 -- 172.21.15.114:0/4034565738 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc50c0a11a0 msgr2=0x7fc50c0995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 1 --2- 172.21.15.114:0/4034565738 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc50c0a11a0 0x7fc50c0995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:27:05.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 1 monclient: found mon.noname-a 2024-09-16T05:27:05.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:05.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:05.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 1 -- 172.21.15.114:0/4034565738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc50c097640 con 0x7fc50c09d1a0 2024-09-16T05:27:05.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50b7fe640 1 --2- 172.21.15.114:0/4034565738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc50c09d580 0x7fc50c0983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:27:05.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc51090e640 10 monclient: get_auth_request con 0x7fc50c0a11a0 auth_method 0 2024-09-16T05:27:05.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc51090e640 1 --2- 172.21.15.114:0/4034565738 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc50c0a11a0 0x7fc50c0995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:27:05.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50bfff640 1 --2- 172.21.15.114:0/4034565738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc50c09d1a0 0x7fc50c13dd10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc4f400c780 tx=0x7fc4f400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 10 client.?.objecter ms_handle_connect 0x7fc50c09d1a0 2024-09-16T05:27:05.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:05.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 1 -- 172.21.15.114:0/4034565738 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4018070 con 0x7fc50c09d1a0 2024-09-16T05:27:05.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:05.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 10 monclient: dump: 2024-09-16T05:27:05.334 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.334 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.334 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.334 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.335 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.335 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.335 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.335 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.335 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.335 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 10 monclient: _finish_auth 0 2024-09-16T05:27:05.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:27:05.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:05.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 1 -- 172.21.15.114:0/4034565738 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4f40072f0 con 0x7fc50c09d1a0 2024-09-16T05:27:05.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:05.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 1 -- 172.21.15.114:0/4034565738 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f40047f0 con 0x7fc50c09d1a0 2024-09-16T05:27:05.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 5 monclient: authenticate success, global_id 134244 2024-09-16T05:27:05.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient: _renew_subs 2024-09-16T05:27:05.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:05.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 -- 172.21.15.114:0/4034565738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc50c06e0a0 con 0x7fc50c09d1a0 2024-09-16T05:27:05.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50a7fc640 4 set_mon_vals no callback set 2024-09-16T05:27:05.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:05.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 10 monclient: dump: 2024-09-16T05:27:05.336 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.336 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.336 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.336 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.336 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.337 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.337 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.337 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.337 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.337 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 client.134244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:05.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient: _renew_subs 2024-09-16T05:27:05.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:05.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 -- 172.21.15.114:0/4034565738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc50c097920 con 0x7fc50c09d1a0 2024-09-16T05:27:05.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 librados: init done 2024-09-16T05:27:05.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 client.134244.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:05.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 15 client.134244.objecter _session_command_op_assign -1 1 2024-09-16T05:27:05.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 20 client.134244.objecter _get_session osd=-1 returning homeless 2024-09-16T05:27:05.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 client.134244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:05.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:05.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc512371640 1 -- 172.21.15.114:0/4034565738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc50c09ca90 con 0x7fc50c09d1a0 2024-09-16T05:27:05.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 1 -- 172.21.15.114:0/4034565738 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc4f401b070 con 0x7fc50c09d1a0 2024-09-16T05:27:05.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 1 --2- 172.21.15.114:0/4034565738 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc4f0037810 0x7fc4f0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 1 -- 172.21.15.114:0/4034565738 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7fc4f4047f60 con 0x7fc50c09d1a0 2024-09-16T05:27:05.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 10 client.134244.objecter ms_dispatch 0x7fc50c000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:05.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 3 client.134244.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:05.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 3 client.134244.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:05.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 10 client.134244.objecter checking command 1 2024-09-16T05:27:05.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 1 --2- 172.21.15.114:0/4034565738 >> v2:172.21.15.149:6800/1092505261 conn(0x7fc4f003ecc0 0x7fc4f0041160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 20 client.134244.objecter _get_session s=0x7fc4f003e610 osd=4 3 2024-09-16T05:27:05.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 20 client.134244.objecter put_session s=0x7fc4f003e610 osd=4 3 2024-09-16T05:27:05.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 15 client.134244.objecter _session_command_op_remove -1 1 2024-09-16T05:27:05.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 20 client.134244.objecter _get_session s=0x7fc4f003e610 osd=4 3 2024-09-16T05:27:05.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 20 client.134244.objecter get_session s=0x7fc4f003e610 osd=4 3 2024-09-16T05:27:05.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 15 client.134244.objecter _session_command_op_assign 4 1 2024-09-16T05:27:05.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 20 client.134244.objecter put_session s=0x7fc4f003e610 osd=4 4 2024-09-16T05:27:05.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 10 client.134244.objecter _send_command 1 2024-09-16T05:27:05.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 1 -- 172.21.15.114:0/4034565738 --> v2:172.21.15.149:6800/1092505261 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc4f00417f0 con 0x7fc4f003ecc0 2024-09-16T05:27:05.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 20 client.134244.objecter dump_active .. 0 homeless 2024-09-16T05:27:05.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 1 -- 172.21.15.114:0/4034565738 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=695) ==== 24+0+0 (secure 0 0 0) 0x7fc4f404a630 con 0x7fc50c09d1a0 2024-09-16T05:27:05.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc508ff9640 10 monclient: handle_get_version_reply finishing 1 version 695 2024-09-16T05:27:05.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50b7fe640 1 --2- 172.21.15.114:0/4034565738 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc4f0037810 0x7fc4f0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc51090e640 1 --2- 172.21.15.114:0/4034565738 >> v2:172.21.15.149:6800/1092505261 conn(0x7fc4f003ecc0 0x7fc4f0041160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.315+0000 7fc50b7fe640 10 monclient: get_auth_request con 0x7fc4f0037810 auth_method 0 2024-09-16T05:27:05.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.319+0000 7fc51090e640 10 monclient: get_auth_request con 0x7fc4f003ecc0 auth_method 0 2024-09-16T05:27:05.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.319+0000 7fc51090e640 1 --2- 172.21.15.114:0/4034565738 >> v2:172.21.15.149:6800/1092505261 conn(0x7fc4f003ecc0 0x7fc4f0041160 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.319+0000 7fc508ff9640 10 client.134244.objecter ms_handle_connect 0x7fc4f003ecc0 2024-09-16T05:27:05.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.319+0000 7fc508ff9640 10 client.134244.objecter ms_handle_connect 0x7fc4f0037810 2024-09-16T05:27:05.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.319+0000 7fc50b7fe640 1 --2- 172.21.15.114:0/4034565738 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc4f0037810 0x7fc4f0039cd0 secure :-1 s=READY pgs=3641 cs=0 l=1 rev1=1 crypto rx=0x7fc5040ad000 tx=0x7fc5040a4f90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.319+0000 7fc508ff9640 1 -- 172.21.15.114:0/4034565738 <== osd.4 v2:172.21.15.149:6800/1092505261 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc4f00417f0 con 0x7fc4f003ecc0 2024-09-16T05:27:05.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.319+0000 7fc508ff9640 10 client.134244.objecter ms_dispatch 0x7fc50c000ba0 command_reply(tid 1: 0 ) 2024-09-16T05:27:05.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.319+0000 7fc508ff9640 10 client.134244.objecter _finish_command 1 = system:0 2024-09-16T05:27:05.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.319+0000 7fc508ff9640 20 client.134244.objecter put_session s=0x7fc4f003e610 osd=4 4 2024-09-16T05:27:05.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.319+0000 7fc508ff9640 15 client.134244.objecter _session_command_op_remove 4 1 2024-09-16T05:27:05.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc512371640 10 client.134244.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T05:27:05.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc512371640 15 client.134244.objecter _session_command_op_assign -1 2 2024-09-16T05:27:05.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc512371640 20 client.134244.objecter _get_session s=0x7fc4f003e610 osd=4 3 2024-09-16T05:27:05.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc512371640 20 client.134244.objecter put_session s=0x7fc4f003e610 osd=4 3 2024-09-16T05:27:05.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc512371640 20 client.134244.objecter _get_session s=0x7fc4f003e610 osd=4 3 2024-09-16T05:27:05.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc512371640 15 client.134244.objecter _session_command_op_remove -1 2 2024-09-16T05:27:05.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc512371640 20 client.134244.objecter get_session s=0x7fc4f003e610 osd=4 3 2024-09-16T05:27:05.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc512371640 15 client.134244.objecter _session_command_op_assign 4 2 2024-09-16T05:27:05.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc512371640 20 client.134244.objecter put_session s=0x7fc4f003e610 osd=4 4 2024-09-16T05:27:05.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc512371640 10 client.134244.objecter _send_command 2 2024-09-16T05:27:05.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc512371640 1 -- 172.21.15.114:0/4034565738 --> v2:172.21.15.149:6800/1092505261 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc50c098a90 con 0x7fc4f003ecc0 2024-09-16T05:27:05.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc508ff9640 1 -- 172.21.15.114:0/4034565738 <== osd.4 v2:172.21.15.149:6800/1092505261 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc50c098a90 con 0x7fc4f003ecc0 2024-09-16T05:27:05.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc508ff9640 10 client.134244.objecter ms_dispatch 0x7fc50c000ba0 command_reply(tid 2: 0 ) 2024-09-16T05:27:05.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc508ff9640 10 client.134244.objecter _finish_command 2 = system:0 2024-09-16T05:27:05.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc508ff9640 20 client.134244.objecter put_session s=0x7fc4f003e610 osd=4 4 2024-09-16T05:27:05.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.331+0000 7fc508ff9640 15 client.134244.objecter _session_command_op_remove 4 2 2024-09-16T05:27:05.346 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:27:05.346 INFO:teuthology.orchestra.run.smithi114.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T05:27:05.346 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:27:05.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 10 librados: watch_flush enter 2024-09-16T05:27:05.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 10 librados: watch_flush exit 2024-09-16T05:27:05.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 10 client.134244.objecter close_session for osd.4 2024-09-16T05:27:05.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 1 -- 172.21.15.114:0/4034565738 >> v2:172.21.15.149:6800/1092505261 conn(0x7fc4f003ecc0 msgr2=0x7fc4f0041160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 1 --2- 172.21.15.114:0/4034565738 >> v2:172.21.15.149:6800/1092505261 conn(0x7fc4f003ecc0 0x7fc4f0041160 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 20 client.134244.objecter put_session s=0x7fc4f003e610 osd=4 1 2024-09-16T05:27:05.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 20 client.134244.objecter shutdown clearing up homeless session... 2024-09-16T05:27:05.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 10 client.134244.objecter successfully canceled tick 2024-09-16T05:27:05.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 1 -- 172.21.15.114:0/4034565738 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc4f0037810 msgr2=0x7fc4f0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 1 --2- 172.21.15.114:0/4034565738 >> v2:172.21.15.149:6812/1185531135 conn(0x7fc4f0037810 0x7fc4f0039cd0 secure :-1 s=READY pgs=3641 cs=0 l=1 rev1=1 crypto rx=0x7fc5040ad000 tx=0x7fc5040a4f90 comp rx=0 tx=0).stop 2024-09-16T05:27:05.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 10 monclient: shutdown 2024-09-16T05:27:05.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 1 -- 172.21.15.114:0/4034565738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc50c09d1a0 msgr2=0x7fc50c13dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 1 --2- 172.21.15.114:0/4034565738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc50c09d1a0 0x7fc50c13dd10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc4f400c780 tx=0x7fc4f400cc40 comp rx=0 tx=0).stop 2024-09-16T05:27:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc51090e640 1 -- 172.21.15.114:0/4034565738 reap_dead start 2024-09-16T05:27:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 1 -- 172.21.15.114:0/4034565738 shutdown_connections 2024-09-16T05:27:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 1 -- 172.21.15.114:0/4034565738 >> 172.21.15.114:0/4034565738 conn(0x7fc50c0a2490 msgr2=0x7fc50c0716f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 1 -- 172.21.15.114:0/4034565738 shutdown_connections 2024-09-16T05:27:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 1 -- 172.21.15.114:0/4034565738 wait complete. 2024-09-16T05:27:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.335+0000 7fc512371640 1 librados: shutdown 2024-09-16T05:27:05.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T05:27:05.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 10 monclient: get_monmap_and_config 2024-09-16T05:27:05.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 10 monclient: build_initial_monmap 2024-09-16T05:27:05.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 10 monclient: monmap: 2024-09-16T05:27:05.453 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:05.453 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:05.458844+0000 2024-09-16T05:27:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:05.458844+0000 2024-09-16T05:27:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:05.454 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 1 Processor -- start 2024-09-16T05:27:05.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 1 -- start start 2024-09-16T05:27:05.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 10 monclient: init 2024-09-16T05:27:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-16T05:27:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe43409b6c0 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4340a15d0 addr v1:172.21.15.149:6789/0 2024-09-16T05:27:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4340a19e0 0x7fe43413dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 10 monclient(hunting): picked mon.noname-b con 0x7fe4340a19e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe43409d4e0 con 0x7fe43409b6c0 2024-09-16T05:27:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe43409f430 con 0x7fe4340a15d0 2024-09-16T05:27:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe43409e390 con 0x7fe4340a19e0 2024-09-16T05:27:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.455+0000 7fe43b641640 10 monclient(hunting): authenticate will time out at 2110.087993s 2024-09-16T05:27:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe4393dd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe43409b6c0 0x7fe4340a0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37684/0 (socket says 172.21.15.114:37684) 2024-09-16T05:27:05.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe4393dd640 1 -- 172.21.15.114:0/3395637139 learned_addr learned my addr 172.21.15.114:0/3395637139 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:27:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 1 --2- 172.21.15.114:0/3395637139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4340a19e0 0x7fe43413dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 10 monclient(hunting): get_auth_request con 0x7fe4340a19e0 auth_method 0 2024-09-16T05:27:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 1 -- 172.21.15.114:0/3395637139 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2288293688 0 0) 0x7fe43409d4e0 con 0x7fe43409b6c0 2024-09-16T05:27:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 10 monclient(hunting): _negotiate 2024-09-16T05:27:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 10 monclient(hunting): my global_id is 134247 2024-09-16T05:27:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 1 -- 172.21.15.114:0/3395637139 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe418003530 con 0x7fe43409b6c0 2024-09-16T05:27:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 10 monclient(hunting): handle_auth_done global_id 104459 payload 274 2024-09-16T05:27:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 1 -- 172.21.15.114:0/3395637139 >> v1:172.21.15.149:6789/0 conn(0x7fe4340a15d0 legacy=0x7fe43413a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 1 -- 172.21.15.114:0/3395637139 >> v1:172.21.15.114:6789/0 conn(0x7fe43409b6c0 legacy=0x7fe4340a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 10 monclient: _finish_hunting 0 2024-09-16T05:27:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 1 monclient: found mon.noname-b 2024-09-16T05:27:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:27:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 1 -- 172.21.15.114:0/3395637139 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe43409c2f0 con 0x7fe4340a19e0 2024-09-16T05:27:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe439bde640 1 --2- 172.21.15.114:0/3395637139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4340a19e0 0x7fe43413dd80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe424009dc0 tx=0x7fe42401c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47c5553bb9a965ea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 1 -- 172.21.15.114:0/3395637139 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4240b2070 con 0x7fe4340a19e0 2024-09-16T05:27:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:27:05.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 10 monclient: dump: 2024-09-16T05:27:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.462 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 10 monclient: _finish_auth 0 2024-09-16T05:27:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 10 monclient: _check_auth_tickets 2024-09-16T05:27:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 1 -- 172.21.15.114:0/3395637139 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4240a5710 con 0x7fe4340a19e0 2024-09-16T05:27:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe43b641640 5 monclient: authenticate success, global_id 104459 2024-09-16T05:27:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe43b641640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe43b641640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe43b641640 4 set_mon_vals no callback set 2024-09-16T05:27:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 1 -- 172.21.15.114:0/3395637139 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4240a5ac0 con 0x7fe4340a19e0 2024-09-16T05:27:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe433fff640 10 monclient: dump: 2024-09-16T05:27:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.463 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.464 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.464 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.464 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.464 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe43b641640 10 monclient: shutdown 2024-09-16T05:27:05.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe43b641640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:05.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe43b641640 1 -- 172.21.15.114:0/3395637139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4340a19e0 msgr2=0x7fe43413dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe43b641640 1 --2- 172.21.15.114:0/3395637139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4340a19e0 0x7fe43413dd80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe424009dc0 tx=0x7fe42401c850 comp rx=0 tx=0).stop 2024-09-16T05:27:05.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe43b641640 1 -- 172.21.15.114:0/3395637139 shutdown_connections 2024-09-16T05:27:05.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe43b641640 1 --2- 172.21.15.114:0/3395637139 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4340a19e0 0x7fe43413dd80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe43b641640 1 -- 172.21.15.114:0/3395637139 >> 172.21.15.114:0/3395637139 conn(0x7fe4340a1fb0 msgr2=0x7fe4340a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:05.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe43b641640 1 -- 172.21.15.114:0/3395637139 shutdown_connections 2024-09-16T05:27:05.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe43b641640 1 -- 172.21.15.114:0/3395637139 wait complete. 2024-09-16T05:27:05.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe43b641640 10 monclient: build_initial_monmap 2024-09-16T05:27:05.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.459+0000 7fe43b641640 10 monclient: monmap: 2024-09-16T05:27:05.465 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:05.465 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:05.465 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:05.465 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:05.465 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:05.465 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.465 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:05.465 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:05.465 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:05.465 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 librados: starting msgr at 2024-09-16T05:27:05.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 librados: starting objecter 2024-09-16T05:27:05.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 Processor -- start 2024-09-16T05:27:05.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 -- start start 2024-09-16T05:27:05.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 librados: setting wanted keys 2024-09-16T05:27:05.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 librados: calling monclient init 2024-09-16T05:27:05.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient: init 2024-09-16T05:27:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:27:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe43409b6c0 0x7fe434134790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient(hunting): picked mon.noname-c con 0x7fe43409b6c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe4340a15d0 0x7fe434134cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient(hunting): picked mon.noname-b con 0x7fe4340a15d0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4340a19e0 0x7fe4340994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient(hunting): picked mon.noname-a con 0x7fe4340a19e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe434140c60 con 0x7fe4340a19e0 2024-09-16T05:27:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fe434140ae0 con 0x7fe4340a15d0 2024-09-16T05:27:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe434140de0 con 0x7fe43409b6c0 2024-09-16T05:27:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient(hunting): authenticate will time out at 2110.092736s 2024-09-16T05:27:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4340a19e0 0x7fe4340994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe4393dd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe43409b6c0 0x7fe434134790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4340a19e0 0x7fe4340994b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35310/0 (socket says 172.21.15.114:35310) 2024-09-16T05:27:05.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 1 -- 172.21.15.114:0/295141553 learned_addr learned my addr 172.21.15.114:0/295141553 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:05.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 10 monclient(hunting): get_auth_request con 0x7fe4340a19e0 auth_method 0 2024-09-16T05:27:05.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe4393dd640 10 monclient(hunting): get_auth_request con 0x7fe43409b6c0 auth_method 0 2024-09-16T05:27:05.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe4393dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe4393dd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe4393dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:05.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:05.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:05.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe4393dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:05.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe4393dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:05.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe4393dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:05.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 10 monclient(hunting): handle_auth_done global_id 134250 payload 824 2024-09-16T05:27:05.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe438bdc640 1 --2- 172.21.15.114:0/295141553 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe4340a15d0 0x7fe434134cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 1 -- 172.21.15.114:0/295141553 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe43409b6c0 msgr2=0x7fe434134790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 1 --2- 172.21.15.114:0/295141553 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe43409b6c0 0x7fe434134790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 1 -- 172.21.15.114:0/295141553 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe4340a15d0 msgr2=0x7fe434134cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 1 --2- 172.21.15.114:0/295141553 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe4340a15d0 0x7fe434134cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 10 monclient: _finish_hunting 0 2024-09-16T05:27:05.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 1 monclient: found mon.noname-a 2024-09-16T05:27:05.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:05.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:05.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 1 -- 172.21.15.114:0/295141553 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe43412c1f0 con 0x7fe4340a19e0 2024-09-16T05:27:05.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe438bdc640 10 monclient: get_auth_request con 0x7fe4340a15d0 auth_method 0 2024-09-16T05:27:05.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe438bdc640 1 --2- 172.21.15.114:0/295141553 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe4340a15d0 0x7fe434134cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:27:05.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe439bde640 1 --2- 172.21.15.114:0/295141553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4340a19e0 0x7fe4340994b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe4240b0040 tx=0x7fe4240195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 10 client.?.objecter ms_handle_connect 0x7fe4340a19e0 2024-09-16T05:27:05.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:05.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 1 -- 172.21.15.114:0/295141553 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4240bd020 con 0x7fe4340a19e0 2024-09-16T05:27:05.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:05.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 10 monclient: dump: 2024-09-16T05:27:05.474 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.474 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.474 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.474 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.474 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.475 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.475 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.475 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.475 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.475 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 10 monclient: _finish_auth 0 2024-09-16T05:27:05.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:27:05.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:05.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 5 monclient: authenticate success, global_id 134250 2024-09-16T05:27:05.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient: _renew_subs 2024-09-16T05:27:05.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:05.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 -- 172.21.15.114:0/295141553 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe434135620 con 0x7fe4340a19e0 2024-09-16T05:27:05.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 1 -- 172.21.15.114:0/295141553 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4240b3040 con 0x7fe4340a19e0 2024-09-16T05:27:05.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:05.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 1 -- 172.21.15.114:0/295141553 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4240b2050 con 0x7fe4340a19e0 2024-09-16T05:27:05.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:05.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 10 monclient: dump: 2024-09-16T05:27:05.476 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.476 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.476 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.476 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.476 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.476 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.476 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.476 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.476 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.477 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe4337fe640 4 set_mon_vals no callback set 2024-09-16T05:27:05.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 client.134250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:05.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient: _renew_subs 2024-09-16T05:27:05.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:05.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 -- 172.21.15.114:0/295141553 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe43409b050 con 0x7fe4340a19e0 2024-09-16T05:27:05.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 librados: init done 2024-09-16T05:27:05.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 1 -- 172.21.15.114:0/295141553 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe424003ba0 con 0x7fe4340a19e0 2024-09-16T05:27:05.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:05.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:05.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:05.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe43b641640 1 -- 172.21.15.114:0/295141553 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4340008e0 con 0x7fe4340a19e0 2024-09-16T05:27:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 1 --2- 172.21.15.114:0/295141553 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe41c037840 0x7fe41c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 1 -- 172.21.15.114:0/295141553 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7fe4240e3550 con 0x7fe4340a19e0 2024-09-16T05:27:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 10 client.134250.objecter ms_dispatch 0x7fe434000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 3 client.134250.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 3 client.134250.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 20 client.134250.objecter dump_active .. 0 homeless 2024-09-16T05:27:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe4393dd640 1 --2- 172.21.15.114:0/295141553 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe41c037840 0x7fe41c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe4393dd640 10 monclient: get_auth_request con 0x7fe41c037840 auth_method 0 2024-09-16T05:27:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe431ffb640 10 client.134250.objecter ms_handle_connect 0x7fe41c037840 2024-09-16T05:27:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.463+0000 7fe4393dd640 1 --2- 172.21.15.114:0/295141553 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe41c037840 0x7fe41c039d00 secure :-1 s=READY pgs=3642 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0011f0 tx=0x7fe42c007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.467+0000 7fe431ffb640 1 -- 172.21.15.114:0/295141553 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fe4240c7020 con 0x7fe4340a19e0 2024-09-16T05:27:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.467+0000 7fe431ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.467+0000 7fe431ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:27:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.627+0000 7fe43b641640 1 -- 172.21.15.114:0/295141553 --> v2:172.21.15.149:6812/1185531135 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fe4340686f0 con 0x7fe41c037840 2024-09-16T05:27:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.627+0000 7fe431ffb640 1 -- 172.21.15.114:0/295141553 <== mgr.4108 v2:172.21.15.149:6812/1185531135 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe4340686f0 con 0x7fe41c037840 2024-09-16T05:27:05.627 INFO:teuthology.orchestra.run.smithi114.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T05:27:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 10 librados: watch_flush enter 2024-09-16T05:27:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 10 librados: watch_flush exit 2024-09-16T05:27:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 20 client.134250.objecter shutdown clearing up homeless session... 2024-09-16T05:27:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 10 client.134250.objecter successfully canceled tick 2024-09-16T05:27:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 1 -- 172.21.15.114:0/295141553 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe41c037840 msgr2=0x7fe41c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 1 --2- 172.21.15.114:0/295141553 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe41c037840 0x7fe41c039d00 secure :-1 s=READY pgs=3642 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0011f0 tx=0x7fe42c007d40 comp rx=0 tx=0).stop 2024-09-16T05:27:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 10 monclient: shutdown 2024-09-16T05:27:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 1 -- 172.21.15.114:0/295141553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4340a19e0 msgr2=0x7fe4340994b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 1 --2- 172.21.15.114:0/295141553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4340a19e0 0x7fe4340994b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe4240b0040 tx=0x7fe4240195f0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 1 -- 172.21.15.114:0/295141553 shutdown_connections 2024-09-16T05:27:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 1 --2- 172.21.15.114:0/295141553 >> v2:172.21.15.149:6812/1185531135 conn(0x7fe41c037840 0x7fe41c039d00 unknown :-1 s=CLOSED pgs=3642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 1 --2- 172.21.15.114:0/295141553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4340a19e0 0x7fe4340994b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 1 --2- 172.21.15.114:0/295141553 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe4340a15d0 0x7fe434134cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 1 --2- 172.21.15.114:0/295141553 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe43409b6c0 0x7fe434134790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 1 -- 172.21.15.114:0/295141553 >> 172.21.15.114:0/295141553 conn(0x7fe4340a1fb0 msgr2=0x7fe434137860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:05.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 1 -- 172.21.15.114:0/295141553 shutdown_connections 2024-09-16T05:27:05.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 1 -- 172.21.15.114:0/295141553 wait complete. 2024-09-16T05:27:05.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.631+0000 7fe43b641640 1 librados: shutdown 2024-09-16T05:27:05.644 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T05:27:05.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T05:27:05.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 10 monclient: get_monmap_and_config 2024-09-16T05:27:05.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 10 monclient: build_initial_monmap 2024-09-16T05:27:05.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 10 monclient: monmap: 2024-09-16T05:27:05.768 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:05.768 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.768 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:05.773077+0000 2024-09-16T05:27:05.768 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:05.773077+0000 2024-09-16T05:27:05.768 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:05.768 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.768 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:05.768 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:05.768 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:05.768 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:05.768 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:05.768 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 1 Processor -- start 2024-09-16T05:27:05.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 1 -- start start 2024-09-16T05:27:05.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 10 monclient: init 2024-09-16T05:27:05.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:05.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-16T05:27:05.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e8c068250 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:05.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e8c09f390 0x7f7e8c134be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 10 monclient(hunting): picked mon.noname-b con 0x7f7e8c09f390 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:05.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f7e8c139710 0x7f7e8c13bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 10 monclient(hunting): picked mon.noname-c con 0x7f7e8c139710 addr v2:172.21.15.149:3300/0 2024-09-16T05:27:05.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e8c0a32a0 con 0x7f7e8c068250 2024-09-16T05:27:05.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f7e8c09fef0 con 0x7f7e8c139710 2024-09-16T05:27:05.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7e8c068630 con 0x7f7e8c09f390 2024-09-16T05:27:05.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:05.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e92f2e640 10 monclient(hunting): authenticate will time out at 2110.403300s 2024-09-16T05:27:05.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e90cca640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7e8c068250 0x7f7e8c1344d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37692/0 (socket says 172.21.15.114:37692) 2024-09-16T05:27:05.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 1 --2- 172.21.15.114:0/1920981479 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e8c09f390 0x7f7e8c134be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e90cca640 1 -- 172.21.15.114:0/1920981479 learned_addr learned my addr 172.21.15.114:0/1920981479 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:27:05.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 10 monclient(hunting): get_auth_request con 0x7f7e8c09f390 auth_method 0 2024-09-16T05:27:05.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:05.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:05.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:05.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 10 monclient(hunting): handle_auth_done global_id 104465 payload 274 2024-09-16T05:27:05.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 1 -- 172.21.15.114:0/1920981479 >> v2:172.21.15.149:3300/0 conn(0x7f7e8c139710 msgr2=0x7f7e8c13bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:05.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 1 --2- 172.21.15.114:0/1920981479 >> v2:172.21.15.149:3300/0 conn(0x7f7e8c139710 0x7f7e8c13bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 1 -- 172.21.15.114:0/1920981479 >> v1:172.21.15.114:6789/0 conn(0x7f7e8c068250 legacy=0x7f7e8c1344d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:27:05.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 1 monclient: found mon.noname-b 2024-09-16T05:27:05.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:05.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:27:05.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8b7fe640 1 -- 172.21.15.114:0/1920981479 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3823642015 0 0) 0x7f7e8c0a32a0 con 0x7f7e8c068250 2024-09-16T05:27:05.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8bfff640 1 -- 172.21.15.114:0/1920981479 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e8c0a1c40 con 0x7f7e8c09f390 2024-09-16T05:27:05.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.771+0000 7f7e8b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T05:27:05.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e8bfff640 1 --2- 172.21.15.114:0/1920981479 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e8c09f390 0x7f7e8c134be0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7e780097c0 tx=0x7f7e7801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23d24a8376eb7c8e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e8b7fe640 1 -- 172.21.15.114:0/1920981479 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e780b0070 con 0x7f7e8c09f390 2024-09-16T05:27:05.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e8b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:27:05.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e8b7fe640 10 monclient: dump: 2024-09-16T05:27:05.781 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.781 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.781 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.782 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.782 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.782 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.782 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.782 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.782 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.782 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e8b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:27:05.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e8b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:27:05.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:05.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 5 monclient: authenticate success, global_id 104465 2024-09-16T05:27:05.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e8b7fe640 1 -- 172.21.15.114:0/1920981479 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e780ab600 con 0x7f7e8c09f390 2024-09-16T05:27:05.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:05.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e8b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:05.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:05.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e8b7fe640 1 -- 172.21.15.114:0/1920981479 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e780ab9d0 con 0x7f7e8c09f390 2024-09-16T05:27:05.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 4 set_mon_vals no callback set 2024-09-16T05:27:05.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e8b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:05.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e8b7fe640 10 monclient: dump: 2024-09-16T05:27:05.783 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.783 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.783 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.783 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.783 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.783 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.783 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.784 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.784 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.784 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 10 monclient: shutdown 2024-09-16T05:27:05.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:05.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1920981479 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e8c09f390 msgr2=0x7f7e8c134be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 1 --2- 172.21.15.114:0/1920981479 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e8c09f390 0x7f7e8c134be0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7e780097c0 tx=0x7f7e7801c360 comp rx=0 tx=0).stop 2024-09-16T05:27:05.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1920981479 shutdown_connections 2024-09-16T05:27:05.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 1 --2- 172.21.15.114:0/1920981479 >> v2:172.21.15.149:3300/0 conn(0x7f7e8c139710 0x7f7e8c13bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 1 --2- 172.21.15.114:0/1920981479 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e8c09f390 0x7f7e8c134be0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1920981479 >> 172.21.15.114:0/1920981479 conn(0x7f7e8c0a63e0 msgr2=0x7f7e8c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:05.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1920981479 shutdown_connections 2024-09-16T05:27:05.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1920981479 wait complete. 2024-09-16T05:27:05.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 10 monclient: build_initial_monmap 2024-09-16T05:27:05.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 10 monclient: monmap: 2024-09-16T05:27:05.785 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:05.785 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:05.785 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:05.785 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:05.785 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:05.785 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.785 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:05.785 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:05.785 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:05.785 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 1 librados: starting msgr at 2024-09-16T05:27:05.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 1 librados: starting objecter 2024-09-16T05:27:05.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 1 Processor -- start 2024-09-16T05:27:05.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 1 -- start start 2024-09-16T05:27:05.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 1 librados: setting wanted keys 2024-09-16T05:27:05.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 1 librados: calling monclient init 2024-09-16T05:27:05.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 10 monclient: init 2024-09-16T05:27:05.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:05.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:27:05.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.775+0000 7f7e92f2e640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7e8c068250 0x7f7e8c1cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 monclient(hunting): picked mon.noname-b con 0x7f7e8c068250 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:05.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e8c09f390 0x7f7e8c133be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 monclient(hunting): picked mon.noname-c con 0x7f7e8c09f390 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:05.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e8c139710 0x7f7e8c1cf190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 monclient(hunting): picked mon.noname-a con 0x7f7e8c139710 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:05.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8c13e860 con 0x7f7e8c139710 2024-09-16T05:27:05.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8c135330 con 0x7f7e8c068250 2024-09-16T05:27:05.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7e8c13e9e0 con 0x7f7e8c09f390 2024-09-16T05:27:05.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:05.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 monclient(hunting): authenticate will time out at 2110.408054s 2024-09-16T05:27:05.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e8c09f390 0x7f7e8c133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e8c09f390 0x7f7e8c133be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51902/0 (socket says 172.21.15.114:51902) 2024-09-16T05:27:05.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 1 -- 172.21.15.114:0/1224148482 learned_addr learned my addr 172.21.15.114:0/1224148482 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:05.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 10 monclient(hunting): get_auth_request con 0x7f7e8c09f390 auth_method 0 2024-09-16T05:27:05.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e914cb640 1 --2- 172.21.15.114:0/1224148482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e8c139710 0x7f7e8c1cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e914cb640 10 monclient(hunting): get_auth_request con 0x7f7e8c139710 auth_method 0 2024-09-16T05:27:05.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e914cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e914cb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e914cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:05.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:05.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:05.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e914cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:05.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e914cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:05.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e914cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:05.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e90cca640 1 --2- 172.21.15.114:0/1224148482 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7e8c068250 0x7f7e8c1cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 10 monclient(hunting): handle_auth_done global_id 104468 payload 824 2024-09-16T05:27:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 1 -- 172.21.15.114:0/1224148482 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7e8c068250 msgr2=0x7f7e8c1cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 1 --2- 172.21.15.114:0/1224148482 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7e8c068250 0x7f7e8c1cbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 1 -- 172.21.15.114:0/1224148482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e8c139710 msgr2=0x7f7e8c1cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 1 --2- 172.21.15.114:0/1224148482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e8c139710 0x7f7e8c1cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:27:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 1 monclient: found mon.noname-c 2024-09-16T05:27:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:27:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 1 -- 172.21.15.114:0/1224148482 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e8c1215a0 con 0x7f7e8c09f390 2024-09-16T05:27:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e90cca640 10 monclient: get_auth_request con 0x7f7e8c068250 auth_method 0 2024-09-16T05:27:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e90cca640 1 --2- 172.21.15.114:0/1224148482 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7e8c068250 0x7f7e8c1cbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T05:27:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8bfff640 1 --2- 172.21.15.114:0/1224148482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e8c09f390 0x7f7e8c133be0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7e7801aa10 tx=0x7f7e7801aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 client.?.objecter ms_handle_connect 0x7f7e8c09f390 2024-09-16T05:27:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:05.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 1 -- 172.21.15.114:0/1224148482 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e780b0050 con 0x7f7e8c09f390 2024-09-16T05:27:05.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:27:05.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 monclient: dump: 2024-09-16T05:27:05.790 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.790 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.790 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.790 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.790 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.790 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.790 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.790 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.790 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.790 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 monclient: _finish_auth 0 2024-09-16T05:27:05.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 monclient: _check_auth_tickets 2024-09-16T05:27:05.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:05.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 1 -- 172.21.15.114:0/1224148482 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e780bc070 con 0x7f7e8c09f390 2024-09-16T05:27:05.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 5 monclient: authenticate success, global_id 104468 2024-09-16T05:27:05.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 monclient: _renew_subs 2024-09-16T05:27:05.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:05.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1224148482 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e8c09d8f0 con 0x7f7e8c09f390 2024-09-16T05:27:05.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:05.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 1 -- 172.21.15.114:0/1224148482 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e780b5a70 con 0x7f7e8c09f390 2024-09-16T05:27:05.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:05.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 monclient: dump: 2024-09-16T05:27:05.791 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.792 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.792 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.792 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.792 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.792 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.792 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.792 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.792 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.792 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e8affd640 4 set_mon_vals no callback set 2024-09-16T05:27:05.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 client.104468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:05.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 monclient: _renew_subs 2024-09-16T05:27:05.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:05.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1224148482 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e8c0a5700 con 0x7f7e8c09f390 2024-09-16T05:27:05.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 1 librados: init done 2024-09-16T05:27:05.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 client.104468.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:05.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 15 client.104468.objecter _session_command_op_assign -1 1 2024-09-16T05:27:05.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 20 client.104468.objecter _get_session osd=-1 returning homeless 2024-09-16T05:27:05.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 client.104468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:05.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:05.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1224148482 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7e8c11f2b0 con 0x7f7e8c09f390 2024-09-16T05:27:05.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 1 -- 172.21.15.114:0/1224148482 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7e780b5c10 con 0x7f7e8c09f390 2024-09-16T05:27:05.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 1 --2- 172.21.15.114:0/1224148482 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7e70037810 0x7f7e70039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 1 -- 172.21.15.114:0/1224148482 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7f7e780e3500 con 0x7f7e8c09f390 2024-09-16T05:27:05.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 client.104468.objecter ms_dispatch 0x7f7e8c000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:05.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 3 client.104468.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:05.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 3 client.104468.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:05.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 client.104468.objecter checking command 1 2024-09-16T05:27:05.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 1 --2- 172.21.15.114:0/1224148482 >> v2:172.21.15.149:6804/1244703827 conn(0x7f7e7003ecb0 0x7f7e70041150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 20 client.104468.objecter _get_session s=0x7f7e7003e600 osd=5 3 2024-09-16T05:27:05.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 20 client.104468.objecter put_session s=0x7f7e7003e600 osd=5 3 2024-09-16T05:27:05.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 15 client.104468.objecter _session_command_op_remove -1 1 2024-09-16T05:27:05.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 20 client.104468.objecter _get_session s=0x7f7e7003e600 osd=5 3 2024-09-16T05:27:05.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 20 client.104468.objecter get_session s=0x7f7e7003e600 osd=5 3 2024-09-16T05:27:05.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 15 client.104468.objecter _session_command_op_assign 5 1 2024-09-16T05:27:05.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 20 client.104468.objecter put_session s=0x7f7e7003e600 osd=5 4 2024-09-16T05:27:05.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 client.104468.objecter _send_command 1 2024-09-16T05:27:05.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 1 -- 172.21.15.114:0/1224148482 --> v2:172.21.15.149:6804/1244703827 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e700417e0 con 0x7f7e7003ecb0 2024-09-16T05:27:05.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 20 client.104468.objecter dump_active .. 0 homeless 2024-09-16T05:27:05.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 1 -- 172.21.15.114:0/1224148482 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_get_version_reply(handle=1 version=695) ==== 24+0+0 (secure 0 0 0) 0x7f7e780ea050 con 0x7f7e8c09f390 2024-09-16T05:27:05.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 monclient: handle_get_version_reply finishing 1 version 695 2024-09-16T05:27:05.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e90cca640 1 --2- 172.21.15.114:0/1224148482 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7e70037810 0x7f7e70039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e90cca640 10 monclient: get_auth_request con 0x7f7e70037810 auth_method 0 2024-09-16T05:27:05.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e914cb640 1 --2- 172.21.15.114:0/1224148482 >> v2:172.21.15.149:6804/1244703827 conn(0x7f7e7003ecb0 0x7f7e70041150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e914cb640 10 monclient: get_auth_request con 0x7f7e7003ecb0 auth_method 0 2024-09-16T05:27:05.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e914cb640 1 --2- 172.21.15.114:0/1224148482 >> v2:172.21.15.149:6804/1244703827 conn(0x7f7e7003ecb0 0x7f7e70041150 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 client.104468.objecter ms_handle_connect 0x7f7e7003ecb0 2024-09-16T05:27:05.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e897fa640 10 client.104468.objecter ms_handle_connect 0x7f7e70037810 2024-09-16T05:27:05.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.779+0000 7f7e90cca640 1 --2- 172.21.15.114:0/1224148482 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7e70037810 0x7f7e70039cd0 secure :-1 s=READY pgs=3643 cs=0 l=1 rev1=1 crypto rx=0x7f7e80000f70 tx=0x7f7e80009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.783+0000 7f7e897fa640 1 -- 172.21.15.114:0/1224148482 <== osd.5 v2:172.21.15.149:6804/1244703827 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e700417e0 con 0x7f7e7003ecb0 2024-09-16T05:27:05.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.783+0000 7f7e897fa640 10 client.104468.objecter ms_dispatch 0x7f7e8c000ba0 command_reply(tid 1: 0 ) 2024-09-16T05:27:05.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.783+0000 7f7e897fa640 10 client.104468.objecter _finish_command 1 = system:0 2024-09-16T05:27:05.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.783+0000 7f7e897fa640 20 client.104468.objecter put_session s=0x7f7e7003e600 osd=5 4 2024-09-16T05:27:05.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.783+0000 7f7e897fa640 15 client.104468.objecter _session_command_op_remove 5 1 2024-09-16T05:27:05.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 10 client.104468.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T05:27:05.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 15 client.104468.objecter _session_command_op_assign -1 2 2024-09-16T05:27:05.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 20 client.104468.objecter _get_session s=0x7f7e7003e600 osd=5 3 2024-09-16T05:27:05.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 20 client.104468.objecter put_session s=0x7f7e7003e600 osd=5 3 2024-09-16T05:27:05.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 20 client.104468.objecter _get_session s=0x7f7e7003e600 osd=5 3 2024-09-16T05:27:05.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 15 client.104468.objecter _session_command_op_remove -1 2 2024-09-16T05:27:05.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 20 client.104468.objecter get_session s=0x7f7e7003e600 osd=5 3 2024-09-16T05:27:05.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 15 client.104468.objecter _session_command_op_assign 5 2 2024-09-16T05:27:05.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 20 client.104468.objecter put_session s=0x7f7e7003e600 osd=5 4 2024-09-16T05:27:05.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 10 client.104468.objecter _send_command 2 2024-09-16T05:27:05.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1224148482 --> v2:172.21.15.149:6804/1244703827 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7e8c121220 con 0x7f7e7003ecb0 2024-09-16T05:27:05.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e897fa640 1 -- 172.21.15.114:0/1224148482 <== osd.5 v2:172.21.15.149:6804/1244703827 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7e8c121220 con 0x7f7e7003ecb0 2024-09-16T05:27:05.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e897fa640 10 client.104468.objecter ms_dispatch 0x7f7e8c000ba0 command_reply(tid 2: 0 ) 2024-09-16T05:27:05.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e897fa640 10 client.104468.objecter _finish_command 2 = system:0 2024-09-16T05:27:05.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e897fa640 20 client.104468.objecter put_session s=0x7f7e7003e600 osd=5 4 2024-09-16T05:27:05.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e897fa640 15 client.104468.objecter _session_command_op_remove 5 2 2024-09-16T05:27:05.798 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T05:27:05.798 INFO:teuthology.orchestra.run.smithi114.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T05:27:05.798 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T05:27:05.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 10 librados: watch_flush enter 2024-09-16T05:27:05.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 10 librados: watch_flush exit 2024-09-16T05:27:05.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 10 client.104468.objecter close_session for osd.5 2024-09-16T05:27:05.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1224148482 >> v2:172.21.15.149:6804/1244703827 conn(0x7f7e7003ecb0 msgr2=0x7f7e70041150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 1 --2- 172.21.15.114:0/1224148482 >> v2:172.21.15.149:6804/1244703827 conn(0x7f7e7003ecb0 0x7f7e70041150 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 20 client.104468.objecter put_session s=0x7f7e7003e600 osd=5 1 2024-09-16T05:27:05.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 20 client.104468.objecter shutdown clearing up homeless session... 2024-09-16T05:27:05.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 10 client.104468.objecter successfully canceled tick 2024-09-16T05:27:05.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1224148482 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7e70037810 msgr2=0x7f7e70039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 1 --2- 172.21.15.114:0/1224148482 >> v2:172.21.15.149:6812/1185531135 conn(0x7f7e70037810 0x7f7e70039cd0 secure :-1 s=READY pgs=3643 cs=0 l=1 rev1=1 crypto rx=0x7f7e80000f70 tx=0x7f7e80009040 comp rx=0 tx=0).stop 2024-09-16T05:27:05.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 10 monclient: shutdown 2024-09-16T05:27:05.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:05.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1224148482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e8c09f390 msgr2=0x7f7e8c133be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 1 --2- 172.21.15.114:0/1224148482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e8c09f390 0x7f7e8c133be0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7e7801aa10 tx=0x7f7e7801aba0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e914cb640 1 -- 172.21.15.114:0/1224148482 reap_dead start 2024-09-16T05:27:05.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1224148482 shutdown_connections 2024-09-16T05:27:05.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1224148482 >> 172.21.15.114:0/1224148482 conn(0x7f7e8c0a63e0 msgr2=0x7f7e8c1346e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:05.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1224148482 shutdown_connections 2024-09-16T05:27:05.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 1 -- 172.21.15.114:0/1224148482 wait complete. 2024-09-16T05:27:05.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.795+0000 7f7e92f2e640 1 librados: shutdown 2024-09-16T05:27:05.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T05:27:05.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.919+0000 7fb1ef074640 10 monclient: get_monmap_and_config 2024-09-16T05:27:05.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.919+0000 7fb1ef074640 10 monclient: build_initial_monmap 2024-09-16T05:27:05.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.919+0000 7fb1ef074640 10 monclient: monmap: 2024-09-16T05:27:05.918 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:05.918 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.918 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:05.923614+0000 2024-09-16T05:27:05.918 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:05.923614+0000 2024-09-16T05:27:05.918 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:05.918 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.919 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:05.919 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:05.919 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:05.919 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:05.919 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:05.919 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 1 Processor -- start 2024-09-16T05:27:05.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 1 -- start start 2024-09-16T05:27:05.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 10 monclient: init 2024-09-16T05:27:05.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:05.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T05:27:05.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1e809b6c0 addr v1:172.21.15.149:6789/0 2024-09-16T05:27:05.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1e80a15d0 0x7fb1e813a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e80a15d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:05.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb1e80a19e0 0x7fb1e813cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 10 monclient(hunting): picked mon.noname-c con 0x7fb1e80a19e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:27:05.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e809d4e0 con 0x7fb1e809b6c0 2024-09-16T05:27:05.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fb1e809e390 con 0x7fb1e80a19e0 2024-09-16T05:27:05.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb1e809ece0 con 0x7fb1e80a15d0 2024-09-16T05:27:05.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:05.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 10 monclient(hunting): authenticate will time out at 2110.552762s 2024-09-16T05:27:05.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1e80a15d0 0x7fb1e813a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1e80a15d0 0x7fb1e813a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51908/0 (socket says 172.21.15.114:51908) 2024-09-16T05:27:05.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 1 -- 172.21.15.114:0/423451064 learned_addr learned my addr 172.21.15.114:0/423451064 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:05.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 10 monclient(hunting): get_auth_request con 0x7fb1e80a15d0 auth_method 0 2024-09-16T05:27:05.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:05.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:05.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:05.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ed611640 1 --2- 172.21.15.114:0/423451064 >> v2:172.21.15.149:3300/0 conn(0x7fb1e80a19e0 0x7fb1e813cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ed611640 10 monclient(hunting): get_auth_request con 0x7fb1e80a19e0 auth_method 0 2024-09-16T05:27:05.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ed611640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ed611640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ed611640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 10 monclient(hunting): handle_auth_done global_id 104471 payload 274 2024-09-16T05:27:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 1 -- 172.21.15.114:0/423451064 >> v2:172.21.15.149:3300/0 conn(0x7fb1e80a19e0 msgr2=0x7fb1e813cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 1 --2- 172.21.15.114:0/423451064 >> v2:172.21.15.149:3300/0 conn(0x7fb1e80a19e0 0x7fb1e813cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 1 -- 172.21.15.114:0/423451064 >> v1:172.21.15.149:6789/0 conn(0x7fb1e809b6c0 legacy=0x7fb1e80a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:27:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 1 monclient: found mon.noname-b 2024-09-16T05:27:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.114:3301/0 2024-09-16T05:27:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 1 -- 172.21.15.114:0/423451064 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e809f430 con 0x7fb1e80a15d0 2024-09-16T05:27:05.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e7fff640 1 --2- 172.21.15.114:0/423451064 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1e80a15d0 0x7fb1e813a650 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0006fd0 tx=0x7fb1d001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=295cec8681e621d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e77fe640 1 -- 172.21.15.114:0/423451064 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d00b0070 con 0x7fb1e80a15d0 2024-09-16T05:27:05.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:27:05.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e77fe640 10 monclient: dump: 2024-09-16T05:27:05.931 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.931 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.931 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.931 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.931 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.931 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.931 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.931 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.931 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.931 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e77fe640 10 monclient: _finish_auth 0 2024-09-16T05:27:05.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e77fe640 10 monclient: _check_auth_tickets 2024-09-16T05:27:05.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:05.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e77fe640 1 -- 172.21.15.114:0/423451064 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1d00ab600 con 0x7fb1e80a15d0 2024-09-16T05:27:05.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:05.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e77fe640 1 -- 172.21.15.114:0/423451064 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d00ab9d0 con 0x7fb1e80a15d0 2024-09-16T05:27:05.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 5 monclient: authenticate success, global_id 104471 2024-09-16T05:27:05.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:05.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1ef074640 4 set_mon_vals no callback set 2024-09-16T05:27:05.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:05.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.923+0000 7fb1e77fe640 10 monclient: dump: 2024-09-16T05:27:05.932 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.932 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.932 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.932 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.933 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.933 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.933 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.933 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.933 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.933 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient: shutdown 2024-09-16T05:27:05.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:05.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 -- 172.21.15.114:0/423451064 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1e80a15d0 msgr2=0x7fb1e813a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 --2- 172.21.15.114:0/423451064 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1e80a15d0 0x7fb1e813a650 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0006fd0 tx=0x7fb1d001c3c0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 -- 172.21.15.114:0/423451064 shutdown_connections 2024-09-16T05:27:05.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 --2- 172.21.15.114:0/423451064 >> v2:172.21.15.149:3300/0 conn(0x7fb1e80a19e0 0x7fb1e813cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 --2- 172.21.15.114:0/423451064 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1e80a15d0 0x7fb1e813a650 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 -- 172.21.15.114:0/423451064 >> 172.21.15.114:0/423451064 conn(0x7fb1e80a1fb0 msgr2=0x7fb1e80a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:05.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 -- 172.21.15.114:0/423451064 shutdown_connections 2024-09-16T05:27:05.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 -- 172.21.15.114:0/423451064 wait complete. 2024-09-16T05:27:05.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient: build_initial_monmap 2024-09-16T05:27:05.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient: monmap: 2024-09-16T05:27:05.934 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:05.934 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:05.934 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:05.934 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:05.934 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:05.934 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.934 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:05.934 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:05.934 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:05.934 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 librados: starting msgr at 2024-09-16T05:27:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 librados: starting objecter 2024-09-16T05:27:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 Processor -- start 2024-09-16T05:27:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 -- start start 2024-09-16T05:27:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 librados: setting wanted keys 2024-09-16T05:27:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 librados: calling monclient init 2024-09-16T05:27:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient: init 2024-09-16T05:27:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:27:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb1e809b6c0 0x7fb1e80949f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e809b6c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1e80a15d0 0x7fb1e8091c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient(hunting): picked mon.noname-c con 0x7fb1e80a15d0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1e80a19e0 0x7fb1e8092150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient(hunting): picked mon.noname-a con 0x7fb1e80a19e0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:05.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e813f900 con 0x7fb1e80a19e0 2024-09-16T05:27:05.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e809cad0 con 0x7fb1e809b6c0 2024-09-16T05:27:05.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:05.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb1e813fa80 con 0x7fb1e80a15d0 2024-09-16T05:27:05.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:05.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient(hunting): authenticate will time out at 2110.557897s 2024-09-16T05:27:05.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1e80a15d0 0x7fb1e8091c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1e80a15d0 0x7fb1e8091c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51920/0 (socket says 172.21.15.114:51920) 2024-09-16T05:27:05.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 1 -- 172.21.15.114:0/2343970713 learned_addr learned my addr 172.21.15.114:0/2343970713 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:05.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 10 monclient(hunting): get_auth_request con 0x7fb1e80a15d0 auth_method 0 2024-09-16T05:27:05.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ed611640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1e80a19e0 0x7fb1e8092150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ed611640 10 monclient(hunting): get_auth_request con 0x7fb1e80a19e0 auth_method 0 2024-09-16T05:27:05.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ed611640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ed611640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ed611640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:05.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:05.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:05.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ed611640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:05.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ed611640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:05.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ed611640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:05.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ece10640 1 --2- 172.21.15.114:0/2343970713 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb1e809b6c0 0x7fb1e80949f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ece10640 10 monclient(hunting): get_auth_request con 0x7fb1e809b6c0 auth_method 0 2024-09-16T05:27:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ece10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ece10640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ece10640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 10 monclient(hunting): handle_auth_done global_id 104474 payload 824 2024-09-16T05:27:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 1 -- 172.21.15.114:0/2343970713 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb1e809b6c0 msgr2=0x7fb1e80949f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 1 --2- 172.21.15.114:0/2343970713 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb1e809b6c0 0x7fb1e80949f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 1 -- 172.21.15.114:0/2343970713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1e80a19e0 msgr2=0x7fb1e8092150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 1 --2- 172.21.15.114:0/2343970713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1e80a19e0 0x7fb1e8092150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:27:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 1 monclient: found mon.noname-c 2024-09-16T05:27:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:27:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 1 -- 172.21.15.114:0/2343970713 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e80958c0 con 0x7fb1e80a15d0 2024-09-16T05:27:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ed611640 1 --2- 172.21.15.114:0/2343970713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1e80a19e0 0x7fb1e8092150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:27:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e7fff640 1 --2- 172.21.15.114:0/2343970713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1e80a15d0 0x7fb1e8091c10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb1d001c8a0 tx=0x7fb1d0019160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 10 client.?.objecter ms_handle_connect 0x7fb1e80a15d0 2024-09-16T05:27:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 1 -- 172.21.15.114:0/2343970713 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d00b0040 con 0x7fb1e80a15d0 2024-09-16T05:27:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:27:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 10 monclient: dump: 2024-09-16T05:27:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.940 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.940 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.940 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.940 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 10 monclient: _finish_auth 0 2024-09-16T05:27:05.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 10 monclient: _check_auth_tickets 2024-09-16T05:27:05.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:05.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 1 -- 172.21.15.114:0/2343970713 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1d0005ba0 con 0x7fb1e80a15d0 2024-09-16T05:27:05.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 5 monclient: authenticate success, global_id 104474 2024-09-16T05:27:05.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient: _renew_subs 2024-09-16T05:27:05.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:05.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 -- 172.21.15.114:0/2343970713 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1e8094f30 con 0x7fb1e80a15d0 2024-09-16T05:27:05.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:05.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 1 -- 172.21.15.114:0/2343970713 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d00ab7b0 con 0x7fb1e80a15d0 2024-09-16T05:27:05.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:05.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:05.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 10 monclient: dump: 2024-09-16T05:27:05.941 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:05.941 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:05.941 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.941 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:05.941 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:05.941 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:05.941 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:05.941 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:05.941 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:05.941 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:05.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 client.104474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:05.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient: _renew_subs 2024-09-16T05:27:05.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:05.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 -- 172.21.15.114:0/2343970713 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1e809c6a0 con 0x7fb1e80a15d0 2024-09-16T05:27:05.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 librados: init done 2024-09-16T05:27:05.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e6ffd640 4 set_mon_vals no callback set 2024-09-16T05:27:05.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 1 -- 172.21.15.114:0/2343970713 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb1d00ab950 con 0x7fb1e80a15d0 2024-09-16T05:27:05.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 1 --2- 172.21.15.114:0/2343970713 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb1c8037810 0x7fb1c8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:05.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:05.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:05.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:05.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 1 -- 172.21.15.114:0/2343970713 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7fb1d00e2770 con 0x7fb1e80a15d0 2024-09-16T05:27:05.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1ef074640 1 -- 172.21.15.114:0/2343970713 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1e80a0ec0 con 0x7fb1e80a15d0 2024-09-16T05:27:05.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 10 client.104474.objecter ms_dispatch 0x7fb1e8000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:05.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 3 client.104474.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:05.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 3 client.104474.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:05.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.927+0000 7fb1e57fa640 20 client.104474.objecter dump_active .. 0 homeless 2024-09-16T05:27:05.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.931+0000 7fb1ece10640 1 --2- 172.21.15.114:0/2343970713 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb1c8037810 0x7fb1c8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:05.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.931+0000 7fb1ece10640 10 monclient: get_auth_request con 0x7fb1c8037810 auth_method 0 2024-09-16T05:27:05.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.931+0000 7fb1e57fa640 10 client.104474.objecter ms_handle_connect 0x7fb1c8037810 2024-09-16T05:27:05.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.931+0000 7fb1ece10640 1 --2- 172.21.15.114:0/2343970713 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb1c8037810 0x7fb1c8039cd0 secure :-1 s=READY pgs=3644 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009f20 tx=0x7fb1d80094d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:05.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.935+0000 7fb1e57fa640 1 -- 172.21.15.114:0/2343970713 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb1d00160d0 con 0x7fb1e80a15d0 2024-09-16T05:27:05.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.935+0000 7fb1e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:05.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:05.935+0000 7fb1e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:27:06.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.091+0000 7fb1ef074640 1 -- 172.21.15.114:0/2343970713 --> v2:172.21.15.149:6812/1185531135 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fb1e8094190 con 0x7fb1c8037810 2024-09-16T05:27:06.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.091+0000 7fb1e57fa640 1 -- 172.21.15.114:0/2343970713 <== mgr.4108 v2:172.21.15.149:6812/1185531135 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb1e8094190 con 0x7fb1c8037810 2024-09-16T05:27:06.089 INFO:teuthology.orchestra.run.smithi114.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T05:27:06.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 10 librados: watch_flush enter 2024-09-16T05:27:06.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 10 librados: watch_flush exit 2024-09-16T05:27:06.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 20 client.104474.objecter shutdown clearing up homeless session... 2024-09-16T05:27:06.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 10 client.104474.objecter successfully canceled tick 2024-09-16T05:27:06.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 1 -- 172.21.15.114:0/2343970713 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb1c8037810 msgr2=0x7fb1c8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:06.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 1 --2- 172.21.15.114:0/2343970713 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb1c8037810 0x7fb1c8039cd0 secure :-1 s=READY pgs=3644 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009f20 tx=0x7fb1d80094d0 comp rx=0 tx=0).stop 2024-09-16T05:27:06.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 10 monclient: shutdown 2024-09-16T05:27:06.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:06.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 1 -- 172.21.15.114:0/2343970713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1e80a15d0 msgr2=0x7fb1e8091c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:06.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 1 --2- 172.21.15.114:0/2343970713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1e80a15d0 0x7fb1e8091c10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb1d001c8a0 tx=0x7fb1d0019160 comp rx=0 tx=0).stop 2024-09-16T05:27:06.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 1 -- 172.21.15.114:0/2343970713 shutdown_connections 2024-09-16T05:27:06.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 1 --2- 172.21.15.114:0/2343970713 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb1c8037810 0x7fb1c8039cd0 unknown :-1 s=CLOSED pgs=3644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:06.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 1 --2- 172.21.15.114:0/2343970713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1e80a19e0 0x7fb1e8092150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:06.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 1 --2- 172.21.15.114:0/2343970713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb1e80a15d0 0x7fb1e8091c10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:06.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 1 --2- 172.21.15.114:0/2343970713 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb1e809b6c0 0x7fb1e80949f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:06.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 1 -- 172.21.15.114:0/2343970713 >> 172.21.15.114:0/2343970713 conn(0x7fb1e80a1fb0 msgr2=0x7fb1e806e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:06.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 1 -- 172.21.15.114:0/2343970713 shutdown_connections 2024-09-16T05:27:06.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 1 -- 172.21.15.114:0/2343970713 wait complete. 2024-09-16T05:27:06.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.095+0000 7fb1ef074640 1 librados: shutdown 2024-09-16T05:27:06.108 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:27:06.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.215+0000 7fcb4ad7f640 10 monclient: get_monmap_and_config 2024-09-16T05:27:06.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.215+0000 7fcb4ad7f640 10 monclient: build_initial_monmap 2024-09-16T05:27:06.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.215+0000 7fcb4ad7f640 10 monclient: monmap: 2024-09-16T05:27:06.214 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:06.214 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:06.214 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:06.219025+0000 2024-09-16T05:27:06.214 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:06.219025+0000 2024-09-16T05:27:06.214 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:06.214 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:06.214 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:06.214 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:06.214 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:06.214 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:06.214 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:06.214 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:06.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.215+0000 7fcb4ad7f640 1 Processor -- start 2024-09-16T05:27:06.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.215+0000 7fcb4ad7f640 1 -- start start 2024-09-16T05:27:06.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.215+0000 7fcb4ad7f640 10 monclient: init 2024-09-16T05:27:06.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:06.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T05:27:06.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb440a0ec0 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:06.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb4409e1d0 addr v1:172.21.15.149:6789/0 2024-09-16T05:27:06.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcb4409e5e0 0x7fcb4413cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:06.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 10 monclient(hunting): picked mon.noname-a con 0x7fcb4409e5e0 addr v2:172.21.15.114:3300/0 2024-09-16T05:27:06.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb4409a760 con 0x7fcb440a0ec0 2024-09-16T05:27:06.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb440a1870 con 0x7fcb4409e1d0 2024-09-16T05:27:06.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:06.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fcb440a12a0 con 0x7fcb4409e5e0 2024-09-16T05:27:06.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:06.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 10 monclient(hunting): authenticate will time out at 2110.848185s 2024-09-16T05:27:06.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb48b1b640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fcb440a0ec0 0x7fcb4409dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37696/0 (socket says 172.21.15.114:37696) 2024-09-16T05:27:06.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb48b1b640 1 -- 172.21.15.114:0/1640695899 learned_addr learned my addr 172.21.15.114:0/1640695899 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:27:06.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 1 --2- 172.21.15.114:0/1640695899 >> v2:172.21.15.114:3300/0 conn(0x7fcb4409e5e0 0x7fcb4413cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:06.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 10 monclient(hunting): get_auth_request con 0x7fcb4409e5e0 auth_method 0 2024-09-16T05:27:06.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:06.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:06.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:06.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:06.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:06.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:06.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 10 monclient(hunting): handle_auth_done global_id 134262 payload 274 2024-09-16T05:27:06.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 1 -- 172.21.15.114:0/1640695899 >> v1:172.21.15.149:6789/0 conn(0x7fcb4409e1d0 legacy=0x7fcb44139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:06.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 1 -- 172.21.15.114:0/1640695899 >> v1:172.21.15.114:6789/0 conn(0x7fcb440a0ec0 legacy=0x7fcb4409dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:06.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 10 monclient: _finish_hunting 0 2024-09-16T05:27:06.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 1 monclient: found mon.noname-a 2024-09-16T05:27:06.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:06.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:06.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 1 -- 172.21.15.114:0/1640695899 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb44099790 con 0x7fcb4409e5e0 2024-09-16T05:27:06.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4931c640 1 --2- 172.21.15.114:0/1640695899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb4409e5e0 0x7fcb4413cd70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcb44074140 tx=0x7fcb3c01c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c29c1f002524b1a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:06.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb3b7fe640 1 -- 172.21.15.114:0/1640695899 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c0b2070 con 0x7fcb4409e5e0 2024-09-16T05:27:06.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:06.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb3b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:06.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb3b7fe640 10 monclient: dump: 2024-09-16T05:27:06.225 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:06.225 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:06.225 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:06.225 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:06.225 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:06.225 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:06.225 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:06.225 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:06.225 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:06.226 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:06.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb3b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:27:06.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb3b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:27:06.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:06.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb3b7fe640 1 -- 172.21.15.114:0/1640695899 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb3c0a5810 con 0x7fcb4409e5e0 2024-09-16T05:27:06.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 5 monclient: authenticate success, global_id 134262 2024-09-16T05:27:06.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:06.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb3b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:06.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:06.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 4 set_mon_vals no callback set 2024-09-16T05:27:06.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb3b7fe640 1 -- 172.21.15.114:0/1640695899 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c0a5be0 con 0x7fcb4409e5e0 2024-09-16T05:27:06.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:06.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:06.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb3b7fe640 10 monclient: dump: 2024-09-16T05:27:06.227 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:06.227 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:06.227 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:06.227 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:06.227 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:06.227 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:06.227 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:06.227 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:06.227 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:06.227 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:06.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 10 monclient: shutdown 2024-09-16T05:27:06.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:06.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 1 -- 172.21.15.114:0/1640695899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb4409e5e0 msgr2=0x7fcb4413cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:06.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 1 --2- 172.21.15.114:0/1640695899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb4409e5e0 0x7fcb4413cd70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcb44074140 tx=0x7fcb3c01c840 comp rx=0 tx=0).stop 2024-09-16T05:27:06.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 1 -- 172.21.15.114:0/1640695899 shutdown_connections 2024-09-16T05:27:06.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 1 --2- 172.21.15.114:0/1640695899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb4409e5e0 0x7fcb4413cd70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:06.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 1 -- 172.21.15.114:0/1640695899 >> 172.21.15.114:0/1640695899 conn(0x7fcb440a1fb0 msgr2=0x7fcb440a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:06.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 1 -- 172.21.15.114:0/1640695899 shutdown_connections 2024-09-16T05:27:06.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.219+0000 7fcb4ad7f640 1 -- 172.21.15.114:0/1640695899 wait complete. 2024-09-16T05:27:06.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient: build_initial_monmap 2024-09-16T05:27:06.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient: monmap: 2024-09-16T05:27:06.228 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:06.228 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:06.228 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:06.228 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:06.228 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:06.228 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:06.229 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:06.229 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:06.229 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:06.229 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:06.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 librados: starting msgr at 2024-09-16T05:27:06.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 librados: starting objecter 2024-09-16T05:27:06.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 Processor -- start 2024-09-16T05:27:06.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 -- start start 2024-09-16T05:27:06.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 librados: setting wanted keys 2024-09-16T05:27:06.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 librados: calling monclient init 2024-09-16T05:27:06.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient: init 2024-09-16T05:27:06.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:06.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:27:06.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb4409e1d0 0x7fcb44098740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:06.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient(hunting): picked mon.noname-b con 0x7fcb4409e1d0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb4409e5e0 0x7fcb44093dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient(hunting): picked mon.noname-c con 0x7fcb4409e5e0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb440a0ec0 0x7fcb44094300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient(hunting): picked mon.noname-a con 0x7fcb440a0ec0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4413fc50 con 0x7fcb440a0ec0 2024-09-16T05:27:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4413fad0 con 0x7fcb4409e1d0 2024-09-16T05:27:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fcb4413fdd0 con 0x7fcb4409e5e0 2024-09-16T05:27:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient(hunting): authenticate will time out at 2110.853180s 2024-09-16T05:27:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4931c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb440a0ec0 0x7fcb44094300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb4409e5e0 0x7fcb44093dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4931c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb440a0ec0 0x7fcb44094300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35356/0 (socket says 172.21.15.114:35356) 2024-09-16T05:27:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb4409e5e0 0x7fcb44093dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:51936/0 (socket says 172.21.15.114:51936) 2024-09-16T05:27:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4931c640 1 -- 172.21.15.114:0/1738366932 learned_addr learned my addr 172.21.15.114:0/1738366932 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4931c640 10 monclient(hunting): get_auth_request con 0x7fcb440a0ec0 auth_method 0 2024-09-16T05:27:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4931c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4931c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4931c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 10 monclient(hunting): get_auth_request con 0x7fcb4409e5e0 auth_method 0 2024-09-16T05:27:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4931c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4931c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4931c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb48b1b640 1 --2- 172.21.15.114:0/1738366932 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb4409e1d0 0x7fcb44098740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb48b1b640 10 monclient(hunting): get_auth_request con 0x7fcb4409e1d0 auth_method 0 2024-09-16T05:27:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb48b1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb48b1b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb48b1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 10 monclient(hunting): handle_auth_done global_id 104477 payload 824 2024-09-16T05:27:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 1 -- 172.21.15.114:0/1738366932 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb4409e1d0 msgr2=0x7fcb44098740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 1 --2- 172.21.15.114:0/1738366932 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb4409e1d0 0x7fcb44098740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 1 -- 172.21.15.114:0/1738366932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb440a0ec0 msgr2=0x7fcb44094300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 1 --2- 172.21.15.114:0/1738366932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb440a0ec0 0x7fcb44094300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:27:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 1 monclient: found mon.noname-c 2024-09-16T05:27:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:27:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 1 -- 172.21.15.114:0/1738366932 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb44097cd0 con 0x7fcb4409e5e0 2024-09-16T05:27:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4931c640 1 --2- 172.21.15.114:0/1738366932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb440a0ec0 0x7fcb44094300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:27:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3bfff640 1 --2- 172.21.15.114:0/1738366932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb4409e5e0 0x7fcb44093dc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400b3e0 tx=0x7fcb3400b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 10 client.?.objecter ms_handle_connect 0x7fcb4409e5e0 2024-09-16T05:27:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 1 -- 172.21.15.114:0/1738366932 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb34015020 con 0x7fcb4409e5e0 2024-09-16T05:27:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:27:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 10 monclient: dump: 2024-09-16T05:27:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:06.234 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:06.234 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:06.234 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:06.234 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:06.234 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:06.234 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:06.234 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:06.234 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:06.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 10 monclient: _finish_auth 0 2024-09-16T05:27:06.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 10 monclient: _check_auth_tickets 2024-09-16T05:27:06.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:06.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 1 -- 172.21.15.114:0/1738366932 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb34011830 con 0x7fcb4409e5e0 2024-09-16T05:27:06.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 5 monclient: authenticate success, global_id 104477 2024-09-16T05:27:06.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 1 -- 172.21.15.114:0/1738366932 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb34011c60 con 0x7fcb4409e5e0 2024-09-16T05:27:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 10 monclient: dump: 2024-09-16T05:27:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:06.235 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb3affd640 4 set_mon_vals no callback set 2024-09-16T05:27:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient: _renew_subs 2024-09-16T05:27:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 -- 172.21.15.114:0/1738366932 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb4409c6a0 con 0x7fcb4409e5e0 2024-09-16T05:27:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 client.104477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient: _renew_subs 2024-09-16T05:27:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 -- 172.21.15.114:0/1738366932 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb44097710 con 0x7fcb4409e5e0 2024-09-16T05:27:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 librados: init done 2024-09-16T05:27:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 1 -- 172.21.15.114:0/1738366932 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcb34018020 con 0x7fcb4409e5e0 2024-09-16T05:27:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb4ad7f640 1 -- 172.21.15.114:0/1738366932 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb4409dac0 con 0x7fcb4409e5e0 2024-09-16T05:27:06.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 1 --2- 172.21.15.114:0/1738366932 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcb30037840 0x7fcb30039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:06.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 1 -- 172.21.15.114:0/1738366932 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7fcb340489f0 con 0x7fcb4409e5e0 2024-09-16T05:27:06.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 10 client.104477.objecter ms_dispatch 0x7fcb44000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:06.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 3 client.104477.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:06.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 3 client.104477.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:06.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb397fa640 20 client.104477.objecter dump_active .. 0 homeless 2024-09-16T05:27:06.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb48b1b640 1 --2- 172.21.15.114:0/1738366932 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcb30037840 0x7fcb30039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:06.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.223+0000 7fcb48b1b640 10 monclient: get_auth_request con 0x7fcb30037840 auth_method 0 2024-09-16T05:27:06.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.227+0000 7fcb397fa640 10 client.104477.objecter ms_handle_connect 0x7fcb30037840 2024-09-16T05:27:06.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.227+0000 7fcb48b1b640 1 --2- 172.21.15.114:0/1738366932 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcb30037840 0x7fcb30039d00 secure :-1 s=READY pgs=3645 cs=0 l=1 rev1=1 crypto rx=0x7fcb40000f70 tx=0x7fcb40009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:06.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.227+0000 7fcb397fa640 1 -- 172.21.15.114:0/1738366932 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fcb3401f020 con 0x7fcb4409e5e0 2024-09-16T05:27:06.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.227+0000 7fcb397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:06.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.227+0000 7fcb397fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:27:06.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.391+0000 7fcb4ad7f640 1 -- 172.21.15.114:0/1738366932 --> v2:172.21.15.149:6812/1185531135 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcb440964d0 con 0x7fcb30037840 2024-09-16T05:27:06.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.399+0000 7fcb397fa640 1 -- 172.21.15.114:0/1738366932 <== mgr.4108 v2:172.21.15.149:6812/1185531135 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+113709 (secure 0 0 0) 0x7fcb440964d0 con 0x7fcb30037840 2024-09-16T05:27:06.398 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:27:06.398 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-16T05:27:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 10 librados: watch_flush enter 2024-09-16T05:27:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 10 librados: watch_flush exit 2024-09-16T05:27:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 20 client.104477.objecter shutdown clearing up homeless session... 2024-09-16T05:27:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 10 client.104477.objecter successfully canceled tick 2024-09-16T05:27:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 1 -- 172.21.15.114:0/1738366932 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcb30037840 msgr2=0x7fcb30039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 1 --2- 172.21.15.114:0/1738366932 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcb30037840 0x7fcb30039d00 secure :-1 s=READY pgs=3645 cs=0 l=1 rev1=1 crypto rx=0x7fcb40000f70 tx=0x7fcb40009040 comp rx=0 tx=0).stop 2024-09-16T05:27:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 10 monclient: shutdown 2024-09-16T05:27:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 1 -- 172.21.15.114:0/1738366932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb4409e5e0 msgr2=0x7fcb44093dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 1 --2- 172.21.15.114:0/1738366932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb4409e5e0 0x7fcb44093dc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400b3e0 tx=0x7fcb3400b8a0 comp rx=0 tx=0).stop 2024-09-16T05:27:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 1 -- 172.21.15.114:0/1738366932 shutdown_connections 2024-09-16T05:27:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 1 --2- 172.21.15.114:0/1738366932 >> v2:172.21.15.149:6812/1185531135 conn(0x7fcb30037840 0x7fcb30039d00 unknown :-1 s=CLOSED pgs=3645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 1 --2- 172.21.15.114:0/1738366932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb440a0ec0 0x7fcb44094300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 1 --2- 172.21.15.114:0/1738366932 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb4409e5e0 0x7fcb44093dc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 1 --2- 172.21.15.114:0/1738366932 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb4409e1d0 0x7fcb44098740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 1 -- 172.21.15.114:0/1738366932 >> 172.21.15.114:0/1738366932 conn(0x7fcb440a1fb0 msgr2=0x7fcb4409c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 1 -- 172.21.15.114:0/1738366932 shutdown_connections 2024-09-16T05:27:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 1 -- 172.21.15.114:0/1738366932 wait complete. 2024-09-16T05:27:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:06.403+0000 7fcb4ad7f640 1 librados: shutdown 2024-09-16T05:27:06.415 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1268,"stamp":"2024-09-16T05:27:04.513473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":464520,"kb_used_data":60696,"kb_used_omap":9,"kb_used_meta":403574,"kb_avail":565766520,"statfs":{"total":579820584960,"available":579344916480,"internally_reserved":0,"allocated":62152704,"data_stored":32273216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":413260449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003799"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021853+0000","last_change":"2024-09-16T05:08:51.596823+0000","last_active":"2024-09-16T05:25:06.021853+0000","last_peered":"2024-09-16T05:25:06.021853+0000","last_clean":"2024-09-16T05:25:06.021853+0000","last_became_active":"2024-09-16T05:08:51.571632+0000","last_became_peered":"2024-09-16T05:08:51.571632+0000","last_unstale":"2024-09-16T05:25:06.021853+0000","last_undegraded":"2024-09-16T05:25:06.021853+0000","last_fullsized":"2024-09-16T05:25:06.021853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:16.107411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1325,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:49.401087+0000","last_change":"2024-09-16T05:08:51.611318+0000","last_active":"2024-09-16T05:25:49.401087+0000","last_peered":"2024-09-16T05:25:49.401087+0000","last_clean":"2024-09-16T05:25:49.401087+0000","last_became_active":"2024-09-16T05:08:51.585360+0000","last_became_peered":"2024-09-16T05:08:51.585360+0000","last_unstale":"2024-09-16T05:25:49.401087+0000","last_undegraded":"2024-09-16T05:25:49.401087+0000","last_fullsized":"2024-09-16T05:25:49.401087+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021814+0000","last_change":"2024-09-16T05:08:51.532106+0000","last_active":"2024-09-16T05:25:06.021814+0000","last_peered":"2024-09-16T05:25:06.021814+0000","last_clean":"2024-09-16T05:25:06.021814+0000","last_became_active":"2024-09-16T05:08:51.519558+0000","last_became_peered":"2024-09-16T05:08:51.519558+0000","last_unstale":"2024-09-16T05:25:06.021814+0000","last_undegraded":"2024-09-16T05:25:06.021814+0000","last_fullsized":"2024-09-16T05:25:06.021814+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:52:54.764452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1316,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.441285+0000","last_change":"2024-09-16T05:08:51.705649+0000","last_active":"2024-09-16T05:25:05.441285+0000","last_peered":"2024-09-16T05:25:05.441285+0000","last_clean":"2024-09-16T05:25:05.441285+0000","last_became_active":"2024-09-16T05:08:51.689931+0000","last_became_peered":"2024-09-16T05:08:51.689931+0000","last_unstale":"2024-09-16T05:25:05.441285+0000","last_undegraded":"2024-09-16T05:25:05.441285+0000","last_fullsized":"2024-09-16T05:25:05.441285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:31.971156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522829+0000","last_change":"2024-09-16T05:08:51.564493+0000","last_active":"2024-09-16T05:25:05.522829+0000","last_peered":"2024-09-16T05:25:05.522829+0000","last_clean":"2024-09-16T05:25:05.522829+0000","last_became_active":"2024-09-16T05:08:51.546851+0000","last_became_peered":"2024-09-16T05:08:51.546851+0000","last_unstale":"2024-09-16T05:25:05.522829+0000","last_undegraded":"2024-09-16T05:25:05.522829+0000","last_fullsized":"2024-09-16T05:25:05.522829+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:23.731218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.463123+0000","last_change":"2024-09-16T05:08:51.617848+0000","last_active":"2024-09-16T05:25:04.463123+0000","last_peered":"2024-09-16T05:25:04.463123+0000","last_clean":"2024-09-16T05:25:04.463123+0000","last_became_active":"2024-09-16T05:08:51.605843+0000","last_became_peered":"2024-09-16T05:08:51.605843+0000","last_unstale":"2024-09-16T05:25:04.463123+0000","last_undegraded":"2024-09-16T05:25:04.463123+0000","last_fullsized":"2024-09-16T05:25:04.463123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:24:14.376183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1317,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.441115+0000","last_change":"2024-09-16T05:08:51.615740+0000","last_active":"2024-09-16T05:25:05.441115+0000","last_peered":"2024-09-16T05:25:05.441115+0000","last_clean":"2024-09-16T05:25:05.441115+0000","last_became_active":"2024-09-16T05:08:51.609820+0000","last_became_peered":"2024-09-16T05:08:51.609820+0000","last_unstale":"2024-09-16T05:25:05.441115+0000","last_undegraded":"2024-09-16T05:25:05.441115+0000","last_fullsized":"2024-09-16T05:25:05.441115+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:35:22.111340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1322,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021670+0000","last_change":"2024-09-16T05:08:51.603805+0000","last_active":"2024-09-16T05:25:06.021670+0000","last_peered":"2024-09-16T05:25:06.021670+0000","last_clean":"2024-09-16T05:25:06.021670+0000","last_became_active":"2024-09-16T05:08:51.567268+0000","last_became_peered":"2024-09-16T05:08:51.567268+0000","last_unstale":"2024-09-16T05:25:06.021670+0000","last_undegraded":"2024-09-16T05:25:06.021670+0000","last_fullsized":"2024-09-16T05:25:06.021670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:43:35.723700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1326,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:26:54.410462+0000","last_change":"2024-09-16T05:08:51.640189+0000","last_active":"2024-09-16T05:26:54.410462+0000","last_peered":"2024-09-16T05:26:54.410462+0000","last_clean":"2024-09-16T05:26:54.410462+0000","last_became_active":"2024-09-16T05:08:51.634768+0000","last_became_peered":"2024-09-16T05:08:51.634768+0000","last_unstale":"2024-09-16T05:26:54.410462+0000","last_undegraded":"2024-09-16T05:26:54.410462+0000","last_fullsized":"2024-09-16T05:26:54.410462+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1323,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.735797+0000","last_change":"2024-09-16T05:08:51.589983+0000","last_active":"2024-09-16T05:25:04.735797+0000","last_peered":"2024-09-16T05:25:04.735797+0000","last_clean":"2024-09-16T05:25:04.735797+0000","last_became_active":"2024-09-16T05:08:51.589818+0000","last_became_peered":"2024-09-16T05:08:51.589818+0000","last_unstale":"2024-09-16T05:25:04.735797+0000","last_undegraded":"2024-09-16T05:25:04.735797+0000","last_fullsized":"2024-09-16T05:25:04.735797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:55:59.967412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.465088+0000","last_change":"2024-09-16T05:08:51.609659+0000","last_active":"2024-09-16T05:25:04.465088+0000","last_peered":"2024-09-16T05:25:04.465088+0000","last_clean":"2024-09-16T05:25:04.465088+0000","last_became_active":"2024-09-16T05:08:51.605719+0000","last_became_peered":"2024-09-16T05:08:51.605719+0000","last_unstale":"2024-09-16T05:25:04.465088+0000","last_undegraded":"2024-09-16T05:25:04.465088+0000","last_fullsized":"2024-09-16T05:25:04.465088+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:43:56.008078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1326,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:26:54.410414+0000","last_change":"2024-09-16T05:08:51.630514+0000","last_active":"2024-09-16T05:26:54.410414+0000","last_peered":"2024-09-16T05:26:54.410414+0000","last_clean":"2024-09-16T05:26:54.410414+0000","last_became_active":"2024-09-16T05:08:51.611801+0000","last_became_peered":"2024-09-16T05:08:51.611801+0000","last_unstale":"2024-09-16T05:26:54.410414+0000","last_undegraded":"2024-09-16T05:26:54.410414+0000","last_fullsized":"2024-09-16T05:26:54.410414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522975+0000","last_change":"2024-09-16T05:08:51.605439+0000","last_active":"2024-09-16T05:25:05.522975+0000","last_peered":"2024-09-16T05:25:05.522975+0000","last_clean":"2024-09-16T05:25:05.522975+0000","last_became_active":"2024-09-16T05:08:51.586611+0000","last_became_peered":"2024-09-16T05:08:51.586611+0000","last_unstale":"2024-09-16T05:25:05.522975+0000","last_undegraded":"2024-09-16T05:25:05.522975+0000","last_fullsized":"2024-09-16T05:25:05.522975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:39:46.639778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":1493,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:39.399603+0000","last_change":"2024-09-16T05:07:39.580660+0000","last_active":"2024-09-16T05:25:39.399603+0000","last_peered":"2024-09-16T05:25:39.399603+0000","last_clean":"2024-09-16T05:25:39.399603+0000","last_became_active":"2024-09-16T05:07:39.576756+0000","last_became_peered":"2024-09-16T05:07:39.576756+0000","last_unstale":"2024-09-16T05:25:39.399603+0000","last_undegraded":"2024-09-16T05:25:39.399603+0000","last_fullsized":"2024-09-16T05:25:39.399603+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:38.167879+0000","last_clean_scrub_stamp":"2024-09-16T05:07:38.167879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1343,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:26:34.407559+0000","last_change":"2024-09-16T05:07:48.267201+0000","last_active":"2024-09-16T05:26:34.407559+0000","last_peered":"2024-09-16T05:26:34.407559+0000","last_clean":"2024-09-16T05:26:34.407559+0000","last_became_active":"2024-09-16T05:07:46.246409+0000","last_became_peered":"2024-09-16T05:07:46.246409+0000","last_unstale":"2024-09-16T05:26:34.407559+0000","last_undegraded":"2024-09-16T05:26:34.407559+0000","last_fullsized":"2024-09-16T05:26:34.407559+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010413149999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1323,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.735867+0000","last_change":"2024-09-16T05:08:51.582564+0000","last_active":"2024-09-16T05:25:04.735867+0000","last_peered":"2024-09-16T05:25:04.735867+0000","last_clean":"2024-09-16T05:25:04.735867+0000","last_became_active":"2024-09-16T05:08:51.582424+0000","last_became_peered":"2024-09-16T05:08:51.582424+0000","last_unstale":"2024-09-16T05:25:04.735867+0000","last_undegraded":"2024-09-16T05:25:04.735867+0000","last_fullsized":"2024-09-16T05:25:04.735867+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:43.882076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1333,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.441020+0000","last_change":"2024-09-16T05:07:48.266504+0000","last_active":"2024-09-16T05:25:05.441020+0000","last_peered":"2024-09-16T05:25:05.441020+0000","last_clean":"2024-09-16T05:25:05.441020+0000","last_became_active":"2024-09-16T05:07:46.254322+0000","last_became_peered":"2024-09-16T05:07:46.254322+0000","last_unstale":"2024-09-16T05:25:05.441020+0000","last_undegraded":"2024-09-16T05:25:05.441020+0000","last_fullsized":"2024-09-16T05:25:05.441020+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:00.175590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093680599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522665+0000","last_change":"2024-09-16T05:08:51.549772+0000","last_active":"2024-09-16T05:25:05.522665+0000","last_peered":"2024-09-16T05:25:05.522665+0000","last_clean":"2024-09-16T05:25:05.522665+0000","last_became_active":"2024-09-16T05:08:51.539573+0000","last_became_peered":"2024-09-16T05:08:51.539573+0000","last_unstale":"2024-09-16T05:25:05.522665+0000","last_undegraded":"2024-09-16T05:25:05.522665+0000","last_fullsized":"2024-09-16T05:25:05.522665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:10:54.876819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1362,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.462909+0000","last_change":"2024-09-16T05:07:48.263513+0000","last_active":"2024-09-16T05:25:04.462909+0000","last_peered":"2024-09-16T05:25:04.462909+0000","last_clean":"2024-09-16T05:25:04.462909+0000","last_became_active":"2024-09-16T05:07:46.252254+0000","last_became_peered":"2024-09-16T05:07:46.252254+0000","last_unstale":"2024-09-16T05:25:04.462909+0000","last_undegraded":"2024-09-16T05:25:04.462909+0000","last_fullsized":"2024-09-16T05:25:04.462909+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:33.736560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081560699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.465371+0000","last_change":"2024-09-16T05:08:51.542756+0000","last_active":"2024-09-16T05:25:04.465371+0000","last_peered":"2024-09-16T05:25:04.465371+0000","last_clean":"2024-09-16T05:25:04.465371+0000","last_became_active":"2024-09-16T05:08:51.541976+0000","last_became_peered":"2024-09-16T05:08:51.541976+0000","last_unstale":"2024-09-16T05:25:04.465371+0000","last_undegraded":"2024-09-16T05:25:04.465371+0000","last_fullsized":"2024-09-16T05:25:04.465371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:48:46.043655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":1347,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522619+0000","last_change":"2024-09-16T05:07:48.274329+0000","last_active":"2024-09-16T05:25:05.522619+0000","last_peered":"2024-09-16T05:25:05.522619+0000","last_clean":"2024-09-16T05:25:05.522619+0000","last_became_active":"2024-09-16T05:07:46.257118+0000","last_became_peered":"2024-09-16T05:07:46.257118+0000","last_unstale":"2024-09-16T05:25:05.522619+0000","last_undegraded":"2024-09-16T05:25:05.522619+0000","last_fullsized":"2024-09-16T05:25:05.522619+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:35.932772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020216650000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1323,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021532+0000","last_change":"2024-09-16T05:08:51.506142+0000","last_active":"2024-09-16T05:25:06.021532+0000","last_peered":"2024-09-16T05:25:06.021532+0000","last_clean":"2024-09-16T05:25:06.021532+0000","last_became_active":"2024-09-16T05:08:51.491760+0000","last_became_peered":"2024-09-16T05:08:51.491760+0000","last_unstale":"2024-09-16T05:25:06.021532+0000","last_undegraded":"2024-09-16T05:25:06.021532+0000","last_fullsized":"2024-09-16T05:25:06.021532+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:04:43.526278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1339,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.735843+0000","last_change":"2024-09-16T05:07:48.352028+0000","last_active":"2024-09-16T05:25:04.735843+0000","last_peered":"2024-09-16T05:25:04.735843+0000","last_clean":"2024-09-16T05:25:04.735843+0000","last_became_active":"2024-09-16T05:07:46.255633+0000","last_became_peered":"2024-09-16T05:07:46.255633+0000","last_unstale":"2024-09-16T05:25:04.735843+0000","last_undegraded":"2024-09-16T05:25:04.735843+0000","last_fullsized":"2024-09-16T05:25:04.735843+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:07.079219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000883379,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"21'7","reported_seq":1359,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.462678+0000","last_change":"2024-09-16T05:08:51.617951+0000","last_active":"2024-09-16T05:25:04.462678+0000","last_peered":"2024-09-16T05:25:04.462678+0000","last_clean":"2024-09-16T05:25:04.462678+0000","last_became_active":"2024-09-16T05:08:51.614284+0000","last_became_peered":"2024-09-16T05:08:51.614284+0000","last_unstale":"2024-09-16T05:25:04.462678+0000","last_undegraded":"2024-09-16T05:25:04.462678+0000","last_fullsized":"2024-09-16T05:25:04.462678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:18.960138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1342,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:26:34.407628+0000","last_change":"2024-09-16T05:07:48.267372+0000","last_active":"2024-09-16T05:26:34.407628+0000","last_peered":"2024-09-16T05:26:34.407628+0000","last_clean":"2024-09-16T05:26:34.407628+0000","last_became_active":"2024-09-16T05:07:46.243891+0000","last_became_peered":"2024-09-16T05:07:46.243891+0000","last_unstale":"2024-09-16T05:26:34.407628+0000","last_undegraded":"2024-09-16T05:26:34.407628+0000","last_fullsized":"2024-09-16T05:26:34.407628+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011829709999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1326,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:34.398782+0000","last_change":"2024-09-16T05:08:51.634776+0000","last_active":"2024-09-16T05:25:34.398782+0000","last_peered":"2024-09-16T05:25:34.398782+0000","last_clean":"2024-09-16T05:25:34.398782+0000","last_became_active":"2024-09-16T05:08:51.632213+0000","last_became_peered":"2024-09-16T05:08:51.632213+0000","last_unstale":"2024-09-16T05:25:34.398782+0000","last_undegraded":"2024-09-16T05:25:34.398782+0000","last_fullsized":"2024-09-16T05:25:34.398782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1337,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021856+0000","last_change":"2024-09-16T05:07:48.275545+0000","last_active":"2024-09-16T05:25:06.021856+0000","last_peered":"2024-09-16T05:25:06.021856+0000","last_clean":"2024-09-16T05:25:06.021856+0000","last_became_active":"2024-09-16T05:07:46.252207+0000","last_became_peered":"2024-09-16T05:07:46.252207+0000","last_unstale":"2024-09-16T05:25:06.021856+0000","last_undegraded":"2024-09-16T05:25:06.021856+0000","last_fullsized":"2024-09-16T05:25:06.021856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:36:08.378166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073333700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1329,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522617+0000","last_change":"2024-09-16T05:08:51.499333+0000","last_active":"2024-09-16T05:25:05.522617+0000","last_peered":"2024-09-16T05:25:05.522617+0000","last_clean":"2024-09-16T05:25:05.522617+0000","last_became_active":"2024-09-16T05:08:51.486807+0000","last_became_peered":"2024-09-16T05:08:51.486807+0000","last_unstale":"2024-09-16T05:25:05.522617+0000","last_undegraded":"2024-09-16T05:25:05.522617+0000","last_fullsized":"2024-09-16T05:25:05.522617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:43.084461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1341,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:09.395036+0000","last_change":"2024-09-16T05:07:48.267914+0000","last_active":"2024-09-16T05:25:09.395036+0000","last_peered":"2024-09-16T05:25:09.395036+0000","last_clean":"2024-09-16T05:25:09.395036+0000","last_became_active":"2024-09-16T05:07:46.254034+0000","last_became_peered":"2024-09-16T05:07:46.254034+0000","last_unstale":"2024-09-16T05:25:09.395036+0000","last_undegraded":"2024-09-16T05:25:09.395036+0000","last_fullsized":"2024-09-16T05:25:09.395036+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:07:45.220505+0000","last_clean_scrub_stamp":"2024-09-16T05:07:45.220505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014490219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021562+0000","last_change":"2024-09-16T05:08:51.548570+0000","last_active":"2024-09-16T05:25:06.021562+0000","last_peered":"2024-09-16T05:25:06.021562+0000","last_clean":"2024-09-16T05:25:06.021562+0000","last_became_active":"2024-09-16T05:08:51.532685+0000","last_became_peered":"2024-09-16T05:08:51.532685+0000","last_unstale":"2024-09-16T05:25:06.021562+0000","last_undegraded":"2024-09-16T05:25:06.021562+0000","last_fullsized":"2024-09-16T05:25:06.021562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:14:25.488128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021522+0000","last_change":"2024-09-16T05:08:51.604786+0000","last_active":"2024-09-16T05:25:06.021522+0000","last_peered":"2024-09-16T05:25:06.021522+0000","last_clean":"2024-09-16T05:25:06.021522+0000","last_became_active":"2024-09-16T05:08:51.567861+0000","last_became_peered":"2024-09-16T05:08:51.567861+0000","last_unstale":"2024-09-16T05:25:06.021522+0000","last_undegraded":"2024-09-16T05:25:06.021522+0000","last_fullsized":"2024-09-16T05:25:06.021522+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:48:14.930571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1345,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.463346+0000","last_change":"2024-09-16T05:08:51.563783+0000","last_active":"2024-09-16T05:25:04.463346+0000","last_peered":"2024-09-16T05:25:04.463346+0000","last_clean":"2024-09-16T05:25:04.463346+0000","last_became_active":"2024-09-16T05:08:51.563420+0000","last_became_peered":"2024-09-16T05:08:51.563420+0000","last_unstale":"2024-09-16T05:25:04.463346+0000","last_undegraded":"2024-09-16T05:25:04.463346+0000","last_fullsized":"2024-09-16T05:25:04.463346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:52.527569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1328,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522885+0000","last_change":"2024-09-16T05:08:51.513817+0000","last_active":"2024-09-16T05:25:05.522885+0000","last_peered":"2024-09-16T05:25:05.522885+0000","last_clean":"2024-09-16T05:25:05.522885+0000","last_became_active":"2024-09-16T05:08:51.503720+0000","last_became_peered":"2024-09-16T05:08:51.503720+0000","last_unstale":"2024-09-16T05:25:05.522885+0000","last_undegraded":"2024-09-16T05:25:05.522885+0000","last_fullsized":"2024-09-16T05:25:05.522885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:31.884629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021857+0000","last_change":"2024-09-16T05:08:51.505598+0000","last_active":"2024-09-16T05:25:06.021857+0000","last_peered":"2024-09-16T05:25:06.021857+0000","last_clean":"2024-09-16T05:25:06.021857+0000","last_became_active":"2024-09-16T05:08:51.494038+0000","last_became_peered":"2024-09-16T05:08:51.494038+0000","last_unstale":"2024-09-16T05:25:06.021857+0000","last_undegraded":"2024-09-16T05:25:06.021857+0000","last_fullsized":"2024-09-16T05:25:06.021857+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:39:28.303932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1325,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:26:24.406088+0000","last_change":"2024-09-16T05:08:51.680865+0000","last_active":"2024-09-16T05:26:24.406088+0000","last_peered":"2024-09-16T05:26:24.406088+0000","last_clean":"2024-09-16T05:26:24.406088+0000","last_became_active":"2024-09-16T05:08:51.680588+0000","last_became_peered":"2024-09-16T05:08:51.680588+0000","last_unstale":"2024-09-16T05:26:24.406088+0000","last_undegraded":"2024-09-16T05:26:24.406088+0000","last_fullsized":"2024-09-16T05:26:24.406088+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1346,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.463756+0000","last_change":"2024-09-16T05:08:51.490785+0000","last_active":"2024-09-16T05:25:04.463756+0000","last_peered":"2024-09-16T05:25:04.463756+0000","last_clean":"2024-09-16T05:25:04.463756+0000","last_became_active":"2024-09-16T05:08:51.480748+0000","last_became_peered":"2024-09-16T05:08:51.480748+0000","last_unstale":"2024-09-16T05:25:04.463756+0000","last_undegraded":"2024-09-16T05:25:04.463756+0000","last_fullsized":"2024-09-16T05:25:04.463756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:21:24.554888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1326,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:26:54.410483+0000","last_change":"2024-09-16T05:08:51.623309+0000","last_active":"2024-09-16T05:26:54.410483+0000","last_peered":"2024-09-16T05:26:54.410483+0000","last_clean":"2024-09-16T05:26:54.410483+0000","last_became_active":"2024-09-16T05:08:51.622941+0000","last_became_peered":"2024-09-16T05:08:51.622941+0000","last_unstale":"2024-09-16T05:26:54.410483+0000","last_undegraded":"2024-09-16T05:26:54.410483+0000","last_fullsized":"2024-09-16T05:26:54.410483+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1328,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522520+0000","last_change":"2024-09-16T05:08:51.493220+0000","last_active":"2024-09-16T05:25:05.522520+0000","last_peered":"2024-09-16T05:25:05.522520+0000","last_clean":"2024-09-16T05:25:05.522520+0000","last_became_active":"2024-09-16T05:08:51.486048+0000","last_became_peered":"2024-09-16T05:08:51.486048+0000","last_unstale":"2024-09-16T05:25:05.522520+0000","last_undegraded":"2024-09-16T05:25:05.522520+0000","last_fullsized":"2024-09-16T05:25:05.522520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:52:13.473044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1321,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:06.021819+0000","last_change":"2024-09-16T05:08:51.472111+0000","last_active":"2024-09-16T05:25:06.021819+0000","last_peered":"2024-09-16T05:25:06.021819+0000","last_clean":"2024-09-16T05:25:06.021819+0000","last_became_active":"2024-09-16T05:08:51.469556+0000","last_became_peered":"2024-09-16T05:08:51.469556+0000","last_unstale":"2024-09-16T05:25:06.021819+0000","last_undegraded":"2024-09-16T05:25:06.021819+0000","last_fullsized":"2024-09-16T05:25:06.021819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:14.956562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1345,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:04.462705+0000","last_change":"2024-09-16T05:08:51.533742+0000","last_active":"2024-09-16T05:25:04.462705+0000","last_peered":"2024-09-16T05:25:04.462705+0000","last_clean":"2024-09-16T05:25:04.462705+0000","last_became_active":"2024-09-16T05:08:51.533473+0000","last_became_peered":"2024-09-16T05:08:51.533473+0000","last_unstale":"2024-09-16T05:25:04.462705+0000","last_undegraded":"2024-09-16T05:25:04.462705+0000","last_fullsized":"2024-09-16T05:25:04.462705+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:21.233490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1328,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:25:05.522976+0000","last_change":"2024-09-16T05:08:51.509275+0000","last_active":"2024-09-16T05:25:05.522976+0000","last_peered":"2024-09-16T05:25:05.522976+0000","last_clean":"2024-09-16T05:25:05.522976+0000","last_became_active":"2024-09-16T05:08:51.493446+0000","last_became_peered":"2024-09-16T05:08:51.493446+0000","last_unstale":"2024-09-16T05:25:05.522976+0000","last_undegraded":"2024-09-16T05:25:05.522976+0000","last_fullsized":"2024-09-16T05:25:05.522976+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:08:50.336200+0000","last_clean_scrub_stamp":"2024-09-16T05:08:50.336200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:19.908189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834575084,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73084,"kb_used_data":9940,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298756,"statfs":{"total":96636764160,"available":96561926144,"internally_reserved":0,"allocated":10178560,"data_stored":5202552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575084,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77260,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294580,"statfs":{"total":96636764160,"available":96557649920,"internally_reserved":0,"allocated":10182656,"data_stored":5202571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607788,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73604,"kb_used_data":10460,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298236,"statfs":{"total":96636764160,"available":96561393664,"internally_reserved":0,"allocated":10711040,"data_stored":5731480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607788,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73608,"kb_used_data":10464,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298232,"statfs":{"total":96636764160,"available":96561389568,"internally_reserved":0,"allocated":10715136,"data_stored":5731485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640493,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77248,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294592,"statfs":{"total":96636764160,"available":96557662208,"internally_reserved":0,"allocated":10182656,"data_stored":5202571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640493,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89716,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94282124,"statfs":{"total":96636764160,"available":96544894976,"internally_reserved":0,"allocated":10182656,"data_stored":5202557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81656263},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:27:06.421 INFO:tasks.ceph:pgid 6.16 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.421 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.421 INFO:tasks.ceph:pgid 6.14 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.421 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.421 INFO:tasks.ceph:pgid 6.a last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.422 INFO:tasks.ceph:pgid 6.b last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.422 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.422 INFO:tasks.ceph:pgid 6.9 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.422 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.422 INFO:tasks.ceph:pgid 6.f last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.422 INFO:tasks.ceph:pgid 6.c last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.422 INFO:tasks.ceph:pgid 6.d last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.422 INFO:tasks.ceph:pgid 6.7 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.422 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T05:07:38.167879+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=7, tm_sec=38, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.423 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T05:07:45.220505+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=7, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.423 INFO:tasks.ceph:pgid 6.4 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.423 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T05:07:45.220505+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=7, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.423 INFO:tasks.ceph:pgid 6.5 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.423 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T05:07:45.220505+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=7, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.423 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.423 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T05:07:45.220505+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=7, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.423 INFO:tasks.ceph:pgid 6.0 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.423 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T05:07:45.220505+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=7, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.424 INFO:tasks.ceph:pgid 6.1 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.424 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T05:07:45.220505+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=7, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.424 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.424 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T05:07:45.220505+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=7, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.424 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.424 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T05:07:45.220505+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=7, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.424 INFO:tasks.ceph:pgid 6.11 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.424 INFO:tasks.ceph:pgid 6.10 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.424 INFO:tasks.ceph:pgid 6.13 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.425 INFO:tasks.ceph:pgid 6.12 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.425 INFO:tasks.ceph:pgid 6.1d last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.425 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.425 INFO:tasks.ceph:pgid 6.1f last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.425 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.425 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.425 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.425 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.425 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2024-09-16T05:08:50.336200+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=8, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=27, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:27:06.425 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T05:27:26.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:27:26.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.554+0000 7f907541f640 10 monclient: get_monmap_and_config 2024-09-16T05:27:26.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.554+0000 7f907541f640 10 monclient: build_initial_monmap 2024-09-16T05:27:26.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.554+0000 7f907541f640 10 monclient: monmap: 2024-09-16T05:27:26.554 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:26.555 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:26.555 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:26.559696+0000 2024-09-16T05:27:26.555 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:26.559696+0000 2024-09-16T05:27:26.555 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:26.555 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:26.555 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:26.555 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:26.555 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:26.555 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:26.555 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:26.555 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:26.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 1 Processor -- start 2024-09-16T05:27:26.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 1 -- start start 2024-09-16T05:27:26.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 10 monclient: init 2024-09-16T05:27:26.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:26.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-16T05:27:26.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f90700a0ec0 0x7f907009dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 10 monclient(hunting): picked mon.noname-c con 0x7f90700a0ec0 addr v2:172.21.15.149:3300/0 2024-09-16T05:27:26.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f907009e000 addr v1:172.21.15.149:6789/0 2024-09-16T05:27:26.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9070139640 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:26.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f907009a760 con 0x7f9070139640 2024-09-16T05:27:26.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90700a1870 con 0x7f907009e000 2024-09-16T05:27:26.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:26.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7f90700994b0 con 0x7f90700a0ec0 2024-09-16T05:27:26.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:26.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 10 monclient(hunting): authenticate will time out at 2131.188854s 2024-09-16T05:27:26.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906f7fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9070139640 0x7f907013ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56388/0 (socket says 172.21.15.114:56388) 2024-09-16T05:27:26.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906f7fe640 1 -- 172.21.15.114:0/567394059 learned_addr learned my addr 172.21.15.114:0/567394059 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:27:26.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 -- 172.21.15.114:0/567394059 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1232351537 0 0) 0x7f907009a760 con 0x7f9070139640 2024-09-16T05:27:26.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:27:26.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:26.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:26.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient(hunting): my global_id is 134271 2024-09-16T05:27:26.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 -- 172.21.15.114:0/567394059 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9058003530 con 0x7f9070139640 2024-09-16T05:27:26.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906effd640 1 --2- 172.21.15.114:0/567394059 >> v2:172.21.15.149:3300/0 conn(0x7f90700a0ec0 0x7f907009dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:26.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906effd640 10 monclient(hunting): get_auth_request con 0x7f90700a0ec0 auth_method 0 2024-09-16T05:27:26.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:26.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:26.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:26.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 -- 172.21.15.114:0/567394059 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4121981902 0 0) 0x7f9058003530 con 0x7f9070139640 2024-09-16T05:27:26.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 -- 172.21.15.114:0/567394059 >> v2:172.21.15.149:3300/0 conn(0x7f90700a0ec0 msgr2=0x7f907009dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:26.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 --2- 172.21.15.114:0/567394059 >> v2:172.21.15.149:3300/0 conn(0x7f90700a0ec0 0x7f907009dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 -- 172.21.15.114:0/567394059 >> v1:172.21.15.149:6789/0 conn(0x7f907009e000 legacy=0x7f907009e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:26.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:27:26.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:27:26.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:26.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:27:26.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 -- 172.21.15.114:0/567394059 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9070099790 con 0x7f9070139640 2024-09-16T05:27:26.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient: _finish_auth 0 2024-09-16T05:27:26.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:27:26.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:26.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 -- 172.21.15.114:0/567394059 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7f9068001a20 con 0x7f9070139640 2024-09-16T05:27:26.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:26.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:27:26.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient: dump: 2024-09-16T05:27:26.564 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:26.564 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:26.564 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.564 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.564 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:26.565 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:26.565 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:26.565 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:26.565 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:26.565 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:26.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:27:26.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:26.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 -- 172.21.15.114:0/567394059 >> v1:172.21.15.114:6789/0 conn(0x7f9070139640 legacy=0x7f907013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:26.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:27:26.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 --2- 172.21.15.114:0/567394059 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90580039c0 0x7f9058003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient(hunting): picked mon.b con 0x7f90580039c0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:26.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 --2- 172.21.15.114:0/567394059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9058004810 0x7f9058004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient(hunting): picked mon.c con 0x7f9058004810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:26.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 --2- 172.21.15.114:0/567394059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90580052f0 0x7f905800dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient(hunting): picked mon.a con 0x7f90580052f0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:26.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:26.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 -- 172.21.15.114:0/567394059 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f90700994b0 con 0x7f90580052f0 2024-09-16T05:27:26.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:26.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 -- 172.21.15.114:0/567394059 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f905800e3d0 con 0x7f90580039c0 2024-09-16T05:27:26.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:26.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 1 -- 172.21.15.114:0/567394059 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f905800e5d0 con 0x7f9058004810 2024-09-16T05:27:26.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:26.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:27:26.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 1 --2- 172.21.15.114:0/567394059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9058004810 0x7f9058004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:26.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 10 monclient(hunting): get_auth_request con 0x7f9058004810 auth_method 0 2024-09-16T05:27:26.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:26.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:26.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:27:26.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906f7fe640 1 --2- 172.21.15.114:0/567394059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90580052f0 0x7f905800dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:26.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:26.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:26.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:27:26.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906effd640 1 --2- 172.21.15.114:0/567394059 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90580039c0 0x7f9058003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:26.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906f7fe640 10 monclient(hunting): get_auth_request con 0x7f90580052f0 auth_method 0 2024-09-16T05:27:26.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:26.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:26.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:27:26.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906effd640 10 monclient(hunting): get_auth_request con 0x7f90580039c0 auth_method 0 2024-09-16T05:27:26.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:26.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:26.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:27:26.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 10 monclient(hunting): handle_auth_done global_id 134271 payload 293 2024-09-16T05:27:26.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 1 -- 172.21.15.114:0/567394059 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90580039c0 msgr2=0x7f9058003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:26.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 1 --2- 172.21.15.114:0/567394059 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90580039c0 0x7f9058003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 1 -- 172.21.15.114:0/567394059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90580052f0 msgr2=0x7f905800dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:26.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 1 --2- 172.21.15.114:0/567394059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90580052f0 0x7f905800dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:27:26.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 1 monclient: found mon.c 2024-09-16T05:27:26.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:26.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:26.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 1 -- 172.21.15.114:0/567394059 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9058003530 con 0x7f9058004810 2024-09-16T05:27:26.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:27:26.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:27:26.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:26.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 5 monclient: authenticate success, global_id 134271 2024-09-16T05:27:26.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f907541f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:26.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906f7fe640 1 --2- 172.21.15.114:0/567394059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90580052f0 0x7f905800dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T05:27:26.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.558+0000 7f906f7fe640 1 -- 172.21.15.114:0/567394059 reap_dead start 2024-09-16T05:27:26.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 1 --2- 172.21.15.114:0/567394059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9058004810 0x7f9058004c90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f90700a1a40 tx=0x7f906401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24895e90db19f00b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:26.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906dffb640 1 -- 172.21.15.114:0/567394059 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90640013d0 con 0x7f9058004810 2024-09-16T05:27:26.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:26.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:26.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906dffb640 10 monclient: dump: 2024-09-16T05:27:26.569 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:26.569 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:26.569 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.569 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.569 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:26.569 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:26.570 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:26.570 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:26.570 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:26.570 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:26.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:26.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906dffb640 1 -- 172.21.15.114:0/567394059 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90640052d0 con 0x7f9058004810 2024-09-16T05:27:26.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:26.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906dffb640 1 -- 172.21.15.114:0/567394059 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9064003650 con 0x7f9058004810 2024-09-16T05:27:26.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:26.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 4 set_mon_vals no callback set 2024-09-16T05:27:26.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:26.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:26.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906dffb640 10 monclient: dump: 2024-09-16T05:27:26.570 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:26.571 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:26.571 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.571 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.571 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:26.571 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:26.571 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:26.571 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:26.571 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:26.571 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:26.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 10 monclient: shutdown 2024-09-16T05:27:26.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:26.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 -- 172.21.15.114:0/567394059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9058004810 msgr2=0x7f9058004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:26.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 --2- 172.21.15.114:0/567394059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9058004810 0x7f9058004c90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f90700a1a40 tx=0x7f906401c360 comp rx=0 tx=0).stop 2024-09-16T05:27:26.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 -- 172.21.15.114:0/567394059 shutdown_connections 2024-09-16T05:27:26.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 --2- 172.21.15.114:0/567394059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9058004810 0x7f9058004c90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 -- 172.21.15.114:0/567394059 >> 172.21.15.114:0/567394059 conn(0x7f90700a1fb0 msgr2=0x7f90700a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:26.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 -- 172.21.15.114:0/567394059 shutdown_connections 2024-09-16T05:27:26.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 -- 172.21.15.114:0/567394059 wait complete. 2024-09-16T05:27:26.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 10 monclient: build_initial_monmap 2024-09-16T05:27:26.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 10 monclient: monmap: 2024-09-16T05:27:26.573 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:26.573 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:26.573 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:26.573 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:26.573 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:26.573 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:26.574 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:26.574 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:26.574 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:26.574 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:26.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 librados: starting msgr at 2024-09-16T05:27:26.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 librados: starting objecter 2024-09-16T05:27:26.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 Processor -- start 2024-09-16T05:27:26.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 -- start start 2024-09-16T05:27:26.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 librados: setting wanted keys 2024-09-16T05:27:26.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 librados: calling monclient init 2024-09-16T05:27:26.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 10 monclient: init 2024-09-16T05:27:26.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:26.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:27:26.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9058004810 0x7f90700730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 10 monclient(hunting): picked mon.noname-b con 0x7f9058004810 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:26.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907012a570 0x7f907012a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 10 monclient(hunting): picked mon.noname-c con 0x7f907012a570 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:26.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907012aeb0 0x7f90701ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 10 monclient(hunting): picked mon.noname-a con 0x7f907012aeb0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:26.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:26.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f90700a1df0 con 0x7f907012aeb0 2024-09-16T05:27:26.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:26.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7f90700a1c70 con 0x7f9058004810 2024-09-16T05:27:26.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:26.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f907009c230 con 0x7f907012a570 2024-09-16T05:27:26.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:26.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 10 monclient(hunting): authenticate will time out at 2131.194813s 2024-09-16T05:27:26.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907012a570 0x7f907012a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:26.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907012aeb0 0x7f90701ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:26.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907012a570 0x7f907012a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:41470/0 (socket says 172.21.15.114:41470) 2024-09-16T05:27:26.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 1 -- 172.21.15.114:0/1165229386 learned_addr learned my addr 172.21.15.114:0/1165229386 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:26.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 10 monclient(hunting): get_auth_request con 0x7f907012a570 auth_method 0 2024-09-16T05:27:26.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:26.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:26.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:26.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906f7fe640 10 monclient(hunting): get_auth_request con 0x7f907012aeb0 auth_method 0 2024-09-16T05:27:26.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:26.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906effd640 1 --2- 172.21.15.114:0/1165229386 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9058004810 0x7f90700730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906effd640 10 monclient(hunting): get_auth_request con 0x7f9058004810 auth_method 0 2024-09-16T05:27:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 10 monclient(hunting): handle_auth_done global_id 104480 payload 824 2024-09-16T05:27:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 1 -- 172.21.15.114:0/1165229386 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9058004810 msgr2=0x7f90700730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 1 --2- 172.21.15.114:0/1165229386 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9058004810 0x7f90700730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 1 -- 172.21.15.114:0/1165229386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907012aeb0 msgr2=0x7f90701ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:26.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 1 --2- 172.21.15.114:0/1165229386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907012aeb0 0x7f90701ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:27:26.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 1 monclient: found mon.noname-c 2024-09-16T05:27:26.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:26.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.114:3301/0 2024-09-16T05:27:26.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 1 -- 172.21.15.114:0/1165229386 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907012bd20 con 0x7f907012a570 2024-09-16T05:27:26.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906e7fc640 1 --2- 172.21.15.114:0/1165229386 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907012a570 0x7f907012a970 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9064005040 tx=0x7f9058005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:26.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f90577fe640 10 client.?.objecter ms_handle_connect 0x7f907012a570 2024-09-16T05:27:26.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f90577fe640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:26.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f90577fe640 1 -- 172.21.15.114:0/1165229386 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90640b9020 con 0x7f907012a570 2024-09-16T05:27:26.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f90577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:26.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f90577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:27:26.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f90577fe640 10 monclient: dump: 2024-09-16T05:27:26.584 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:26.584 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:26.584 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.584 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.584 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:26.584 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:26.584 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:26.584 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:26.584 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:26.584 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:26.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f90577fe640 10 monclient: _finish_auth 0 2024-09-16T05:27:26.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f90577fe640 10 monclient: _check_auth_tickets 2024-09-16T05:27:26.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f90577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f90577fe640 1 -- 172.21.15.114:0/1165229386 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9064007040 con 0x7f907012a570 2024-09-16T05:27:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f907541f640 5 monclient: authenticate success, global_id 104480 2024-09-16T05:27:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f90577fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f90577fe640 1 -- 172.21.15.114:0/1165229386 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9064007410 con 0x7f907012a570 2024-09-16T05:27:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f90577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f90577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:27:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f90577fe640 10 monclient: dump: 2024-09-16T05:27:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:26.586 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:26.586 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:26.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.562+0000 7f906d7fa640 4 set_mon_vals no callback set 2024-09-16T05:27:26.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f907541f640 10 monclient: _renew_subs 2024-09-16T05:27:26.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f907541f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:26.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f907541f640 1 -- 172.21.15.114:0/1165229386 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f907009c6a0 con 0x7f907012a570 2024-09-16T05:27:26.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f907541f640 10 client.104480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:26.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f907541f640 10 monclient: _renew_subs 2024-09-16T05:27:26.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f907541f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:26.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f907541f640 1 -- 172.21.15.114:0/1165229386 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f90700a32b0 con 0x7f907012a570 2024-09-16T05:27:26.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f907541f640 1 librados: init done 2024-09-16T05:27:26.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f90577fe640 1 -- 172.21.15.114:0/1165229386 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f90640bc020 con 0x7f907012a570 2024-09-16T05:27:26.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f907541f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:26.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f907541f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:26.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f907541f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.114:3301/0 2024-09-16T05:27:26.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f907541f640 1 -- 172.21.15.114:0/1165229386 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f907009e4a0 con 0x7f907012a570 2024-09-16T05:27:26.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f90577fe640 1 --2- 172.21.15.114:0/1165229386 >> v2:172.21.15.149:6812/1185531135 conn(0x7f905c037810 0x7f905c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f90577fe640 1 -- 172.21.15.114:0/1165229386 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7f90640e30c0 con 0x7f907012a570 2024-09-16T05:27:26.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f90577fe640 10 client.104480.objecter ms_dispatch 0x7f9070000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:26.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f90577fe640 3 client.104480.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:26.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f90577fe640 3 client.104480.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:26.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f90577fe640 20 client.104480.objecter dump_active .. 0 homeless 2024-09-16T05:27:26.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f906effd640 1 --2- 172.21.15.114:0/1165229386 >> v2:172.21.15.149:6812/1185531135 conn(0x7f905c037810 0x7f905c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:26.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f906effd640 10 monclient: get_auth_request con 0x7f905c037810 auth_method 0 2024-09-16T05:27:26.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f90577fe640 10 client.104480.objecter ms_handle_connect 0x7f905c037810 2024-09-16T05:27:26.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.566+0000 7f906effd640 1 --2- 172.21.15.114:0/1165229386 >> v2:172.21.15.149:6812/1185531135 conn(0x7f905c037810 0x7f905c039cd0 secure :-1 s=READY pgs=3646 cs=0 l=1 rev1=1 crypto rx=0x7f906000ba30 tx=0x7f9060008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:26.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.570+0000 7f90577fe640 1 -- 172.21.15.114:0/1165229386 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f90640c3030 con 0x7f907012a570 2024-09-16T05:27:26.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.570+0000 7f90577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:26.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.570+0000 7f90577fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:27:26.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.726+0000 7f907541f640 1 -- 172.21.15.114:0/1165229386 --> v2:172.21.15.149:6812/1185531135 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f907012b670 con 0x7f905c037810 2024-09-16T05:27:26.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.734+0000 7f90577fe640 1 -- 172.21.15.114:0/1165229386 <== mgr.4108 v2:172.21.15.149:6812/1185531135 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114103 (secure 0 0 0) 0x7f9060004030 con 0x7f905c037810 2024-09-16T05:27:26.735 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T05:27:26.736 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-16T05:27:26.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 10 librados: watch_flush enter 2024-09-16T05:27:26.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 10 librados: watch_flush exit 2024-09-16T05:27:26.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 20 client.104480.objecter shutdown clearing up homeless session... 2024-09-16T05:27:26.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 10 client.104480.objecter successfully canceled tick 2024-09-16T05:27:26.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 1 -- 172.21.15.114:0/1165229386 >> v2:172.21.15.149:6812/1185531135 conn(0x7f905c037810 msgr2=0x7f905c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:26.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 1 --2- 172.21.15.114:0/1165229386 >> v2:172.21.15.149:6812/1185531135 conn(0x7f905c037810 0x7f905c039cd0 secure :-1 s=READY pgs=3646 cs=0 l=1 rev1=1 crypto rx=0x7f906000ba30 tx=0x7f9060008040 comp rx=0 tx=0).stop 2024-09-16T05:27:26.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 10 monclient: shutdown 2024-09-16T05:27:26.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:26.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 1 -- 172.21.15.114:0/1165229386 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907012a570 msgr2=0x7f907012a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:26.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 1 --2- 172.21.15.114:0/1165229386 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907012a570 0x7f907012a970 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9064005040 tx=0x7f9058005840 comp rx=0 tx=0).stop 2024-09-16T05:27:26.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 1 -- 172.21.15.114:0/1165229386 shutdown_connections 2024-09-16T05:27:26.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 1 --2- 172.21.15.114:0/1165229386 >> v2:172.21.15.149:6812/1185531135 conn(0x7f905c037810 0x7f905c039cd0 unknown :-1 s=CLOSED pgs=3646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 1 --2- 172.21.15.114:0/1165229386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f907012aeb0 0x7f90701ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 1 --2- 172.21.15.114:0/1165229386 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f907012a570 0x7f907012a970 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 1 --2- 172.21.15.114:0/1165229386 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9058004810 0x7f90700730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 1 -- 172.21.15.114:0/1165229386 >> 172.21.15.114:0/1165229386 conn(0x7f90700a1fb0 msgr2=0x7f9070138b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:26.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 1 -- 172.21.15.114:0/1165229386 shutdown_connections 2024-09-16T05:27:26.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 1 -- 172.21.15.114:0/1165229386 wait complete. 2024-09-16T05:27:26.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.738+0000 7f907541f640 1 librados: shutdown 2024-09-16T05:27:26.752 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1279,"stamp":"2024-09-16T05:27:26.520688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":464376,"kb_used_data":60696,"kb_used_omap":9,"kb_used_meta":403574,"kb_avail":565766664,"statfs":{"total":579820584960,"available":579345063936,"internally_reserved":0,"allocated":62152704,"data_stored":32273216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":413260449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003948"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":1330,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:11.666651+0000","last_change":"2024-09-16T05:27:11.666760+0000","last_active":"2024-09-16T05:27:11.666651+0000","last_peered":"2024-09-16T05:27:11.666651+0000","last_clean":"2024-09-16T05:27:11.666651+0000","last_became_active":"2024-09-16T05:08:51.571632+0000","last_became_peered":"2024-09-16T05:08:51.571632+0000","last_unstale":"2024-09-16T05:27:11.666651+0000","last_undegraded":"2024-09-16T05:27:11.666651+0000","last_fullsized":"2024-09-16T05:27:11.666651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:11.666453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:11.666453+0000","last_clean_scrub_stamp":"2024-09-16T05:27:11.666453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:29.097682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1333,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:08.548552+0000","last_change":"2024-09-16T05:27:08.548654+0000","last_active":"2024-09-16T05:27:08.548552+0000","last_peered":"2024-09-16T05:27:08.548552+0000","last_clean":"2024-09-16T05:27:08.548552+0000","last_became_active":"2024-09-16T05:08:51.585360+0000","last_became_peered":"2024-09-16T05:08:51.585360+0000","last_unstale":"2024-09-16T05:27:08.548552+0000","last_undegraded":"2024-09-16T05:27:08.548552+0000","last_fullsized":"2024-09-16T05:27:08.548552+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:08.548339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:08.548339+0000","last_clean_scrub_stamp":"2024-09-16T05:27:08.548339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:33:49.984858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1330,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:09.686218+0000","last_change":"2024-09-16T05:27:09.686272+0000","last_active":"2024-09-16T05:27:09.686218+0000","last_peered":"2024-09-16T05:27:09.686218+0000","last_clean":"2024-09-16T05:27:09.686218+0000","last_became_active":"2024-09-16T05:08:51.519558+0000","last_became_peered":"2024-09-16T05:08:51.519558+0000","last_unstale":"2024-09-16T05:27:09.686218+0000","last_undegraded":"2024-09-16T05:27:09.686218+0000","last_fullsized":"2024-09-16T05:27:09.686218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:09.686047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:09.686047+0000","last_clean_scrub_stamp":"2024-09-16T05:27:09.686047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:57:55.355689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1325,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:07.163414+0000","last_change":"2024-09-16T05:27:07.163467+0000","last_active":"2024-09-16T05:27:07.163414+0000","last_peered":"2024-09-16T05:27:07.163414+0000","last_clean":"2024-09-16T05:27:07.163414+0000","last_became_active":"2024-09-16T05:08:51.689931+0000","last_became_peered":"2024-09-16T05:08:51.689931+0000","last_unstale":"2024-09-16T05:27:07.163414+0000","last_undegraded":"2024-09-16T05:27:07.163414+0000","last_fullsized":"2024-09-16T05:27:07.163414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:07.163237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:07.163237+0000","last_clean_scrub_stamp":"2024-09-16T05:27:07.163237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:45.945798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1338,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:10.620919+0000","last_change":"2024-09-16T05:27:10.620974+0000","last_active":"2024-09-16T05:27:10.620919+0000","last_peered":"2024-09-16T05:27:10.620919+0000","last_clean":"2024-09-16T05:27:10.620919+0000","last_became_active":"2024-09-16T05:08:51.546851+0000","last_became_peered":"2024-09-16T05:08:51.546851+0000","last_unstale":"2024-09-16T05:27:10.620919+0000","last_undegraded":"2024-09-16T05:27:10.620919+0000","last_fullsized":"2024-09-16T05:27:10.620919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:10.620754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:10.620754+0000","last_clean_scrub_stamp":"2024-09-16T05:27:10.620754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:34:48.547567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1355,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:11.549103+0000","last_change":"2024-09-16T05:27:11.549157+0000","last_active":"2024-09-16T05:27:11.549103+0000","last_peered":"2024-09-16T05:27:11.549103+0000","last_clean":"2024-09-16T05:27:11.549103+0000","last_became_active":"2024-09-16T05:08:51.605843+0000","last_became_peered":"2024-09-16T05:08:51.605843+0000","last_unstale":"2024-09-16T05:27:11.549103+0000","last_undegraded":"2024-09-16T05:27:11.549103+0000","last_fullsized":"2024-09-16T05:27:11.549103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:11.548923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:11.548923+0000","last_clean_scrub_stamp":"2024-09-16T05:27:11.548923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:13.860989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1326,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:06.140517+0000","last_change":"2024-09-16T05:27:06.140543+0000","last_active":"2024-09-16T05:27:06.140517+0000","last_peered":"2024-09-16T05:27:06.140517+0000","last_clean":"2024-09-16T05:27:06.140517+0000","last_became_active":"2024-09-16T05:08:51.609820+0000","last_became_peered":"2024-09-16T05:08:51.609820+0000","last_unstale":"2024-09-16T05:27:06.140517+0000","last_undegraded":"2024-09-16T05:27:06.140517+0000","last_fullsized":"2024-09-16T05:27:06.140517+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:06.140425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:06.140425+0000","last_clean_scrub_stamp":"2024-09-16T05:27:06.140425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:50:42.840374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1331,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:12.684918+0000","last_change":"2024-09-16T05:27:12.684969+0000","last_active":"2024-09-16T05:27:12.684918+0000","last_peered":"2024-09-16T05:27:12.684918+0000","last_clean":"2024-09-16T05:27:12.684918+0000","last_became_active":"2024-09-16T05:08:51.567268+0000","last_became_peered":"2024-09-16T05:08:51.567268+0000","last_unstale":"2024-09-16T05:27:12.684918+0000","last_undegraded":"2024-09-16T05:27:12.684918+0000","last_fullsized":"2024-09-16T05:27:12.684918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:12.684738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:12.684738+0000","last_clean_scrub_stamp":"2024-09-16T05:27:12.684738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:30.347933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1334,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:12.649379+0000","last_change":"2024-09-16T05:27:12.649454+0000","last_active":"2024-09-16T05:27:12.649379+0000","last_peered":"2024-09-16T05:27:12.649379+0000","last_clean":"2024-09-16T05:27:12.649379+0000","last_became_active":"2024-09-16T05:08:51.634768+0000","last_became_peered":"2024-09-16T05:08:51.634768+0000","last_unstale":"2024-09-16T05:27:12.649379+0000","last_undegraded":"2024-09-16T05:27:12.649379+0000","last_fullsized":"2024-09-16T05:27:12.649379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:12.649175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:12.649175+0000","last_clean_scrub_stamp":"2024-09-16T05:27:12.649175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:27:48.363639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1332,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:08.465533+0000","last_change":"2024-09-16T05:27:08.465590+0000","last_active":"2024-09-16T05:27:08.465533+0000","last_peered":"2024-09-16T05:27:08.465533+0000","last_clean":"2024-09-16T05:27:08.465533+0000","last_became_active":"2024-09-16T05:08:51.589818+0000","last_became_peered":"2024-09-16T05:08:51.589818+0000","last_unstale":"2024-09-16T05:27:08.465533+0000","last_undegraded":"2024-09-16T05:27:08.465533+0000","last_fullsized":"2024-09-16T05:27:08.465533+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:08.465356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:08.465356+0000","last_clean_scrub_stamp":"2024-09-16T05:27:08.465356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:35.744016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1355,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:10.595669+0000","last_change":"2024-09-16T05:27:10.595731+0000","last_active":"2024-09-16T05:27:10.595669+0000","last_peered":"2024-09-16T05:27:10.595669+0000","last_clean":"2024-09-16T05:27:10.595669+0000","last_became_active":"2024-09-16T05:08:51.605719+0000","last_became_peered":"2024-09-16T05:08:51.605719+0000","last_unstale":"2024-09-16T05:27:10.595669+0000","last_undegraded":"2024-09-16T05:27:10.595669+0000","last_fullsized":"2024-09-16T05:27:10.595669+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:10.595494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:10.595494+0000","last_clean_scrub_stamp":"2024-09-16T05:27:10.595494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:59:12.848098+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1334,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:10.563464+0000","last_change":"2024-09-16T05:27:10.563517+0000","last_active":"2024-09-16T05:27:10.563464+0000","last_peered":"2024-09-16T05:27:10.563464+0000","last_clean":"2024-09-16T05:27:10.563464+0000","last_became_active":"2024-09-16T05:08:51.611801+0000","last_became_peered":"2024-09-16T05:08:51.611801+0000","last_unstale":"2024-09-16T05:27:10.563464+0000","last_undegraded":"2024-09-16T05:27:10.563464+0000","last_fullsized":"2024-09-16T05:27:10.563464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:10.563286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:10.563286+0000","last_clean_scrub_stamp":"2024-09-16T05:27:10.563286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:30.171730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1338,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:11.670991+0000","last_change":"2024-09-16T05:27:11.671044+0000","last_active":"2024-09-16T05:27:11.670991+0000","last_peered":"2024-09-16T05:27:11.670991+0000","last_clean":"2024-09-16T05:27:11.670991+0000","last_became_active":"2024-09-16T05:08:51.586611+0000","last_became_peered":"2024-09-16T05:08:51.586611+0000","last_unstale":"2024-09-16T05:27:11.670991+0000","last_undegraded":"2024-09-16T05:27:11.670991+0000","last_fullsized":"2024-09-16T05:27:11.670991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:11.670817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:11.670817+0000","last_clean_scrub_stamp":"2024-09-16T05:27:11.670817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:55.469533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"695'119","reported_seq":1503,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:04.596839+0000","last_change":"2024-09-16T05:27:04.596971+0000","last_active":"2024-09-16T05:27:04.596839+0000","last_peered":"2024-09-16T05:27:04.596839+0000","last_clean":"2024-09-16T05:27:04.596839+0000","last_became_active":"2024-09-16T05:07:39.576756+0000","last_became_peered":"2024-09-16T05:07:39.576756+0000","last_unstale":"2024-09-16T05:27:04.596839+0000","last_undegraded":"2024-09-16T05:27:04.596839+0000","last_fullsized":"2024-09-16T05:27:04.596839+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'119","last_scrub_stamp":"2024-09-16T05:27:04.596642+0000","last_deep_scrub":"695'119","last_deep_scrub_stamp":"2024-09-16T05:27:04.596642+0000","last_clean_scrub_stamp":"2024-09-16T05:27:04.596642+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:08:29.408891+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"695'2","reported_seq":1352,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:06.524616+0000","last_change":"2024-09-16T05:27:06.524735+0000","last_active":"2024-09-16T05:27:06.524616+0000","last_peered":"2024-09-16T05:27:06.524616+0000","last_clean":"2024-09-16T05:27:06.524616+0000","last_became_active":"2024-09-16T05:07:46.246409+0000","last_became_peered":"2024-09-16T05:07:46.246409+0000","last_unstale":"2024-09-16T05:27:06.524616+0000","last_undegraded":"2024-09-16T05:27:06.524616+0000","last_fullsized":"2024-09-16T05:27:06.524616+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'2","last_scrub_stamp":"2024-09-16T05:27:06.524429+0000","last_deep_scrub":"695'2","last_deep_scrub_stamp":"2024-09-16T05:27:06.524429+0000","last_clean_scrub_stamp":"2024-09-16T05:27:06.524429+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:31.370317+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0010413149999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1332,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:07.462574+0000","last_change":"2024-09-16T05:27:07.462628+0000","last_active":"2024-09-16T05:27:07.462574+0000","last_peered":"2024-09-16T05:27:07.462574+0000","last_clean":"2024-09-16T05:27:07.462574+0000","last_became_active":"2024-09-16T05:08:51.582424+0000","last_became_peered":"2024-09-16T05:08:51.582424+0000","last_unstale":"2024-09-16T05:27:07.462574+0000","last_undegraded":"2024-09-16T05:27:07.462574+0000","last_fullsized":"2024-09-16T05:27:07.462574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:07.462385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:07.462385+0000","last_clean_scrub_stamp":"2024-09-16T05:27:07.462385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:55:04.534616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1342,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:05.162117+0000","last_change":"2024-09-16T05:27:05.162167+0000","last_active":"2024-09-16T05:27:05.162117+0000","last_peered":"2024-09-16T05:27:05.162117+0000","last_clean":"2024-09-16T05:27:05.162117+0000","last_became_active":"2024-09-16T05:07:46.254322+0000","last_became_peered":"2024-09-16T05:07:46.254322+0000","last_unstale":"2024-09-16T05:27:05.162117+0000","last_undegraded":"2024-09-16T05:27:05.162117+0000","last_fullsized":"2024-09-16T05:27:05.162117+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:05.161932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:05.161932+0000","last_clean_scrub_stamp":"2024-09-16T05:27:05.161932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:48.796114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00093680599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1338,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:09.652025+0000","last_change":"2024-09-16T05:27:09.652081+0000","last_active":"2024-09-16T05:27:09.652025+0000","last_peered":"2024-09-16T05:27:09.652025+0000","last_clean":"2024-09-16T05:27:09.652025+0000","last_became_active":"2024-09-16T05:08:51.539573+0000","last_became_peered":"2024-09-16T05:08:51.539573+0000","last_unstale":"2024-09-16T05:27:09.652025+0000","last_undegraded":"2024-09-16T05:27:09.652025+0000","last_fullsized":"2024-09-16T05:27:09.652025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:09.651848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:09.651848+0000","last_clean_scrub_stamp":"2024-09-16T05:27:09.651848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:36.521282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1371,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:05.605489+0000","last_change":"2024-09-16T05:27:05.605544+0000","last_active":"2024-09-16T05:27:05.605489+0000","last_peered":"2024-09-16T05:27:05.605489+0000","last_clean":"2024-09-16T05:27:05.605489+0000","last_became_active":"2024-09-16T05:07:46.252254+0000","last_became_peered":"2024-09-16T05:07:46.252254+0000","last_unstale":"2024-09-16T05:27:05.605489+0000","last_undegraded":"2024-09-16T05:27:05.605489+0000","last_fullsized":"2024-09-16T05:27:05.605489+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:05.605319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:05.605319+0000","last_clean_scrub_stamp":"2024-09-16T05:27:05.605319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:44.945727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081560699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1355,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:08.600005+0000","last_change":"2024-09-16T05:27:08.600056+0000","last_active":"2024-09-16T05:27:08.600005+0000","last_peered":"2024-09-16T05:27:08.600005+0000","last_clean":"2024-09-16T05:27:08.600005+0000","last_became_active":"2024-09-16T05:08:51.541976+0000","last_became_peered":"2024-09-16T05:08:51.541976+0000","last_unstale":"2024-09-16T05:27:08.600005+0000","last_undegraded":"2024-09-16T05:27:08.600005+0000","last_fullsized":"2024-09-16T05:27:08.600005+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:08.599837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:08.599837+0000","last_clean_scrub_stamp":"2024-09-16T05:27:08.599837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:33.657070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"695'3","reported_seq":1357,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:04.599315+0000","last_change":"2024-09-16T05:27:04.599358+0000","last_active":"2024-09-16T05:27:04.599315+0000","last_peered":"2024-09-16T05:27:04.599315+0000","last_clean":"2024-09-16T05:27:04.599315+0000","last_became_active":"2024-09-16T05:07:46.257118+0000","last_became_peered":"2024-09-16T05:07:46.257118+0000","last_unstale":"2024-09-16T05:27:04.599315+0000","last_undegraded":"2024-09-16T05:27:04.599315+0000","last_fullsized":"2024-09-16T05:27:04.599315+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'3","last_scrub_stamp":"2024-09-16T05:27:04.599222+0000","last_deep_scrub":"695'3","last_deep_scrub_stamp":"2024-09-16T05:27:04.599222+0000","last_clean_scrub_stamp":"2024-09-16T05:27:04.599222+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:35:11.809119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0020216650000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1332,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:08.664451+0000","last_change":"2024-09-16T05:27:08.664551+0000","last_active":"2024-09-16T05:27:08.664451+0000","last_peered":"2024-09-16T05:27:08.664451+0000","last_clean":"2024-09-16T05:27:08.664451+0000","last_became_active":"2024-09-16T05:08:51.491760+0000","last_became_peered":"2024-09-16T05:08:51.491760+0000","last_unstale":"2024-09-16T05:27:08.664451+0000","last_undegraded":"2024-09-16T05:27:08.664451+0000","last_fullsized":"2024-09-16T05:27:08.664451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:08.664241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:08.664241+0000","last_clean_scrub_stamp":"2024-09-16T05:27:08.664241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:46.407311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1348,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:06.486489+0000","last_change":"2024-09-16T05:27:06.486547+0000","last_active":"2024-09-16T05:27:06.486489+0000","last_peered":"2024-09-16T05:27:06.486489+0000","last_clean":"2024-09-16T05:27:06.486489+0000","last_became_active":"2024-09-16T05:07:46.255633+0000","last_became_peered":"2024-09-16T05:07:46.255633+0000","last_unstale":"2024-09-16T05:27:06.486489+0000","last_undegraded":"2024-09-16T05:27:06.486489+0000","last_fullsized":"2024-09-16T05:27:06.486489+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:06.486306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:06.486306+0000","last_clean_scrub_stamp":"2024-09-16T05:27:06.486306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:41:26.672894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000883379,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"695'8","reported_seq":1369,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:12.552063+0000","last_change":"2024-09-16T05:27:12.552129+0000","last_active":"2024-09-16T05:27:12.552063+0000","last_peered":"2024-09-16T05:27:12.552063+0000","last_clean":"2024-09-16T05:27:12.552063+0000","last_became_active":"2024-09-16T05:08:51.614284+0000","last_became_peered":"2024-09-16T05:08:51.614284+0000","last_unstale":"2024-09-16T05:27:12.552063+0000","last_undegraded":"2024-09-16T05:27:12.552063+0000","last_fullsized":"2024-09-16T05:27:12.552063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'8","last_scrub_stamp":"2024-09-16T05:27:12.551892+0000","last_deep_scrub":"695'8","last_deep_scrub_stamp":"2024-09-16T05:27:12.551892+0000","last_clean_scrub_stamp":"2024-09-16T05:27:12.551892+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:27.630830+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1350,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:05.553076+0000","last_change":"2024-09-16T05:27:05.553212+0000","last_active":"2024-09-16T05:27:05.553076+0000","last_peered":"2024-09-16T05:27:05.553076+0000","last_clean":"2024-09-16T05:27:05.553076+0000","last_became_active":"2024-09-16T05:07:46.243891+0000","last_became_peered":"2024-09-16T05:07:46.243891+0000","last_unstale":"2024-09-16T05:27:05.553076+0000","last_undegraded":"2024-09-16T05:27:05.553076+0000","last_fullsized":"2024-09-16T05:27:05.553076+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:05.552875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:05.552875+0000","last_clean_scrub_stamp":"2024-09-16T05:27:05.552875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:36:45.849741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011829709999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1334,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:11.609257+0000","last_change":"2024-09-16T05:27:11.609364+0000","last_active":"2024-09-16T05:27:11.609257+0000","last_peered":"2024-09-16T05:27:11.609257+0000","last_clean":"2024-09-16T05:27:11.609257+0000","last_became_active":"2024-09-16T05:08:51.632213+0000","last_became_peered":"2024-09-16T05:08:51.632213+0000","last_unstale":"2024-09-16T05:27:11.609257+0000","last_undegraded":"2024-09-16T05:27:11.609257+0000","last_fullsized":"2024-09-16T05:27:11.609257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:11.609025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:11.609025+0000","last_clean_scrub_stamp":"2024-09-16T05:27:11.609025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:39:07.768534+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1347,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:05.752864+0000","last_change":"2024-09-16T05:27:05.752864+0000","last_active":"2024-09-16T05:27:05.752864+0000","last_peered":"2024-09-16T05:27:05.752864+0000","last_clean":"2024-09-16T05:27:05.752864+0000","last_became_active":"2024-09-16T05:07:46.252207+0000","last_became_peered":"2024-09-16T05:07:46.252207+0000","last_unstale":"2024-09-16T05:27:05.752864+0000","last_undegraded":"2024-09-16T05:27:05.752864+0000","last_fullsized":"2024-09-16T05:27:05.752864+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:05.752629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:05.752629+0000","last_clean_scrub_stamp":"2024-09-16T05:27:05.752629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:09.489884+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00073333700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1338,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:06.601448+0000","last_change":"2024-09-16T05:27:06.601497+0000","last_active":"2024-09-16T05:27:06.601448+0000","last_peered":"2024-09-16T05:27:06.601448+0000","last_clean":"2024-09-16T05:27:06.601448+0000","last_became_active":"2024-09-16T05:08:51.486807+0000","last_became_peered":"2024-09-16T05:08:51.486807+0000","last_unstale":"2024-09-16T05:27:06.601448+0000","last_undegraded":"2024-09-16T05:27:06.601448+0000","last_fullsized":"2024-09-16T05:27:06.601448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:06.601294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:06.601294+0000","last_clean_scrub_stamp":"2024-09-16T05:27:06.601294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:45:59.609390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1350,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:07.558472+0000","last_change":"2024-09-16T05:27:07.558472+0000","last_active":"2024-09-16T05:27:07.558472+0000","last_peered":"2024-09-16T05:27:07.558472+0000","last_clean":"2024-09-16T05:27:07.558472+0000","last_became_active":"2024-09-16T05:07:46.254034+0000","last_became_peered":"2024-09-16T05:07:46.254034+0000","last_unstale":"2024-09-16T05:27:07.558472+0000","last_undegraded":"2024-09-16T05:27:07.558472+0000","last_fullsized":"2024-09-16T05:27:07.558472+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:07.558257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:07.558257+0000","last_clean_scrub_stamp":"2024-09-16T05:27:07.558257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:48.381613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0014490219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1330,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:10.664887+0000","last_change":"2024-09-16T05:27:10.664940+0000","last_active":"2024-09-16T05:27:10.664887+0000","last_peered":"2024-09-16T05:27:10.664887+0000","last_clean":"2024-09-16T05:27:10.664887+0000","last_became_active":"2024-09-16T05:08:51.532685+0000","last_became_peered":"2024-09-16T05:08:51.532685+0000","last_unstale":"2024-09-16T05:27:10.664887+0000","last_undegraded":"2024-09-16T05:27:10.664887+0000","last_fullsized":"2024-09-16T05:27:10.664887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:10.664698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:10.664698+0000","last_clean_scrub_stamp":"2024-09-16T05:27:10.664698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:32:15.276930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1330,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:13.718044+0000","last_change":"2024-09-16T05:27:13.718102+0000","last_active":"2024-09-16T05:27:13.718044+0000","last_peered":"2024-09-16T05:27:13.718044+0000","last_clean":"2024-09-16T05:27:13.718044+0000","last_became_active":"2024-09-16T05:08:51.567861+0000","last_became_peered":"2024-09-16T05:08:51.567861+0000","last_unstale":"2024-09-16T05:27:13.718044+0000","last_undegraded":"2024-09-16T05:27:13.718044+0000","last_fullsized":"2024-09-16T05:27:13.718044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:13.717873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:13.717873+0000","last_clean_scrub_stamp":"2024-09-16T05:27:13.717873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:14.808411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1354,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:09.572153+0000","last_change":"2024-09-16T05:27:09.572211+0000","last_active":"2024-09-16T05:27:09.572153+0000","last_peered":"2024-09-16T05:27:09.572153+0000","last_clean":"2024-09-16T05:27:09.572153+0000","last_became_active":"2024-09-16T05:08:51.563420+0000","last_became_peered":"2024-09-16T05:08:51.563420+0000","last_unstale":"2024-09-16T05:27:09.572153+0000","last_undegraded":"2024-09-16T05:27:09.572153+0000","last_fullsized":"2024-09-16T05:27:09.572153+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:09.571976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:09.571976+0000","last_clean_scrub_stamp":"2024-09-16T05:27:09.571976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:19:36.138034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1337,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:08.666126+0000","last_change":"2024-09-16T05:27:08.666225+0000","last_active":"2024-09-16T05:27:08.666126+0000","last_peered":"2024-09-16T05:27:08.666126+0000","last_clean":"2024-09-16T05:27:08.666126+0000","last_became_active":"2024-09-16T05:08:51.503720+0000","last_became_peered":"2024-09-16T05:08:51.503720+0000","last_unstale":"2024-09-16T05:27:08.666126+0000","last_undegraded":"2024-09-16T05:27:08.666126+0000","last_fullsized":"2024-09-16T05:27:08.666126+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:08.665923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:08.665923+0000","last_clean_scrub_stamp":"2024-09-16T05:27:08.665923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:51:30.006539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1330,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:07.713436+0000","last_change":"2024-09-16T05:27:07.713531+0000","last_active":"2024-09-16T05:27:07.713436+0000","last_peered":"2024-09-16T05:27:07.713436+0000","last_clean":"2024-09-16T05:27:07.713436+0000","last_became_active":"2024-09-16T05:08:51.494038+0000","last_became_peered":"2024-09-16T05:08:51.494038+0000","last_unstale":"2024-09-16T05:27:07.713436+0000","last_undegraded":"2024-09-16T05:27:07.713436+0000","last_fullsized":"2024-09-16T05:27:07.713436+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:07.713233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:07.713233+0000","last_clean_scrub_stamp":"2024-09-16T05:27:07.713233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:22:46.699878+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1333,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:13.639409+0000","last_change":"2024-09-16T05:27:13.639463+0000","last_active":"2024-09-16T05:27:13.639409+0000","last_peered":"2024-09-16T05:27:13.639409+0000","last_clean":"2024-09-16T05:27:13.639409+0000","last_became_active":"2024-09-16T05:08:51.680588+0000","last_became_peered":"2024-09-16T05:08:51.680588+0000","last_unstale":"2024-09-16T05:27:13.639409+0000","last_undegraded":"2024-09-16T05:27:13.639409+0000","last_fullsized":"2024-09-16T05:27:13.639409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:13.639225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:13.639225+0000","last_clean_scrub_stamp":"2024-09-16T05:27:13.639225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:39.426771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1355,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:06.628102+0000","last_change":"2024-09-16T05:27:06.628154+0000","last_active":"2024-09-16T05:27:06.628102+0000","last_peered":"2024-09-16T05:27:06.628102+0000","last_clean":"2024-09-16T05:27:06.628102+0000","last_became_active":"2024-09-16T05:08:51.480748+0000","last_became_peered":"2024-09-16T05:08:51.480748+0000","last_unstale":"2024-09-16T05:27:06.628102+0000","last_undegraded":"2024-09-16T05:27:06.628102+0000","last_fullsized":"2024-09-16T05:27:06.628102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:06.627934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:06.627934+0000","last_clean_scrub_stamp":"2024-09-16T05:27:06.627934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:42.927058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1334,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:09.513847+0000","last_change":"2024-09-16T05:27:09.513947+0000","last_active":"2024-09-16T05:27:09.513847+0000","last_peered":"2024-09-16T05:27:09.513847+0000","last_clean":"2024-09-16T05:27:09.513847+0000","last_became_active":"2024-09-16T05:08:51.622941+0000","last_became_peered":"2024-09-16T05:08:51.622941+0000","last_unstale":"2024-09-16T05:27:09.513847+0000","last_undegraded":"2024-09-16T05:27:09.513847+0000","last_fullsized":"2024-09-16T05:27:09.513847+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:09.513643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:09.513643+0000","last_clean_scrub_stamp":"2024-09-16T05:27:09.513643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:48:38.513108+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1337,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:05.603144+0000","last_change":"2024-09-16T05:27:05.603196+0000","last_active":"2024-09-16T05:27:05.603144+0000","last_peered":"2024-09-16T05:27:05.603144+0000","last_clean":"2024-09-16T05:27:05.603144+0000","last_became_active":"2024-09-16T05:08:51.486048+0000","last_became_peered":"2024-09-16T05:08:51.486048+0000","last_unstale":"2024-09-16T05:27:05.603144+0000","last_undegraded":"2024-09-16T05:27:05.603144+0000","last_fullsized":"2024-09-16T05:27:05.603144+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:05.602979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:05.602979+0000","last_clean_scrub_stamp":"2024-09-16T05:27:05.602979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:36.861491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1330,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:06.720575+0000","last_change":"2024-09-16T05:27:06.720633+0000","last_active":"2024-09-16T05:27:06.720575+0000","last_peered":"2024-09-16T05:27:06.720575+0000","last_clean":"2024-09-16T05:27:06.720575+0000","last_became_active":"2024-09-16T05:08:51.469556+0000","last_became_peered":"2024-09-16T05:08:51.469556+0000","last_unstale":"2024-09-16T05:27:06.720575+0000","last_undegraded":"2024-09-16T05:27:06.720575+0000","last_fullsized":"2024-09-16T05:27:06.720575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:06.720409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:06.720409+0000","last_clean_scrub_stamp":"2024-09-16T05:27:06.720409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:37:01.335465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1354,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:07.591265+0000","last_change":"2024-09-16T05:27:07.591318+0000","last_active":"2024-09-16T05:27:07.591265+0000","last_peered":"2024-09-16T05:27:07.591265+0000","last_clean":"2024-09-16T05:27:07.591265+0000","last_became_active":"2024-09-16T05:08:51.533473+0000","last_became_peered":"2024-09-16T05:08:51.533473+0000","last_unstale":"2024-09-16T05:27:07.591265+0000","last_undegraded":"2024-09-16T05:27:07.591265+0000","last_fullsized":"2024-09-16T05:27:07.591265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:07.591050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:07.591050+0000","last_clean_scrub_stamp":"2024-09-16T05:27:07.591050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:01.225741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1337,"reported_epoch":695,"state":"active+clean","last_fresh":"2024-09-16T05:27:07.642868+0000","last_change":"2024-09-16T05:27:07.642921+0000","last_active":"2024-09-16T05:27:07.642868+0000","last_peered":"2024-09-16T05:27:07.642868+0000","last_clean":"2024-09-16T05:27:07.642868+0000","last_became_active":"2024-09-16T05:08:51.493446+0000","last_became_peered":"2024-09-16T05:08:51.493446+0000","last_unstale":"2024-09-16T05:27:07.642868+0000","last_undegraded":"2024-09-16T05:27:07.642868+0000","last_fullsized":"2024-09-16T05:27:07.642868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:07.642678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:27:07.642678+0000","last_clean_scrub_stamp":"2024-09-16T05:27:07.642678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:25.566474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834575089,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73060,"kb_used_data":9940,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298780,"statfs":{"total":96636764160,"available":96561950720,"internally_reserved":0,"allocated":10178560,"data_stored":5202552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575089,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77224,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294616,"statfs":{"total":96636764160,"available":96557686784,"internally_reserved":0,"allocated":10182656,"data_stored":5202571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607792,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73580,"kb_used_data":10460,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298260,"statfs":{"total":96636764160,"available":96561418240,"internally_reserved":0,"allocated":10711040,"data_stored":5731480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607792,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73584,"kb_used_data":10464,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298256,"statfs":{"total":96636764160,"available":96561414144,"internally_reserved":0,"allocated":10715136,"data_stored":5731485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640497,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77224,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294616,"statfs":{"total":96636764160,"available":96557686784,"internally_reserved":0,"allocated":10182656,"data_stored":5202571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640497,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89704,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94282136,"statfs":{"total":96636764160,"available":96544907264,"internally_reserved":0,"allocated":10182656,"data_stored":5202557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81656263},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:27:26.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T05:27:26.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.878+0000 7fb5922bc640 10 monclient: get_monmap_and_config 2024-09-16T05:27:26.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.878+0000 7fb5922bc640 10 monclient: build_initial_monmap 2024-09-16T05:27:26.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.878+0000 7fb5922bc640 10 monclient: monmap: 2024-09-16T05:27:26.878 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:26.878 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:26.878 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:27:26.883171+0000 2024-09-16T05:27:26.878 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:27:26.883171+0000 2024-09-16T05:27:26.879 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:26.879 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:26.879 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a-legacy 2024-09-16T05:27:26.879 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.149:6789/0 mon.noname-c-legacy 2024-09-16T05:27:26.879 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-b 2024-09-16T05:27:26.879 INFO:teuthology.orchestra.run.smithi114.stderr:3: v2:172.21.15.114:3300/0 mon.noname-a 2024-09-16T05:27:26.879 INFO:teuthology.orchestra.run.smithi114.stderr:4: v2:172.21.15.149:3300/0 mon.noname-c 2024-09-16T05:27:26.879 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:26.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.878+0000 7fb5922bc640 1 Processor -- start 2024-09-16T05:27:26.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.878+0000 7fb5922bc640 1 -- start start 2024-09-16T05:27:26.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.878+0000 7fb5922bc640 10 monclient: init 2024-09-16T05:27:26.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb5922bc640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:26.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb5922bc640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-16T05:27:26.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb5922bc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb58c0a0ec0 addr v1:172.21.15.114:6789/0 2024-09-16T05:27:26.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb5922bc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb58c09e1d0 addr v1:172.21.15.149:6789/0 2024-09-16T05:27:26.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb5922bc640 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb58c09e5e0 0x7fb58c13cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb5922bc640 10 monclient(hunting): picked mon.noname-c con 0x7fb58c09e5e0 addr v2:172.21.15.149:3300/0 2024-09-16T05:27:26.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb5922bc640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb58c09a760 con 0x7fb58c0a0ec0 2024-09-16T05:27:26.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb5922bc640 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb58c0a1870 con 0x7fb58c09e1d0 2024-09-16T05:27:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb5922bc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb5922bc640 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 -- 0x7fb58c0a12a0 con 0x7fb58c09e5e0 2024-09-16T05:27:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb5922bc640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb5922bc640 10 monclient(hunting): authenticate will time out at 2131.512496s 2024-09-16T05:27:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb58c0a0ec0 0x7fb58c09dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56390/0 (socket says 172.21.15.114:56390) 2024-09-16T05:27:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 1 -- 172.21.15.114:0/138775651 learned_addr learned my addr 172.21.15.114:0/138775651 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-09-16T05:27:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 -- 172.21.15.114:0/138775651 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3649174400 0 0) 0x7fb58c09a760 con 0x7fb58c0a0ec0 2024-09-16T05:27:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient(hunting): _negotiate 2024-09-16T05:27:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient(hunting): my global_id is 134277 2024-09-16T05:27:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 -- 172.21.15.114:0/138775651 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb578003530 con 0x7fb58c0a0ec0 2024-09-16T05:27:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb590859640 1 --2- 172.21.15.114:0/138775651 >> v2:172.21.15.149:3300/0 conn(0x7fb58c09e5e0 0x7fb58c13cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:26.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb590859640 10 monclient(hunting): get_auth_request con 0x7fb58c09e5e0 auth_method 0 2024-09-16T05:27:26.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb590859640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:26.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb590859640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:26.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb590859640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:26.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb590859640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:26.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb590859640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:26.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 -- 172.21.15.114:0/138775651 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2773320154 0 0) 0x7fb578003530 con 0x7fb58c0a0ec0 2024-09-16T05:27:26.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb590859640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:26.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 -- 172.21.15.114:0/138775651 >> v2:172.21.15.149:3300/0 conn(0x7fb58c09e5e0 msgr2=0x7fb58c13cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:26.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 --2- 172.21.15.114:0/138775651 >> v2:172.21.15.149:3300/0 conn(0x7fb58c09e5e0 0x7fb58c13cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 -- 172.21.15.114:0/138775651 >> v1:172.21.15.149:6789/0 conn(0x7fb58c09e1d0 legacy=0x7fb58c139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:26.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient: _finish_hunting 0 2024-09-16T05:27:26.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 monclient: found mon.noname-a-legacy 2024-09-16T05:27:26.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:26.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.114:6789/0 2024-09-16T05:27:26.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 -- 172.21.15.114:0/138775651 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb58c099790 con 0x7fb58c0a0ec0 2024-09-16T05:27:26.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient: _finish_auth 0 2024-09-16T05:27:26.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient: _check_auth_tickets 2024-09-16T05:27:26.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:26.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 -- 172.21.15.114:0/138775651 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3344850406 0 0) 0x7fb57c001680 con 0x7fb58c0a0ec0 2024-09-16T05:27:26.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:26.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T05:27:26.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient: dump: 2024-09-16T05:27:26.893 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:26.893 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:26.894 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.894 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.894 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:26.894 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:26.894 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:26.894 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:26.894 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:26.894 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:26.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] but i'm connected to v1:172.21.15.114:6789/0, reconnecting 2024-09-16T05:27:26.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:26.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 -- 172.21.15.114:0/138775651 >> v1:172.21.15.114:6789/0 conn(0x7fb58c0a0ec0 legacy=0x7fb58c09dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:26.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:27:26.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 --2- 172.21.15.114:0/138775651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5780039c0 0x7fb578003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient(hunting): picked mon.a con 0x7fb5780039c0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:26.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 --2- 172.21.15.114:0/138775651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb578004810 0x7fb578004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient(hunting): picked mon.c con 0x7fb578004810 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:26.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 --2- 172.21.15.114:0/138775651 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5780051d0 0x7fb57800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient(hunting): picked mon.b con 0x7fb5780051d0 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:26.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:26.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 -- 172.21.15.114:0/138775651 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb58c0a12a0 con 0x7fb5780039c0 2024-09-16T05:27:26.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:26.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 -- 172.21.15.114:0/138775651 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb57800c170 con 0x7fb5780051d0 2024-09-16T05:27:26.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:26.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 1 -- 172.21.15.114:0/138775651 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb57800c370 con 0x7fb578004810 2024-09-16T05:27:26.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:26.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:27:26.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 1 --2- 172.21.15.114:0/138775651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5780039c0 0x7fb578003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:26.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58b7fe640 1 --2- 172.21.15.114:0/138775651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb578004810 0x7fb578004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:26.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 10 monclient(hunting): get_auth_request con 0x7fb5780039c0 auth_method 0 2024-09-16T05:27:26.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:26.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:26.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:27:26.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58b7fe640 10 monclient(hunting): get_auth_request con 0x7fb578004810 auth_method 0 2024-09-16T05:27:26.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:26.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:26.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:27:26.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:26.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:26.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:27:26.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:26.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:26.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T05:27:26.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 10 monclient(hunting): handle_auth_done global_id 134277 payload 293 2024-09-16T05:27:26.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 1 -- 172.21.15.114:0/138775651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb578004810 msgr2=0x7fb578004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:26.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 1 --2- 172.21.15.114:0/138775651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb578004810 0x7fb578004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 1 -- 172.21.15.114:0/138775651 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5780051d0 msgr2=0x7fb57800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:27:26.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 1 --2- 172.21.15.114:0/138775651 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5780051d0 0x7fb57800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:27:26.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 1 monclient: found mon.a 2024-09-16T05:27:26.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:26.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:26.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 1 -- 172.21.15.114:0/138775651 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb578003530 con 0x7fb5780039c0 2024-09-16T05:27:26.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 10 monclient: _finish_auth 0 2024-09-16T05:27:26.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb590859640 1 -- 172.21.15.114:0/138775651 reap_dead start 2024-09-16T05:27:26.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:27:26.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:26.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb5922bc640 5 monclient: authenticate success, global_id 134277 2024-09-16T05:27:26.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb5922bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:26.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.882+0000 7fb58b7fe640 1 --2- 172.21.15.114:0/138775651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb578004810 0x7fb578004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:27:26.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58bfff640 1 --2- 172.21.15.114:0/138775651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5780039c0 0x7fb578003de0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb57c00c240 tx=0x7fb57c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca56790ebd2d77 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:26.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58affd640 1 -- 172.21.15.114:0/138775651 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c007d20 con 0x7fb5780039c0 2024-09-16T05:27:26.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:26.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:26.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58affd640 10 monclient: dump: 2024-09-16T05:27:26.899 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:26.899 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:26.899 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.899 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.899 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:26.899 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:26.899 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:26.899 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:26.899 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:26.899 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:26.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58affd640 1 -- 172.21.15.114:0/138775651 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb57c0a9070 con 0x7fb5780039c0 2024-09-16T05:27:26.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:27:26.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:26.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58affd640 1 -- 172.21.15.114:0/138775651 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c005da0 con 0x7fb5780039c0 2024-09-16T05:27:26.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient: get_monmap_and_config success 2024-09-16T05:27:26.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 4 set_mon_vals no callback set 2024-09-16T05:27:26.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:26.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:26.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58affd640 10 monclient: dump: 2024-09-16T05:27:26.900 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:26.900 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:26.900 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.900 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.900 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:26.900 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:26.901 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:26.901 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:26.901 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:26.901 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:26.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient: shutdown 2024-09-16T05:27:26.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:26.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 -- 172.21.15.114:0/138775651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5780039c0 msgr2=0x7fb578003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:26.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 --2- 172.21.15.114:0/138775651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5780039c0 0x7fb578003de0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb57c00c240 tx=0x7fb57c01cc60 comp rx=0 tx=0).stop 2024-09-16T05:27:26.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 -- 172.21.15.114:0/138775651 shutdown_connections 2024-09-16T05:27:26.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 --2- 172.21.15.114:0/138775651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5780039c0 0x7fb578003de0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 -- 172.21.15.114:0/138775651 >> 172.21.15.114:0/138775651 conn(0x7fb58c0a1fb0 msgr2=0x7fb58c0a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:26.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 -- 172.21.15.114:0/138775651 shutdown_connections 2024-09-16T05:27:26.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 -- 172.21.15.114:0/138775651 wait complete. 2024-09-16T05:27:26.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient: build_initial_monmap 2024-09-16T05:27:26.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient: monmap: 2024-09-16T05:27:26.902 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-09-16T05:27:26.902 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:27:26.902 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-09-16T05:27:26.902 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-09-16T05:27:26.902 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-09-16T05:27:26.902 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:26.902 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.noname-a 2024-09-16T05:27:26.902 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.noname-b 2024-09-16T05:27:26.902 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.noname-c 2024-09-16T05:27:26.902 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:26.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 librados: starting msgr at 2024-09-16T05:27:26.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 librados: starting objecter 2024-09-16T05:27:26.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 Processor -- start 2024-09-16T05:27:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 -- start start 2024-09-16T05:27:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 librados: setting wanted keys 2024-09-16T05:27:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 librados: calling monclient init 2024-09-16T05:27:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient: init 2024-09-16T05:27:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient: _reopen_session rank -1 2024-09-16T05:27:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:27:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5780039c0 0x7fb58c09c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient(hunting): picked mon.noname-c con 0x7fb5780039c0 addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-16T05:27:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb58c12e540 0x7fb58c12e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient(hunting): picked mon.noname-b con 0x7fb58c12e540 addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-09-16T05:27:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb58c12eea0 0x7fb58c1d6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient(hunting): picked mon.noname-a con 0x7fb58c12eea0 addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T05:27:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:26.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb58c09c3b0 con 0x7fb58c12eea0 2024-09-16T05:27:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 -- 0x7fb58c09c230 con 0x7fb58c12e540 2024-09-16T05:27:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient(hunting): start opening mon connection 2024-09-16T05:27:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb58c136e30 con 0x7fb5780039c0 2024-09-16T05:27:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient(hunting): _renew_subs 2024-09-16T05:27:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient(hunting): authenticate will time out at 2131.518708s 2024-09-16T05:27:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5780039c0 0x7fb58c09c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb58c12eea0 0x7fb58c1d6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5780039c0 0x7fb58c09c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:41492/0 (socket says 172.21.15.114:41492) 2024-09-16T05:27:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58bfff640 1 -- 172.21.15.114:0/4194139835 learned_addr learned my addr 172.21.15.114:0/4194139835 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T05:27:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58bfff640 10 monclient(hunting): get_auth_request con 0x7fb5780039c0 auth_method 0 2024-09-16T05:27:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 10 monclient(hunting): get_auth_request con 0x7fb58c12eea0 auth_method 0 2024-09-16T05:27:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T05:27:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T05:27:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T05:27:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58b7fe640 1 --2- 172.21.15.114:0/4194139835 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb58c12e540 0x7fb58c12e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58b7fe640 10 monclient(hunting): get_auth_request con 0x7fb58c12e540 auth_method 0 2024-09-16T05:27:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T05:27:26.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:27:26.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:27:26.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 10 monclient(hunting): handle_auth_done global_id 134280 payload 824 2024-09-16T05:27:26.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 1 -- 172.21.15.114:0/4194139835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5780039c0 msgr2=0x7fb58c09c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:26.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 1 --2- 172.21.15.114:0/4194139835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5780039c0 0x7fb58c09c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 1 -- 172.21.15.114:0/4194139835 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb58c12e540 msgr2=0x7fb58c12e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:26.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 1 --2- 172.21.15.114:0/4194139835 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb58c12e540 0x7fb58c12e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:26.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 10 monclient: _finish_hunting 0 2024-09-16T05:27:26.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 1 monclient: found mon.noname-a 2024-09-16T05:27:26.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:27:26.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.114:3300/0 2024-09-16T05:27:26.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 1 -- 172.21.15.114:0/4194139835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb58c130fa0 con 0x7fb58c12eea0 2024-09-16T05:27:26.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58bfff640 1 --2- 172.21.15.114:0/4194139835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5780039c0 0x7fb58c09c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T05:27:26.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb590859640 1 --2- 172.21.15.114:0/4194139835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb58c12eea0 0x7fb58c1d6040 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb58c06cf30 tx=0x7fb58400ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:26.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb588ff9640 10 client.?.objecter ms_handle_connect 0x7fb58c12eea0 2024-09-16T05:27:26.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb588ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T05:27:26.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb588ff9640 1 -- 172.21.15.114:0/4194139835 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb584018020 con 0x7fb58c12eea0 2024-09-16T05:27:26.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb588ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:26.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb588ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:27:26.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb588ff9640 10 monclient: dump: 2024-09-16T05:27:26.907 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:26.907 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:26.907 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.907 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.907 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:26.907 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:26.908 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:26.908 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:26.908 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:26.908 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:26.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb588ff9640 10 monclient: _finish_auth 0 2024-09-16T05:27:26.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb588ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:27:26.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb588ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:27:26.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 5 monclient: authenticate success, global_id 134280 2024-09-16T05:27:26.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb588ff9640 1 -- 172.21.15.114:0/4194139835 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb58400bb10 con 0x7fb58c12eea0 2024-09-16T05:27:26.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb588ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:27:26.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient: _renew_subs 2024-09-16T05:27:26.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb588ff9640 1 -- 172.21.15.114:0/4194139835 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb584014400 con 0x7fb58c12eea0 2024-09-16T05:27:26.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:26.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 -- 172.21.15.114:0/4194139835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb58c131280 con 0x7fb58c12eea0 2024-09-16T05:27:26.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb58a7fc640 4 set_mon_vals no callback set 2024-09-16T05:27:26.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb588ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:27:26.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb588ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:27:26.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb588ff9640 10 monclient: dump: 2024-09-16T05:27:26.909 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-09-16T05:27:26.909 INFO:teuthology.orchestra.run.smithi114.stderr:fsid c6dbfe9e-ab8c-4165-ae76-3f2273766e2c 2024-09-16T05:27:26.909 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.909 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-09-16T05:06:54.422314+0000 2024-09-16T05:27:26.909 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-09-16T05:27:26.909 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-09-16T05:27:26.909 INFO:teuthology.orchestra.run.smithi114.stderr:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-16T05:27:26.909 INFO:teuthology.orchestra.run.smithi114.stderr:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-09-16T05:27:26.909 INFO:teuthology.orchestra.run.smithi114.stderr:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-16T05:27:26.910 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:26.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 client.134280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T05:27:26.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient: _renew_subs 2024-09-16T05:27:26.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:26.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 -- 172.21.15.114:0/4194139835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb58c12f300 con 0x7fb58c12eea0 2024-09-16T05:27:26.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.886+0000 7fb5922bc640 1 librados: init done 2024-09-16T05:27:26.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.890+0000 7fb588ff9640 1 -- 172.21.15.114:0/4194139835 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb58401b020 con 0x7fb58c12eea0 2024-09-16T05:27:26.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.890+0000 7fb5922bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:26.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.890+0000 7fb5922bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:26.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.890+0000 7fb5922bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:26.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.890+0000 7fb5922bc640 1 -- 172.21.15.114:0/4194139835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb58c136fb0 con 0x7fb58c12eea0 2024-09-16T05:27:26.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.890+0000 7fb588ff9640 1 --2- 172.21.15.114:0/4194139835 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb56c037810 0x7fb56c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T05:27:26.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.890+0000 7fb588ff9640 1 -- 172.21.15.114:0/4194139835 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(695..695 src has 1..695) ==== 4733+0+0 (secure 0 0 0) 0x7fb584048de0 con 0x7fb58c12eea0 2024-09-16T05:27:26.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.890+0000 7fb588ff9640 10 client.134280.objecter ms_dispatch 0x7fb58c000ba0 osd_map(695..695 src has 1..695) 2024-09-16T05:27:26.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.890+0000 7fb588ff9640 3 client.134280.objecter handle_osd_map got epochs [695,695] > 0 2024-09-16T05:27:26.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.890+0000 7fb588ff9640 3 client.134280.objecter handle_osd_map decoding full epoch 695 2024-09-16T05:27:26.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.890+0000 7fb588ff9640 20 client.134280.objecter dump_active .. 0 homeless 2024-09-16T05:27:26.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.890+0000 7fb58bfff640 1 --2- 172.21.15.114:0/4194139835 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb56c037810 0x7fb56c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T05:27:26.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.890+0000 7fb58bfff640 10 monclient: get_auth_request con 0x7fb56c037810 auth_method 0 2024-09-16T05:27:26.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.890+0000 7fb588ff9640 10 client.134280.objecter ms_handle_connect 0x7fb56c037810 2024-09-16T05:27:26.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.890+0000 7fb58bfff640 1 --2- 172.21.15.114:0/4194139835 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb56c037810 0x7fb56c039cd0 secure :-1 s=READY pgs=3647 cs=0 l=1 rev1=1 crypto rx=0x7fb57c0011f0 tx=0x7fb57c0a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T05:27:26.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.894+0000 7fb588ff9640 1 -- 172.21.15.114:0/4194139835 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb584022020 con 0x7fb58c12eea0 2024-09-16T05:27:26.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.894+0000 7fb588ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:27:26.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:26.894+0000 7fb588ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:27:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.054+0000 7fb5922bc640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T05:27:27.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.054+0000 7fb5922bc640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T05:27:27.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.054+0000 7fb5922bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.114:3300/0 2024-09-16T05:27:27.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.054+0000 7fb5922bc640 1 -- 172.21.15.114:0/4194139835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fb58c09c6a0 con 0x7fb58c12eea0 2024-09-16T05:27:27.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.058+0000 7fb588ff9640 1 -- 172.21.15.114:0/4194139835 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) ==== 123+0+0 (secure 0 0 0) 0x7fb584013520 con 0x7fb58c12eea0 2024-09-16T05:27:27.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.058+0000 7fb588ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T05:27:27.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.058+0000 7fb588ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:27:27.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb588ff9640 1 -- 172.21.15.114:0/4194139835 <== mon.0 v2:172.21.15.114:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fb584013780 con 0x7fb58c12eea0 2024-09-16T05:27:27.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb588ff9640 10 monclient: handle_config config(1 keys) 2024-09-16T05:27:27.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb589ffb640 4 set_mon_vals no callback set 2024-09-16T05:27:27.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb589ffb640 10 set_mon_vals mon_health_to_clog = false 2024-09-16T05:27:27.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 10 librados: watch_flush enter 2024-09-16T05:27:27.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 10 librados: watch_flush exit 2024-09-16T05:27:27.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 20 client.134280.objecter shutdown clearing up homeless session... 2024-09-16T05:27:27.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 10 client.134280.objecter successfully canceled tick 2024-09-16T05:27:27.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 1 -- 172.21.15.114:0/4194139835 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb56c037810 msgr2=0x7fb56c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:27.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 1 --2- 172.21.15.114:0/4194139835 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb56c037810 0x7fb56c039cd0 secure :-1 s=READY pgs=3647 cs=0 l=1 rev1=1 crypto rx=0x7fb57c0011f0 tx=0x7fb57c0a7070 comp rx=0 tx=0).stop 2024-09-16T05:27:27.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 10 monclient: shutdown 2024-09-16T05:27:27.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:27:27.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 1 -- 172.21.15.114:0/4194139835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb58c12eea0 msgr2=0x7fb58c1d6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:27:27.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 1 --2- 172.21.15.114:0/4194139835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb58c12eea0 0x7fb58c1d6040 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb58c06cf30 tx=0x7fb58400ef50 comp rx=0 tx=0).stop 2024-09-16T05:27:27.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 1 -- 172.21.15.114:0/4194139835 shutdown_connections 2024-09-16T05:27:27.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 1 --2- 172.21.15.114:0/4194139835 >> v2:172.21.15.149:6812/1185531135 conn(0x7fb56c037810 0x7fb56c039cd0 unknown :-1 s=CLOSED pgs=3647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:27.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 1 --2- 172.21.15.114:0/4194139835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb58c12eea0 0x7fb58c1d6040 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:27.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 1 --2- 172.21.15.114:0/4194139835 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb58c12e540 0x7fb58c12e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:27.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 1 --2- 172.21.15.114:0/4194139835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5780039c0 0x7fb58c09c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T05:27:27.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 1 -- 172.21.15.114:0/4194139835 >> 172.21.15.114:0/4194139835 conn(0x7fb58c0a1fb0 msgr2=0x7fb58c0a32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:27:27.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 1 -- 172.21.15.114:0/4194139835 shutdown_connections 2024-09-16T05:27:27.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 1 -- 172.21.15.114:0/4194139835 wait complete. 2024-09-16T05:27:27.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T05:27:27.062+0000 7fb5922bc640 1 librados: shutdown 2024-09-16T05:27:27.077 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T05:27:27.077 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T05:27:27.078 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T05:27:27.078 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:27:27.143 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T05:27:27.144 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T05:27:27.144 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:27:27.237 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T05:27:27.237 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T05:27:27.237 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:27:27.333 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T05:27:27.334 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T05:27:27.334 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:27:27.407 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T05:27:27.407 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T05:27:27.407 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:27:28.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-09-16T05:27:33.508 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T05:27:33.509 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T05:27:33.509 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:27:33.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~6s 2024-09-16T05:27:33.572 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T05:27:33.573 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T05:27:33.573 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T05:27:33.573 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:27:33.604 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T05:27:33.604 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T05:27:33.604 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T05:27:33.604 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:27:33.630 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T05:27:33.630 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T05:27:33.630 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:27:33.694 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T05:27:33.694 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T05:27:33.694 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:27:33.721 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T05:27:33.721 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T05:27:33.721 DEBUG:teuthology.orchestra.run.smithi114:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2024-09-16T05:27:33.754 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T05:27:33.754 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T05:27:33.976 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T05:27:33.976 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T05:27:34.173 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T05:27:34.173 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T05:27:34.306 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi149.front.sepia.ceph.com 2024-09-16T05:27:34.307 DEBUG:teuthology.orchestra.run.smithi149:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T05:27:34.397 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi149.front.sepia.ceph.com 2024-09-16T05:27:34.397 DEBUG:teuthology.orchestra.run.smithi149:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T05:27:34.549 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi149.front.sepia.ceph.com 2024-09-16T05:27:34.549 DEBUG:teuthology.orchestra.run.smithi149:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T05:27:34.688 INFO:tasks.ceph:Archiving mon data... 2024-09-16T05:27:34.689 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906097/data/mon.a.tgz 2024-09-16T05:27:34.690 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-09-16T05:27:34.696 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.MGeILoqkVk 2024-09-16T05:27:34.696 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.MGeILoqkVk 2024-09-16T05:27:34.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.MGeILoqkVk 2024-09-16T05:27:34.943 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.MGeILoqkVk is 722KB 2024-09-16T05:27:35.009 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.MGeILoqkVk 2024-09-16T05:27:35.014 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906097/data/mon.c.tgz 2024-09-16T05:27:35.014 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-09-16T05:27:35.063 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.cjZRIJW3BS 2024-09-16T05:27:35.064 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.cjZRIJW3BS 2024-09-16T05:27:35.214 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.cjZRIJW3BS 2024-09-16T05:27:35.275 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.cjZRIJW3BS is 537KB 2024-09-16T05:27:35.337 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.cjZRIJW3BS 2024-09-16T05:27:35.343 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906097/data/mon.b.tgz 2024-09-16T05:27:35.344 DEBUG:teuthology.orchestra.run.smithi149:> mktemp 2024-09-16T05:27:35.349 INFO:teuthology.orchestra.run.smithi149.stdout:/tmp/tmp.dQKNszf5t8 2024-09-16T05:27:35.350 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.dQKNszf5t8 2024-09-16T05:27:35.498 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0666 /tmp/tmp.dQKNszf5t8 2024-09-16T05:27:35.563 DEBUG:teuthology.orchestra.remote:smithi149:/tmp/tmp.dQKNszf5t8 is 529KB 2024-09-16T05:27:35.625 DEBUG:teuthology.orchestra.run.smithi149:> rm -fr /tmp/tmp.dQKNszf5t8 2024-09-16T05:27:35.630 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T05:27:35.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T05:27:35.633 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T05:27:35.693 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T05:27:35.735 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T05:27:35.735 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T05:27:35.737 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906097/remote/smithi114/crash 2024-09-16T05:27:35.737 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T05:27:35.752 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906097/remote/smithi149/crash 2024-09-16T05:27:35.753 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T05:27:35.791 INFO:tasks.ceph:Compressing logs... 2024-09-16T05:27:35.792 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T05:27:35.801 DEBUG:teuthology.orchestra.run.smithi149:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T05:27:35.819 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32391.log 2024-09-16T05:27:35.820 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32961.log 2024-09-16T05:27:35.820 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31129.log 2024-09-16T05:27:35.821 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35112.log 2024-09-16T05:27:35.821 INFO:teuthology.orchestra.run.smithi114.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32391.log.gz 2024-09-16T05:27:35.821 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31049.log 2024-09-16T05:27:35.822 INFO:teuthology.orchestra.run.smithi114.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32961.log.gz 2024-09-16T05:27:35.822 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31129.log: /var/log/ceph/ceph-client.admin.35112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32812.log 2024-09-16T05:27:35.826 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32139.log 2024-09-16T05:27:35.826 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31049.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35112.log.gz 2024-09-16T05:27:35.826 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33529.log 2024-09-16T05:27:35.826 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32812.log.gz 2024-09-16T05:27:35.827 INFO:teuthology.orchestra.run.smithi114.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.31049.log.gz 2024-09-16T05:27:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32139.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31129.log.gz 2024-09-16T05:27:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31155.log 2024-09-16T05:27:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33529.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32139.log.gz 2024-09-16T05:27:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25348.log 2024-09-16T05:27:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30088.log 2024-09-16T05:27:35.827 INFO:teuthology.orchestra.run.smithi114.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33529.log.gz 2024-09-16T05:27:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:gzip/var/log/ceph/ceph-client.admin.25348.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35060.log 2024-09-16T05:27:35.827 INFO:teuthology.orchestra.run.smithi114.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31155.log.gz 2024-09-16T05:27:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31075.log 2024-09-16T05:27:35.828 INFO:teuthology.orchestra.run.smithi114.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.25348.log.gz 2024-09-16T05:27:35.828 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29770.log 2024-09-16T05:27:35.828 INFO:teuthology.orchestra.run.smithi114.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.30088.log.gz 2024-09-16T05:27:35.828 INFO:teuthology.orchestra.run.smithi114.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35060.log.gz 2024-09-16T05:27:35.828 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32191.log 2024-09-16T05:27:35.828 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34181.log 2024-09-16T05:27:35.828 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.29770.log: /var/log/ceph/ceph-client.admin.32191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35138.log 2024-09-16T05:27:35.828 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34181.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31075.log.gz 2024-09-16T05:27:35.828 INFO:teuthology.orchestra.run.smithi114.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29770.log.gz 2024-09-16T05:27:35.828 INFO:teuthology.orchestra.run.smithi114.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32191.log.gz 2024-09-16T05:27:35.829 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T05:27:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34330.log 2024-09-16T05:27:35.841 INFO:teuthology.orchestra.run.smithi114.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34181.log.gz 2024-09-16T05:27:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29752.log 2024-09-16T05:27:35.841 INFO:teuthology.orchestra.run.smithi114.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35138.log.gz 2024-09-16T05:27:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31600.log 2024-09-16T05:27:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34330.log: /var/log/ceph/ceph-client.admin.29752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32680.log 2024-09-16T05:27:35.842 INFO:teuthology.orchestra.run.smithi114.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34330.log.gz 2024-09-16T05:27:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31600.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29752.log.gz 2024-09-16T05:27:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29516.log 2024-09-16T05:27:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33432.log 2024-09-16T05:27:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35242.log 2024-09-16T05:27:35.842 INFO:teuthology.orchestra.run.smithi114.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.31600.log.gz 2024-09-16T05:27:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.29516.log: /var/log/ceph/ceph-client.admin.33432.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32680.log.gz 2024-09-16T05:27:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35268.log 2024-09-16T05:27:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29764.log 2024-09-16T05:27:35.842 INFO:teuthology.orchestra.run.smithi114.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33432.log.gz 2024-09-16T05:27:35.842 INFO:teuthology.orchestra.run.smithi114.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29516.log.gz 2024-09-16T05:27:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35242.log: /var/log/ceph/ceph-client.admin.35268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34438.log 2024-09-16T05:27:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.29764.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35242.log.gz 2024-09-16T05:27:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31312.log 2024-09-16T05:27:35.843 INFO:teuthology.orchestra.run.smithi114.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.29764.log.gz 2024-09-16T05:27:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33740.log 2024-09-16T05:27:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34438.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35268.log.gz 2024-09-16T05:27:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33792.log 2024-09-16T05:27:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33740.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.34438.log.gz 2024-09-16T05:27:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33766.log 2024-09-16T05:27:35.843 INFO:teuthology.orchestra.run.smithi114.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31312.log.gz 2024-09-16T05:27:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32732.log 2024-09-16T05:27:35.843 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33740.log.gz 2024-09-16T05:27:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33792.log: /var/log/ceph/ceph-client.admin.33766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32523.log 2024-09-16T05:27:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32732.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33792.log.gz 2024-09-16T05:27:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30995.log 2024-09-16T05:27:35.843 INFO:teuthology.orchestra.run.smithi114.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.32732.log.gz 2024-09-16T05:27:35.843 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33766.log.gz 2024-09-16T05:27:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31851.log 2024-09-16T05:27:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32523.log: /var/log/ceph/ceph-client.admin.30995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34675.log 2024-09-16T05:27:35.844 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32523.log.gz 2024-09-16T05:27:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35034.log 2024-09-16T05:27:35.844 INFO:teuthology.orchestra.run.smithi114.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30995.log.gz 2024-09-16T05:27:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35190.log/var/log/ceph/ceph-client.admin.34675.log: 2024-09-16T05:27:35.844 INFO:teuthology.orchestra.run.smithi114.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.31851.log.gz 2024-09-16T05:27:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33610.log 2024-09-16T05:27:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31678.log 2024-09-16T05:27:35.844 INFO:teuthology.orchestra.run.smithi114.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34675.log.gz 2024-09-16T05:27:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33610.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35034.log.gz 2024-09-16T05:27:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32217.log 2024-09-16T05:27:35.844 INFO:teuthology.orchestra.run.smithi114.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35190.log.gz 2024-09-16T05:27:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31678.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.33610.log.gz 2024-09-16T05:27:35.845 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32758.log 2024-09-16T05:27:35.845 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T05:27:35.845 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32217.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31678.log.gz 2024-09-16T05:27:35.845 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32654.log 2024-09-16T05:27:35.845 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35164.log 2024-09-16T05:27:35.845 INFO:teuthology.orchestra.run.smithi114.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.32217.log.gz 2024-09-16T05:27:35.845 INFO:teuthology.orchestra.run.smithi114.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.32758.log.gz 2024-09-16T05:27:35.845 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35578.log 2024-09-16T05:27:35.845 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33844.log 2024-09-16T05:27:35.845 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35164.log: /var/log/ceph/ceph-client.admin.35578.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32654.log.gz 2024-09-16T05:27:35.845 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29917.log 2024-09-16T05:27:35.845 INFO:teuthology.orchestra.run.smithi114.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35164.log.gz 2024-09-16T05:27:35.845 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33844.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35578.log.gz 2024-09-16T05:27:35.845 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30244.log 2024-09-16T05:27:35.845 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30296.log 2024-09-16T05:27:35.846 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.29917.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33844.log.gz 2024-09-16T05:27:35.846 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30915.log 2024-09-16T05:27:35.846 INFO:teuthology.orchestra.run.smithi114.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.29917.log.gz 2024-09-16T05:27:35.846 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31983.log 2024-09-16T05:27:35.846 INFO:teuthology.orchestra.run.smithi114.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30244.log.gz 2024-09-16T05:27:35.846 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30915.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T05:27:35.846 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30296.log.gz 2024-09-16T05:27:35.846 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30140.log 2024-09-16T05:27:35.857 INFO:teuthology.orchestra.run.smithi114.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30915.log.gz 2024-09-16T05:27:35.857 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32061.log 2024-09-16T05:27:35.857 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30140.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31983.log.gz 2024-09-16T05:27:35.857 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29926.log 2024-09-16T05:27:35.857 INFO:teuthology.orchestra.run.smithi114.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30140.log.gz 2024-09-16T05:27:35.857 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31494.log 2024-09-16T05:27:35.858 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30166.log 2024-09-16T05:27:35.858 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.29926.log: /var/log/ceph/ceph-client.admin.31494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33274.log 2024-09-16T05:27:35.858 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32061.log.gz 2024-09-16T05:27:35.858 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30166.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29926.log.gz 2024-09-16T05:27:35.858 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34903.log 2024-09-16T05:27:35.858 INFO:teuthology.orchestra.run.smithi114.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31494.log.gz 2024-09-16T05:27:35.858 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33196.log 2024-09-16T05:27:35.858 INFO:teuthology.orchestra.run.smithi114.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30166.log.gz 2024-09-16T05:27:35.858 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31390.log/var/log/ceph/ceph-client.admin.34903.log: 2024-09-16T05:27:35.858 INFO:teuthology.orchestra.run.smithi114.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33274.log.gz 2024-09-16T05:27:35.858 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31798.log 2024-09-16T05:27:35.858 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31905.log 2024-09-16T05:27:35.859 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31390.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33196.log.gz 2024-09-16T05:27:35.859 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31798.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.34903.log.gz 2024-09-16T05:27:35.859 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31931.log 2024-09-16T05:27:35.859 INFO:teuthology.orchestra.run.smithi114.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31798.log.gz 2024-09-16T05:27:35.859 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31905.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31652.log 2024-09-16T05:27:35.859 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31390.log.gz 2024-09-16T05:27:35.859 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34077.log 2024-09-16T05:27:35.859 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31931.log: /var/log/ceph/ceph-client.admin.31652.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31905.log.gz 2024-09-16T05:27:35.859 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35294.log 2024-09-16T05:27:35.859 INFO:teuthology.orchestra.run.smithi114.stderr:gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31652.log.gz -5 2024-09-16T05:27:35.859 INFO:teuthology.orchestra.run.smithi114.stderr: --verbose -- /var/log/ceph/ceph-client.admin.33014.log 2024-09-16T05:27:35.859 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34077.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31931.log.gz 2024-09-16T05:27:35.859 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34701.log 2024-09-16T05:27:35.859 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30941.log 2024-09-16T05:27:35.860 INFO:teuthology.orchestra.run.smithi114.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34077.log.gz 2024-09-16T05:27:35.860 INFO:teuthology.orchestra.run.smithi114.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35294.log.gz 2024-09-16T05:27:35.860 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34622.log 2024-09-16T05:27:35.860 INFO:teuthology.orchestra.run.smithi114.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33014.log.gz 2024-09-16T05:27:35.860 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31707.log 2024-09-16T05:27:35.860 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 89.2% --verbose -- /var/log/ceph/ceph-client.admin.34982.log -- replaced with /var/log/ceph/ceph-client.admin.34701.log.gz 2024-09-16T05:27:35.860 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:35.860 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34622.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30941.log.gz 2024-09-16T05:27:35.860 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35216.log 2024-09-16T05:27:35.860 INFO:teuthology.orchestra.run.smithi114.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34622.log.gz 2024-09-16T05:27:35.860 INFO:teuthology.orchestra.run.smithi114.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.31707.log.gz 2024-09-16T05:27:35.860 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34518.log 2024-09-16T05:27:35.860 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33714.log 2024-09-16T05:27:35.861 INFO:teuthology.orchestra.run.smithi114.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34982.log.gz 2024-09-16T05:27:35.861 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35398.log 2024-09-16T05:27:35.861 INFO:teuthology.orchestra.run.smithi114.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35216.log.gz 2024-09-16T05:27:35.861 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30348.log 2024-09-16T05:27:35.861 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34518.log.gz 2024-09-16T05:27:35.861 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33714.log: /var/log/ceph/ceph-client.admin.35398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35086.log 2024-09-16T05:27:35.861 INFO:teuthology.orchestra.run.smithi114.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33714.log.gz 2024-09-16T05:27:35.861 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33999.log 2024-09-16T05:27:35.861 INFO:teuthology.orchestra.run.smithi114.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35398.log.gz 2024-09-16T05:27:35.861 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31023.log 2024-09-16T05:27:35.861 INFO:teuthology.orchestra.run.smithi114.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30348.log.gz 2024-09-16T05:27:35.861 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35450.log 2024-09-16T05:27:35.861 INFO:teuthology.orchestra.run.smithi114.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35086.log.gz 2024-09-16T05:27:35.861 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32706.log 2024-09-16T05:27:35.862 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31023.log: /var/log/ceph/ceph-client.admin.35450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29769.log 2024-09-16T05:27:35.862 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33999.log.gz 2024-09-16T05:27:35.862 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32009.log 2024-09-16T05:27:35.862 INFO:teuthology.orchestra.run.smithi114.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35450.log.gz 2024-09-16T05:27:35.862 INFO:teuthology.orchestra.run.smithi114.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31023.log.gz 2024-09-16T05:27:35.862 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32706.log: /var/log/ceph/ceph-client.admin.29769.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32706.log.gz 2024-09-16T05:27:35.862 INFO:teuthology.orchestra.run.smithi114.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29769.log.gz 2024-09-16T05:27:35.862 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25264.log 2024-09-16T05:27:35.862 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32445.log 2024-09-16T05:27:35.862 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.25264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34155.log 2024-09-16T05:27:35.887 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29453.log 2024-09-16T05:27:35.887 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30197.log 2024-09-16T05:27:35.887 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29587.log 2024-09-16T05:27:35.887 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30649.log 2024-09-16T05:27:35.888 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30258.log 2024-09-16T05:27:35.888 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30598.log 2024-09-16T05:27:35.888 INFO:teuthology.orchestra.run.smithi149.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.30197.log.gz 2024-09-16T05:27:35.888 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29771.log 2024-09-16T05:27:35.888 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29498.log 2024-09-16T05:27:35.888 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30598.log: /var/log/ceph/ceph-client.admin.29771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30022.log 2024-09-16T05:27:35.888 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30120.log 2024-09-16T05:27:35.888 INFO:teuthology.orchestra.run.smithi149.stderr: 91.5%/var/log/ceph/ceph-client.admin.30022.log: -- replaced with /var/log/ceph/ceph-client.admin.30649.log.gz 2024-09-16T05:27:35.888 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30996.log 2024-09-16T05:27:35.888 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30120.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30022.log.gz 2024-09-16T05:27:35.888 INFO:teuthology.orchestra.run.smithi149.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.30598.log.gz 2024-09-16T05:27:35.888 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30995.log 2024-09-16T05:27:35.888 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30650.log 2024-09-16T05:27:35.888 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30995.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.29498.log.gz 2024-09-16T05:27:35.889 INFO:teuthology.orchestra.run.smithi149.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.30996.log.gz 2024-09-16T05:27:35.889 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30703.log 2024-09-16T05:27:35.889 INFO:teuthology.orchestra.run.smithi149.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.29771.log.gz 2024-09-16T05:27:35.889 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29490.log 2024-09-16T05:27:35.889 INFO:teuthology.orchestra.run.smithi149.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.30995.log.gz 2024-09-16T05:27:35.889 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29776.log 2024-09-16T05:27:35.889 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25250.log 2024-09-16T05:27:35.889 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29490.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30703.log.gz 2024-09-16T05:27:35.889 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29944.log 2024-09-16T05:27:35.889 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.25250.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.25250.log.gz 2024-09-16T05:27:35.889 INFO:teuthology.orchestra.run.smithi149.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.30650.log.gz 2024-09-16T05:27:35.889 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25322.log 2024-09-16T05:27:35.889 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29922.log 2024-09-16T05:27:35.889 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29944.log: /var/log/ceph/ceph-client.admin.25322.log: 87.6% 91.4% -- replaced with /var/log/ceph/ceph-client.admin.29490.log.gz 2024-09-16T05:27:35.890 INFO:teuthology.orchestra.run.smithi149.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25322.log.gz 2024-09-16T05:27:35.890 INFO:teuthology.orchestra.run.smithi149.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29944.log.gz 2024-09-16T05:27:35.890 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30993.log 2024-09-16T05:27:35.890 INFO:teuthology.orchestra.run.smithi149.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.30120.log.gz 2024-09-16T05:27:35.890 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29536.log 2024-09-16T05:27:35.890 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29631.log 2024-09-16T05:27:35.890 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29477.log 2024-09-16T05:27:35.890 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29631.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.30258.log.gz 2024-09-16T05:27:35.890 INFO:teuthology.orchestra.run.smithi149.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.30993.log.gz 92.0% -- replaced with /var/log/ceph/ceph-client.admin.29776.log.gz 2024-09-16T05:27:35.890 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30274.log 2024-09-16T05:27:35.890 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:27:35.890 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31079.log 2024-09-16T05:27:35.890 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29955.log 2024-09-16T05:27:35.890 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.31079.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.29477.log.gz 2024-09-16T05:27:35.890 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29987.log 2024-09-16T05:27:35.891 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29955.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.29955.log.gz 2024-09-16T05:27:35.891 INFO:teuthology.orchestra.run.smithi149.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.31079.log.gz 2024-09-16T05:27:35.891 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30699.log 2024-09-16T05:27:35.891 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30370.log 2024-09-16T05:27:35.891 INFO:teuthology.orchestra.run.smithi149.stderr: 91.2%/var/log/ceph/ceph-client.admin.30699.log: -- replaced with /var/log/ceph/ceph-client.admin.29631.log.gz 2024-09-16T05:27:35.891 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30236.log 2024-09-16T05:27:35.891 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30370.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.29987.log.gz 2024-09-16T05:27:35.891 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29736.log 2024-09-16T05:27:35.891 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30236.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30370.log.gz 2024-09-16T05:27:35.891 INFO:teuthology.orchestra.run.smithi149.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.30699.log.gz 2024-09-16T05:27:35.891 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29623.log 2024-09-16T05:27:35.891 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29736.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T05:27:35.891 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29623.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.30274.log.gz 2024-09-16T05:27:35.891 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T05:27:35.894 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-osd.3.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.29736.log.gz 2024-09-16T05:27:35.894 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29787.log 2024-09-16T05:27:35.904 INFO:teuthology.orchestra.run.smithi114.stderr: 88.3% 87.8% -- replaced with /var/log/ceph/ceph-client.admin.25264.log.gz 2024-09-16T05:27:35.904 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29604.log 2024-09-16T05:27:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32445.log: -- replaced with /var/log/ceph/ceph-client.admin.32009.log.gz 2024-09-16T05:27:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33688.log 2024-09-16T05:27:35.905 INFO:teuthology.orchestra.run.smithi114.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32445.log.gz 2024-09-16T05:27:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.29604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30062.log 2024-09-16T05:27:35.905 INFO:teuthology.orchestra.run.smithi114.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34155.log.gz 2024-09-16T05:27:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29608.log 2024-09-16T05:27:35.905 INFO:teuthology.orchestra.run.smithi114.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29604.log.gz 2024-09-16T05:27:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25146.log 2024-09-16T05:27:35.905 INFO:teuthology.orchestra.run.smithi114.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33688.log.gz 2024-09-16T05:27:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.29608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30374.log 2024-09-16T05:27:35.905 INFO:teuthology.orchestra.run.smithi114.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.30062.log.gz 2024-09-16T05:27:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.25146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30400.log 2024-09-16T05:27:35.905 INFO:teuthology.orchestra.run.smithi114.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29608.log.gz 2024-09-16T05:27:35.906 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29446.log 2024-09-16T05:27:35.906 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31468.log 2024-09-16T05:27:35.906 INFO:teuthology.orchestra.run.smithi114.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.25146.log.gz 2024-09-16T05:27:35.906 INFO:teuthology.orchestra.run.smithi114.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.30374.log.gz 2024-09-16T05:27:35.906 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32601.log 2024-09-16T05:27:35.906 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.29446.log: /var/log/ceph/ceph-client.admin.31468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31442.log 2024-09-16T05:27:35.906 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30400.log.gz/var/log/ceph/ceph-client.admin.32601.log: 2024-09-16T05:27:35.906 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34850.log 2024-09-16T05:27:35.906 INFO:teuthology.orchestra.run.smithi114.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29446.log.gz 2024-09-16T05:27:35.906 INFO:teuthology.orchestra.run.smithi114.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31468.log.gz 2024-09-16T05:27:35.906 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31442.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32601.log.gz 2024-09-16T05:27:35.906 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33379.log 2024-09-16T05:27:35.906 INFO:teuthology.orchestra.run.smithi114.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31442.log.gz 2024-09-16T05:27:35.906 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T05:27:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34850.log: /var/log/ceph/ceph-client.admin.33379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30322.log 2024-09-16T05:27:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32419.log 2024-09-16T05:27:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.1.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33379.log.gz 2024-09-16T05:27:35.907 INFO:teuthology.orchestra.run.smithi114.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34850.log.gz 2024-09-16T05:27:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33636.log 2024-09-16T05:27:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31572.log 2024-09-16T05:27:35.907 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30322.log.gz 2024-09-16T05:27:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:gzip/var/log/ceph/ceph-client.admin.33636.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34207.log 2024-09-16T05:27:35.907 INFO:teuthology.orchestra.run.smithi114.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32419.log.gz 2024-09-16T05:27:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31572.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31572.log.gz 2024-09-16T05:27:35.907 INFO:teuthology.orchestra.run.smithi114.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.33636.log.gz 2024-09-16T05:27:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35008.log 2024-09-16T05:27:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31103.log 2024-09-16T05:27:35.907 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31416.log 2024-09-16T05:27:35.908 INFO:teuthology.orchestra.run.smithi114.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34207.log.gz 2024-09-16T05:27:35.908 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32549.log 2024-09-16T05:27:35.908 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35008.log.gz 2024-09-16T05:27:35.908 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29496.log 2024-09-16T05:27:35.908 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31416.log: /var/log/ceph/ceph-client.admin.32549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34544.log 2024-09-16T05:27:35.908 INFO:teuthology.orchestra.run.smithi114.stderr: 90.7%/var/log/ceph/ceph-client.admin.29496.log: -- replaced with /var/log/ceph/ceph-client.admin.31103.log.gz 2024-09-16T05:27:35.908 INFO:teuthology.orchestra.run.smithi114.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32549.log.gz 2024-09-16T05:27:35.908 INFO:teuthology.orchestra.run.smithi114.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31416.log.gz 2024-09-16T05:27:35.908 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29613.log 2024-09-16T05:27:35.908 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34544.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29496.log.gz 2024-09-16T05:27:35.908 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32087.log 2024-09-16T05:27:35.908 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5/var/log/ceph/ceph-client.admin.29613.log: --verbose -- /var/log/ceph/ceph-client.admin.32471.log 2024-09-16T05:27:35.908 INFO:teuthology.orchestra.run.smithi114.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34544.log.gz 2024-09-16T05:27:35.908 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35346.log 2024-09-16T05:27:35.909 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31520.log 2024-09-16T05:27:35.909 INFO:teuthology.orchestra.run.smithi114.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29613.log.gz 2024-09-16T05:27:35.909 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32087.log.gz 2024-09-16T05:27:35.909 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32628.log 2024-09-16T05:27:35.909 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35346.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.32471.log.gz 2024-09-16T05:27:35.909 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31520.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35346.log.gz 2024-09-16T05:27:35.909 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T05:27:35.909 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31520.log.gz 2024-09-16T05:27:35.909 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31338.log 2024-09-16T05:27:35.909 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35502.log 2024-09-16T05:27:35.910 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32628.log.gz 2024-09-16T05:27:35.910 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33248.log 2024-09-16T05:27:35.910 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31338.log: /var/log/ceph/ceph-client.admin.35502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29910.log 2024-09-16T05:27:35.910 INFO:teuthology.orchestra.run.smithi114.stderr: 88.9%/var/log/ceph/ceph-client.admin.33248.log: -- replaced with /var/log/ceph/ceph-client.admin.31338.log.gz 2024-09-16T05:27:35.910 INFO:teuthology.orchestra.run.smithi114.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35502.log.gz 2024-09-16T05:27:35.910 INFO:teuthology.orchestra.run.smithi114.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33248.log.gz 2024-09-16T05:27:35.910 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29605.log 2024-09-16T05:27:35.910 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.29910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34234.log 2024-09-16T05:27:35.910 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.29605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33818.log 2024-09-16T05:27:35.910 INFO:teuthology.orchestra.run.smithi114.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29910.log.gz 2024-09-16T05:27:35.910 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34234.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.29605.log.gz 2024-09-16T05:27:35.910 INFO:teuthology.orchestra.run.smithi114.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.34234.log.gz 2024-09-16T05:27:35.910 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30861.log 2024-09-16T05:27:35.910 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29614.log 2024-09-16T05:27:35.910 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30192.log 2024-09-16T05:27:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.29614.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.33818.log.gz 2024-09-16T05:27:35.911 INFO:teuthology.orchestra.run.smithi114.stderr: 89.8% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29614.log.gz 2024-09-16T05:27:35.911 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30861.log.gz 2024-09-16T05:27:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32271.log 2024-09-16T05:27:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31772.log 2024-09-16T05:27:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34759.log 2024-09-16T05:27:35.911 INFO:teuthology.orchestra.run.smithi114.stderr: 87.7%/var/log/ceph/ceph-client.admin.32271.log: -- replaced with /var/log/ceph/ceph-client.admin.30192.log.gz 2024-09-16T05:27:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31772.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32271.log.gz 2024-09-16T05:27:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34360.log 2024-09-16T05:27:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34759.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31772.log.gz 2024-09-16T05:27:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29925.log 2024-09-16T05:27:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T05:27:35.911 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34360.log: /var/log/ceph/ceph-client.admin.29925.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34759.log.gz 2024-09-16T05:27:35.912 INFO:teuthology.orchestra.run.smithi114.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34360.log.gz 2024-09-16T05:27:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33222.log 2024-09-16T05:27:35.912 INFO:teuthology.orchestra.run.smithi114.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29925.log.gz/var/log/ceph/ceph-mon.c.log: 2024-09-16T05:27:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33870.log 2024-09-16T05:27:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33222.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33222.log.gz 2024-09-16T05:27:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30218.log 2024-09-16T05:27:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31626.log 2024-09-16T05:27:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33870.log: /var/log/ceph/ceph-client.admin.30218.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33870.log.gz 2024-09-16T05:27:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34649.log 2024-09-16T05:27:35.912 INFO:teuthology.orchestra.run.smithi114.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.30218.log.gz 2024-09-16T05:27:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31626.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.31626.log.gz 2024-09-16T05:27:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34386.log 2024-09-16T05:27:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34596.log 2024-09-16T05:27:35.912 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34649.log: /var/log/ceph/ceph-client.admin.34386.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34649.log.gz 2024-09-16T05:27:35.913 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34464.log 2024-09-16T05:27:35.913 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34386.log.gz 2024-09-16T05:27:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34596.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34596.log.gz 2024-09-16T05:27:35.913 INFO:teuthology.orchestra.run.smithi114.stderr: 90.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T05:27:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32933.log 2024-09-16T05:27:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29542.log 2024-09-16T05:27:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33973.log 2024-09-16T05:27:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.29542.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34464.log.gz 2024-09-16T05:27:35.913 INFO:teuthology.orchestra.run.smithi114.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32933.log.gz 2024-09-16T05:27:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33301.log 2024-09-16T05:27:35.913 INFO:teuthology.orchestra.run.smithi114.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29542.log.gz 2024-09-16T05:27:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33170.log 2024-09-16T05:27:35.913 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33973.log: /var/log/ceph/ceph-client.admin.33301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34129.log 2024-09-16T05:27:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33170.log: 87.9% 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33973.log.gz 2024-09-16T05:27:35.914 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33301.log.gz 2024-09-16T05:27:35.914 INFO:teuthology.orchestra.run.smithi114.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33170.log.gzgzip 2024-09-16T05:27:35.914 INFO:teuthology.orchestra.run.smithi114.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31824.log 2024-09-16T05:27:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T05:27:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33353.log 2024-09-16T05:27:35.914 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34129.log: /var/log/ceph/ceph.audit.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31824.log.gz 2024-09-16T05:27:35.915 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34129.log.gz 2024-09-16T05:27:35.915 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32497.log 2024-09-16T05:27:35.915 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34570.log 2024-09-16T05:27:35.915 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33353.log: /var/log/ceph/ceph-client.admin.32497.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33353.log.gz 2024-09-16T05:27:35.915 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31879.log 2024-09-16T05:27:35.915 INFO:teuthology.orchestra.run.smithi114.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.32497.log.gz 2024-09-16T05:27:35.915 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34570.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34570.log.gz 2024-09-16T05:27:35.915 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34876.log 2024-09-16T05:27:35.915 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34103.log 2024-09-16T05:27:35.915 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31879.log: /var/log/ceph/ceph-client.admin.34876.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31879.log.gz 2024-09-16T05:27:35.915 INFO:teuthology.orchestra.run.smithi114.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34876.log.gz 2024-09-16T05:27:35.915 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31364.log 2024-09-16T05:27:35.915 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32868.log 2024-09-16T05:27:35.915 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34103.log: /var/log/ceph/ceph-client.admin.31364.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34103.log.gz 2024-09-16T05:27:35.916 INFO:teuthology.orchestra.run.smithi114.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31364.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30114.log 2024-09-16T05:27:35.916 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32868.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32868.log.gz 2024-09-16T05:27:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29754.log 2024-09-16T05:27:35.916 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34051.log 2024-09-16T05:27:35.919 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29675.log 2024-09-16T05:27:35.920 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29787.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.29787.log.gz 2024-09-16T05:27:35.920 INFO:teuthology.orchestra.run.smithi149.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.30236.log.gz 2024-09-16T05:27:35.920 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30101.log 2024-09-16T05:27:35.920 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29675.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.29675.log.gz 2024-09-16T05:27:35.920 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29966.log 2024-09-16T05:27:35.920 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30101.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30101.log.gz 2024-09-16T05:27:35.920 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30700.log 2024-09-16T05:27:35.927 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29966.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29453.log.gz 2024-09-16T05:27:35.927 INFO:teuthology.orchestra.run.smithi149.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.29922.log.gz 2024-09-16T05:27:35.927 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30599.log 2024-09-16T05:27:35.927 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29744.log 2024-09-16T05:27:35.927 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30599.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.30700.log.gz 2024-09-16T05:27:35.927 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30253.log 2024-09-16T05:27:35.927 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29744.log: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.29966.log.gz 2024-09-16T05:27:35.927 INFO:teuthology.orchestra.run.smithi149.stderr: 91.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30648.log 2024-09-16T05:27:35.928 INFO:teuthology.orchestra.run.smithi149.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30599.log.gz 2024-09-16T05:27:35.928 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29993.log 2024-09-16T05:27:35.932 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30648.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.30648.log.gz 2024-09-16T05:27:35.932 INFO:teuthology.orchestra.run.smithi149.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.29744.log.gz 2024-09-16T05:27:35.932 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T05:27:35.932 INFO:teuthology.orchestra.run.smithi149.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.30253.log.gz 2024-09-16T05:27:35.932 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29993.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T05:27:35.933 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29485.log 2024-09-16T05:27:35.936 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.29993.log.gz 2024-09-16T05:27:35.936 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31078.log 2024-09-16T05:27:35.942 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29485.log: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30597.log 2024-09-16T05:27:35.944 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30114.log: /var/log/ceph/ceph-client.admin.29754.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30114.log.gz 2024-09-16T05:27:35.944 INFO:teuthology.orchestra.run.smithi114.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29754.log.gz 2024-09-16T05:27:35.944 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32165.log 2024-09-16T05:27:35.944 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32035.log 2024-09-16T05:27:35.945 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34051.log: /var/log/ceph/ceph-client.admin.32165.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32165.log.gz 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34051.log.gz 2024-09-16T05:27:35.945 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:35.945 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29471.log 2024-09-16T05:27:35.945 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35476.log 2024-09-16T05:27:35.945 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32035.log: /var/log/ceph/ceph-client.admin.29471.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32035.log.gz 2024-09-16T05:27:35.945 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34412.log 2024-09-16T05:27:35.945 INFO:teuthology.orchestra.run.smithi114.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29471.log.gz 2024-09-16T05:27:35.945 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35476.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35476.log.gz 2024-09-16T05:27:35.945 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32324.log 2024-09-16T05:27:35.945 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33118.log 2024-09-16T05:27:35.945 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34412.log: /var/log/ceph/ceph-client.admin.32324.log: 94.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T05:27:35.945 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34412.log.gz 2024-09-16T05:27:35.945 INFO:teuthology.orchestra.run.smithi114.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32324.log.gz 2024-09-16T05:27:35.946 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33040.log 2024-09-16T05:27:35.946 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33066.log 2024-09-16T05:27:35.946 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33118.log: /var/log/ceph/ceph-client.admin.33040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30270.log 2024-09-16T05:27:35.946 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33066.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33118.log.gz 2024-09-16T05:27:35.946 INFO:teuthology.orchestra.run.smithi114.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33040.log.gz 2024-09-16T05:27:35.946 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29920.log 2024-09-16T05:27:35.946 INFO:teuthology.orchestra.run.smithi114.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33066.log.gz 2024-09-16T05:27:35.946 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31957.log 2024-09-16T05:27:35.946 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30270.log: /var/log/ceph/ceph-client.admin.29920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32987.log 2024-09-16T05:27:35.946 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31957.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29920.log.gz 2024-09-16T05:27:35.946 INFO:teuthology.orchestra.run.smithi114.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30270.log.gz 2024-09-16T05:27:35.946 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31957.log.gz 2024-09-16T05:27:35.946 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21501.log 2024-09-16T05:27:35.946 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32113.log 2024-09-16T05:27:35.947 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32987.log: /var/log/ceph/ceph.tmp-client.admin.21501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34025.log 2024-09-16T05:27:35.947 INFO:teuthology.orchestra.run.smithi114.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21501.log.gz 2024-09-16T05:27:35.947 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32113.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.32987.log.gz 2024-09-16T05:27:35.947 INFO:teuthology.orchestra.run.smithi114.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.32113.log.gz 2024-09-16T05:27:35.947 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33919.log 2024-09-16T05:27:35.947 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33092.log 2024-09-16T05:27:35.947 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34025.log: /var/log/ceph/ceph-client.admin.33919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30967.log 2024-09-16T05:27:35.947 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33092.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34025.log.gz 2024-09-16T05:27:35.947 INFO:teuthology.orchestra.run.smithi114.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33919.log.gz 2024-09-16T05:27:35.947 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34929.log 2024-09-16T05:27:35.947 INFO:teuthology.orchestra.run.smithi114.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33092.log.gz 2024-09-16T05:27:35.947 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27858.log 2024-09-16T05:27:35.947 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30967.log: /var/log/ceph/ceph-client.admin.34929.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30967.log.gz 2024-09-16T05:27:35.947 INFO:teuthology.orchestra.run.smithi114.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.34929.log.gz 2024-09-16T05:27:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29768.log 2024-09-16T05:27:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33503.log 2024-09-16T05:27:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.27858.log: /var/log/ceph/ceph-client.admin.29768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31546.log 2024-09-16T05:27:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33503.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27858.log.gz 2024-09-16T05:27:35.948 INFO:teuthology.orchestra.run.smithi114.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29768.log.gz 2024-09-16T05:27:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33557.log 2024-09-16T05:27:35.948 INFO:teuthology.orchestra.run.smithi114.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33503.log.gz 2024-09-16T05:27:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34727.log 2024-09-16T05:27:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.31546.log: /var/log/ceph/ceph-client.admin.33557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35372.log 2024-09-16T05:27:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34727.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31546.log.gz 2024-09-16T05:27:35.948 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33557.log.gz 2024-09-16T05:27:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33327.log 2024-09-16T05:27:35.948 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34727.log.gz 2024-09-16T05:27:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29610.log 2024-09-16T05:27:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30426.log 2024-09-16T05:27:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33327.log: /var/log/ceph/ceph-client.admin.29610.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35372.log.gz 2024-09-16T05:27:35.949 INFO:teuthology.orchestra.run.smithi114.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.33327.log.gz 2024-09-16T05:27:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33144.log 2024-09-16T05:27:35.949 INFO:teuthology.orchestra.run.smithi114.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29610.log.gz 2024-09-16T05:27:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.30426.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30426.log.gz 2024-09-16T05:27:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33947.log 2024-09-16T05:27:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35554.log 2024-09-16T05:27:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33144.log: /var/log/ceph/ceph-client.admin.33947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32245.log 2024-09-16T05:27:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35554.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33144.log.gz 2024-09-16T05:27:35.949 INFO:teuthology.orchestra.run.smithi114.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33947.log.gz 2024-09-16T05:27:35.949 INFO:teuthology.orchestra.run.smithi114.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35554.log.gz 2024-09-16T05:27:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33584.log 2024-09-16T05:27:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34286.log 2024-09-16T05:27:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33662.log 2024-09-16T05:27:35.950 INFO:teuthology.orchestra.run.smithi114.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32245.log.gz 2024-09-16T05:27:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34286.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33584.log.gz 2024-09-16T05:27:35.950 INFO:teuthology.orchestra.run.smithi114.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34286.log.gz 2024-09-16T05:27:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29568.log 2024-09-16T05:27:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29909.log 2024-09-16T05:27:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.33662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35528.log 2024-09-16T05:27:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.29568.log: /var/log/ceph/ceph-client.admin.29909.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33662.log.gz 2024-09-16T05:27:35.950 INFO:teuthology.orchestra.run.smithi114.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.29568.log.gz 2024-09-16T05:27:35.950 INFO:teuthology.orchestra.run.smithi114.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29909.log.gz 2024-09-16T05:27:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34260.log 2024-09-16T05:27:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34492.log 2024-09-16T05:27:35.951 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34956.log 2024-09-16T05:27:35.951 INFO:teuthology.orchestra.run.smithi114.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35528.log.gz 2024-09-16T05:27:35.951 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34492.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34260.log.gz 2024-09-16T05:27:35.951 INFO:teuthology.orchestra.run.smithi114.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.34492.log.gz 2024-09-16T05:27:35.951 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30889.log 2024-09-16T05:27:35.951 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34824.log 2024-09-16T05:27:35.951 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34956.log: /var/log/ceph/ceph-client.admin.30889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35424.log 2024-09-16T05:27:35.951 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34824.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34956.log.gz 2024-09-16T05:27:35.951 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30889.log.gz 2024-09-16T05:27:35.951 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32575.log 2024-09-16T05:27:35.951 INFO:teuthology.orchestra.run.smithi114.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34824.log.gz 2024-09-16T05:27:35.951 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35424.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35424.log.gz 2024-09-16T05:27:35.951 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32786.log 2024-09-16T05:27:35.951 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35320.log 2024-09-16T05:27:35.952 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.32575.log: /var/log/ceph/ceph-client.admin.32786.log: /var/log/ceph/ceph-client.admin.35320.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32575.log.gz 2024-09-16T05:27:35.952 INFO:teuthology.orchestra.run.smithi114.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32786.log.gz 2024-09-16T05:27:35.952 INFO:teuthology.orchestra.run.smithi114.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35320.log.gz 2024-09-16T05:27:35.955 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.31078.log: 95.2% -- replaced with /var/log/ceph/ceph-client.admin.29587.log.gz 2024-09-16T05:27:35.955 INFO:teuthology.orchestra.run.smithi149.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T05:27:35.955 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25226.log 2024-09-16T05:27:35.955 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30597.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.31078.log.gz 2024-09-16T05:27:35.956 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29688.log 2024-09-16T05:27:35.956 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.25226.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25226.log.gz 2024-09-16T05:27:35.956 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29712.log 2024-09-16T05:27:35.956 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29688.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.30597.log.gz 2024-09-16T05:27:35.956 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25298.log 2024-09-16T05:27:35.956 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29712.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.29712.log.gz 2024-09-16T05:27:35.956 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30086.log 2024-09-16T05:27:35.956 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.25298.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.25298.log.gz 2024-09-16T05:27:35.956 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29461.log 2024-09-16T05:27:35.957 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30086.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30086.log.gz 2024-09-16T05:27:35.957 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30221.log 2024-09-16T05:27:35.959 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29461.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.29688.log.gz 2024-09-16T05:27:35.960 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29986.log 2024-09-16T05:27:35.962 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30221.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.29461.log.gz 2024-09-16T05:27:35.963 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30021.log 2024-09-16T05:27:35.966 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29986.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29986.log.gz 2024-09-16T05:27:35.966 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T05:27:35.969 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30021.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.30021.log.gz 2024-09-16T05:27:35.970 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30052.log 2024-09-16T05:27:35.978 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph.audit.log: 92.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T05:27:35.978 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25274.log 2024-09-16T05:27:35.984 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30052.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.30052.log.gz 2024-09-16T05:27:35.984 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29904.log 2024-09-16T05:27:35.985 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.25274.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25274.log.gz 2024-09-16T05:27:35.986 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30383.log 2024-09-16T05:27:35.988 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29904.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29904.log.gz 2024-09-16T05:27:35.989 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31080.log 2024-09-16T05:27:35.990 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30383.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.30221.log.gz 2024-09-16T05:27:35.993 INFO:teuthology.orchestra.run.smithi149.stderr: 93.5% -- replaced with /var/log/ceph/ceph-client.admin.29485.log.gz 2024-09-16T05:27:36.003 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30994.log 2024-09-16T05:27:36.003 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30702.log 2024-09-16T05:27:36.006 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30994.log: /var/log/ceph/ceph-client.admin.31080.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.30994.log.gz 2024-09-16T05:27:36.006 INFO:teuthology.orchestra.run.smithi149.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.31080.log.gz 2024-09-16T05:27:36.006 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30245.log 2024-09-16T05:27:36.007 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T05:27:36.010 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30702.log: /var/log/ceph/ceph-client.admin.30245.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.30702.log.gz 2024-09-16T05:27:36.010 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29445.log 2024-09-16T05:27:36.016 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-mon.b.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.30245.log.gz 2024-09-16T05:27:36.026 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29816.log 2024-09-16T05:27:36.034 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29445.log: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.30383.log.gz 2024-09-16T05:27:36.042 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T05:27:36.053 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29816.log: 91.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25202.log 2024-09-16T05:27:36.059 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-osd.4.log: 90.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30997.log 2024-09-16T05:27:36.061 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.25202.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.25202.log.gz 2024-09-16T05:27:36.061 INFO:teuthology.orchestra.run.smithi149.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29445.log.gz 2024-09-16T05:27:36.061 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30701.log 2024-09-16T05:27:36.064 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30997.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.30997.log.gz 2024-09-16T05:27:36.065 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29469.log 2024-09-16T05:27:36.066 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.30701.log: -- replaced with /var/log/ceph/ceph-client.admin.29816.log.gz 2024-09-16T05:27:36.067 INFO:teuthology.orchestra.run.smithi149.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.30701.log.gz 2024-09-16T05:27:36.068 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29997.log 2024-09-16T05:27:36.076 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/ceph-client.admin.29469.log: 92.4%/var/log/ceph/ceph-client.admin.29997.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29469.log.gz 2024-09-16T05:27:36.076 INFO:teuthology.orchestra.run.smithi149.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29997.log.gz 2024-09-16T05:27:36.320 INFO:teuthology.orchestra.run.smithi149.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T05:27:36.331 INFO:teuthology.orchestra.run.smithi114.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T05:27:36.390 INFO:teuthology.orchestra.run.smithi149.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.admin.29623.log.gz 2024-09-16T05:27:37.021 INFO:teuthology.orchestra.run.smithi149.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29536.log.gz 2024-09-16T05:27:37.052 INFO:teuthology.orchestra.run.smithi149.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T05:27:37.650 INFO:teuthology.orchestra.run.smithi114.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T05:27:38.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-09-16T05:27:42.595 INFO:teuthology.orchestra.run.smithi114.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T05:27:42.617 INFO:teuthology.orchestra.run.smithi149.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T05:27:42.839 INFO:teuthology.orchestra.run.smithi114.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T05:27:42.846 INFO:teuthology.orchestra.run.smithi149.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T05:27:43.251 INFO:teuthology.orchestra.run.smithi114.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T05:27:43.252 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T05:27:43.253 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m7.450s 2024-09-16T05:27:43.253 INFO:teuthology.orchestra.run.smithi114.stderr:user 0m22.915s 2024-09-16T05:27:43.253 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m0.876s 2024-09-16T05:27:43.832 INFO:teuthology.orchestra.run.smithi149.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T05:27:43.833 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-09-16T05:27:43.833 INFO:teuthology.orchestra.run.smithi149.stderr:real 0m7.995s 2024-09-16T05:27:43.833 INFO:teuthology.orchestra.run.smithi149.stderr:user 0m24.409s 2024-09-16T05:27:43.833 INFO:teuthology.orchestra.run.smithi149.stderr:sys 0m0.847s 2024-09-16T05:27:43.834 INFO:tasks.ceph:Archiving logs... 2024-09-16T05:27:43.834 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906097/remote/smithi114/log 2024-09-16T05:27:43.835 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T05:27:47.181 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906097/remote/smithi149/log 2024-09-16T05:27:47.182 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T05:27:50.340 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T05:27:50.349 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T05:27:50.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T05:27:50.352 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T05:27:50.430 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T05:27:50.442 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T05:27:50.442 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T05:27:50.445 DEBUG:teuthology.orchestra.run.smithi149:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T05:27:50.456 INFO:teuthology.orchestra.run.smithi114.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T05:27:50.456 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================== 2024-09-16T05:27:50.456 INFO:teuthology.orchestra.run.smithi114.stdout:*hv01.front.sepi 67.205.162.81 3 u 43 64 377 0.098 -5.996 0.386 2024-09-16T05:27:50.456 INFO:teuthology.orchestra.run.smithi114.stdout:+hv02.front.sepi 63.231.80.2 3 u 51 64 377 0.064 -6.641 0.407 2024-09-16T05:27:50.456 INFO:teuthology.orchestra.run.smithi114.stdout:+hv03.front.sepi 74.6.168.72 3 u 56 64 377 0.108 -4.570 0.741 2024-09-16T05:27:50.456 INFO:teuthology.orchestra.run.smithi114.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-09-16T05:27:50.458 INFO:teuthology.orchestra.run.smithi149.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T05:27:50.458 INFO:teuthology.orchestra.run.smithi149.stdout:============================================================================== 2024-09-16T05:27:50.458 INFO:teuthology.orchestra.run.smithi149.stdout:*hv01.front.sepi 67.205.162.81 3 u 51 64 377 0.077 -4.810 0.347 2024-09-16T05:27:50.458 INFO:teuthology.orchestra.run.smithi149.stdout:+hv02.front.sepi 63.231.80.2 3 u 49 64 377 0.100 -5.193 0.401 2024-09-16T05:27:50.458 INFO:teuthology.orchestra.run.smithi149.stdout:+hv03.front.sepi 74.6.168.72 3 u 46 64 377 0.112 -2.303 0.339 2024-09-16T05:27:50.458 INFO:teuthology.orchestra.run.smithi149.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-09-16T05:27:50.459 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T05:27:50.467 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T05:27:50.467 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T05:27:50.499 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T05:27:50.552 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T05:27:50.591 INFO:teuthology.task.internal:Duration was 1824.016363 seconds 2024-09-16T05:27:50.591 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T05:27:50.626 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T05:27:50.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T05:27:50.629 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T05:27:50.667 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T05:27:50.667 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T05:27:50.667 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T05:27:50.717 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi149.front.sepia.ceph.com 2024-09-16T05:27:50.717 DEBUG:teuthology.orchestra.run.smithi149:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T05:27:50.728 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T05:27:50.728 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T05:27:50.760 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T05:27:50.797 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T05:27:50.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T05:27:50.828 DEBUG:teuthology.orchestra.run.smithi149:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T05:27:50.917 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T05:27:50.928 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T05:27:50.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T05:27:50.941 DEBUG:teuthology.orchestra.run.smithi149:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T05:27:50.953 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T05:27:50.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:27:50.988 DEBUG:teuthology.orchestra.run.smithi149:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:27:50.998 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-09-16T05:27:51.003 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = core 2024-09-16T05:27:51.026 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:27:51.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:51.071 DEBUG:teuthology.orchestra.run.smithi149:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:27:51.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:51.076 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T05:27:51.086 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T05:27:51.087 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906097/remote/smithi114 2024-09-16T05:27:51.087 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T05:27:51.135 DEBUG:teuthology.misc:Transferring archived files from smithi149:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906097/remote/smithi149 2024-09-16T05:27:51.135 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T05:27:51.168 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T05:27:51.169 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T05:27:51.176 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T05:27:51.199 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T05:27:51.211 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T05:27:51.211 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T05:27:51.242 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T05:27:51.242 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T05:27:51.244 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T05:27:51.249 INFO:teuthology.orchestra.run.smithi114.stdout: 658637 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 05:27 /home/ubuntu/cephtest 2024-09-16T05:27:51.250 INFO:teuthology.orchestra.run.smithi149.stdout: 658462 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 05:27 /home/ubuntu/cephtest 2024-09-16T05:27:51.252 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T05:27:51.275 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T05:27:51.344 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_api_tests} duration: 1824.0163626670837 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T05:27:51.344 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T05:27:51.429 INFO:teuthology.run:pass